A Beginner's Guide to MongoDB Filtering in C#

mongodb is a very popular document database and if you're writing applications in C and you want to leverage mongodb odds are you need to be able to query for documents at some point hi my name is Nick centino and I'm a principal software engineering manager at Microsoft in this video I'm going to walk you through the basics of using C to work with mongodb in order to query different documents that you have in your mongodb database we'll look at some of the basic filters and the methods that we have access to a quick reminder to check that pin comment for information about my free Weekly Newsletter and my courses on dome train okay so this is going to be using the mongod DB C driver if you're not familiar with how to set all of the things up that you see on the screen from this part right up to the beginning here including your connection string and all of that you can check out this video that I posted before and then come right back and continue watching this but that'll walk you through how you can start inserting documents and connect to your mongod DB database what we're going to be looking at is filtering and in particular we're going to use filters to be able to ask mongodb for records that match those filters so to start off what we're going to use is this filter property that we have off of this Builder static class in my particular case I have a collection and I'm not going to be using dto that are mapped specific records so I don't have a dedicated record type that I've created we're just going to be using the bson document to be very generic and open-ended but we asked the Builder static class for Bon documents this generic type here for this filter property and that filter property gives us this filter Builder that we have here and the filter Builder is going to have the different methods on it that we can use to construct filters and combine them together so we're going to start with some really basic options here what I'm doing here is starting with line 18 we have filter builder. empty so I'm assigning this completely empty filter to a variable and what we can do from there is start to combine this empty filter with other things that we're looking for and this is a completely unnecessary step if you're doing something as basic as this filter right here so if we just wanted to match the name property of our record to look for one that says Nick centino so we're looking for me if we just wanted to do that we could have literally replaced that entire part on line 18 with just code from right here so that's all you would need but if you're starting to build more complex filters up and you need to optionally include or exclude different things I like being able to have a filter variable that I can start tacking things onto so that's why this is structure this way and we'll see a little bit more of that later now I want to go run this and show you how it works but we need to see how we call mongod DB that driver class that we have in order to perform the query and I need to show you that and I need to show you the records in the database so that we can see what we should be expecting back so I scroll down a little bit lower we'll see that from line 42 till 55 so this block of code here I just have a couple of different ways that we can go ask for that data collection. find passing in the filter is what we need to do to be able to start getting results but once we pull those results back how we enumerate them or how we look at them can be done in a canful of different ways so what I'm doing on line 46 is I'm just saying hey results I just want to put you into a list and you'll notice that A variation of that on line 50 I'm doing a 4 each Loop over it but results itself is not an I inumerable so we have to ask it to be made as an I inumerable with this method to enumerable but both of these ways are options that you could use right so calling to list on something and materializing or making a copy of a collection could be very expensive for memory if we were able to just stream things in that might be more performant what I have not done at this point in time which will be a follow-up video is doing performance characteristics both for runtime and memory footprint about these two different options so for example if we've called find has it materialize the full result set in memory already if that's the case is there a streaming API that we could use instead all of that still to come this is just the very basics of being able to filter and pull things back so I will show you both these variations the results will be identical when we see them in the console which is good news and before I run this let's go look at Compass so Compass is going to be the tool that you can get directly from mongod DB it's just a an Explorer you can run queries and stuff on it you can even edit the documents that you see on the screen uh and change them to be whatever you'd like or add or delete them right so um I just press this to show the Json for of it but if we're looking for records or documents that have name as Nick centino if we have three documents in this collection you can see two of them do have my name the one at the bottom here that says Dev leader that name is obviously not going to match Nick centino so if we go back to the code we should expect that if we're asking for Nick centino we get two results in both of these cases right it's the same query we're just doing a for each Loop in one case and two list on the other so let's go run this and see what happens okay so it's a little bit messy because we printed out the records in the second case but you can see it starts with finding records converting those results to a list and then it says we got two results in that list I didn't do a four each loop on that list we're just getting the count printed out but in the second example for enumerating the results you can see that it got two records coming back right and both of them have name as Nick centino and name as Nick centino in the second one so it did pull back two of them and they both have unique IDs this one ends in D9 and this one ends in da so two records came back out of the three so that should prove that we filtered on the two that match now briefly I wanted to show you this other syntax that we can use right here on line 23 I personally think that this is a lot more readable than using sort of this a little bit more complex syntax or a little bit more verbose where we have to keep asking the filter Builder to and so basically asking the filter Builder to create this other object that we can add two filters inside of to and them right we can kind of skip that entire thing just by doing this so we're saying I want to take the filter and end it with and then reassign that's and equal to just the filter that we want to do if we read what's on the screen now would be starting with an empty filter and then anding it to this one and then reassigning that back into filter it's exactly what this did but in my opinion as long as you're comfortable with this operator so the and equal I think it's a lot more readable and when you start having more and more filters put together I think the uh amount of verbosity that you get with this is just a little bit Overkill okay next I want to talk about the or operator right so we just saw and equals this is the or equals the pipe operator is or and what I'm going to do is have another filter on name and I'm going to look for text that shouldn't match anything and I want to show you something kind of peculiar about doing this you'll notice that I'm just reusing the variable filter and reassigning it to empty just so I can keep this example going but 27 means we're starting with an empty filter then from there I'm going to or it with the name filter that won't match anything and we should see something interesting when I go run this so let's do the same example here and even though it wasn't supposed to match anything it matched all three records and in fact if you look at the name for all of them none of those names say doesn't match anything right that's what we asked for it says name doesn't match anything and it included everything so why is that the case well the empty filter itself is kind of like match all so what we've effectively done is said take a match all filter and or it with something that won't match anything the result of that is that it will still match everything so we have to be very careful if we're using this pattern that I was talking about with aing because if we do that with an empty filter that means that we're going to match anything all the time because anything you ore with this will always match everything but okay we want to see The Ore filter in action not just accidentally including everything because it was combined this way and we've already seen the and equals so let's do maybe a little bit more complex of an example okay so this particular example uses an and equals and an or equals so if I'm doing this here if I'm and equaling name to not match anything this combination that I have highlighted will say match everything and match it with nothing and because it's an and it means that nothing will end up matching now if we have a filter that matches is nothing and then we or it with this one this says that we should match nothing or match subscribers equal to 1,000 which effectively means that we'll only be matching things where subscribers equals 1,000 so let's do this in a couple of steps I want to prove to you first that by running this we shouldn't get any results because there's nothing with the name no match here that should be pretty obvious and before moving on I just wanted to take a quick note to show you my Dome train courses that I have available now I do have two that are focused on C which you can see right here and right here so we have a getting started and a deep dive course the getting started one is perfect for people that haven't even programmed before I'll teach you all the basics you need to get up and running and working with c and the Deep dive builds directly on top of that so that you have more experience working with c and you feel a lot more capable working with the language to go develop applications together both of these courses are just around 11 1/2 hours of all of the basics of C that you'll need to get up and running of course building on top of all of that I have my refactoring course that's a ailable and this is going to teach you different methods that you can use to refactor code clean up your code base and make sure that you're not trapped working with spaghetti Legacy code if you like the style of my YouTube videos and the way that I teach I think these courses will be perfect for you and there we go zero results but that's not that exciting because we knew that was going to work now if we do this and we're oring it here what should happen we got no results here and this is subscribers equals 1,000 but do we have anything that matches 1,000 subscribers well the answer is no right because we only have things that have a million subscribers here and that's on purpose I want to walk through this example slowly because the only time we saw the or operator working was matching on everything that's not what we want so I want to show you that when we add an or operator in here it will in fact to do what we want and still not match anything because it's not finding anything with 1,000 subscribers okay now if we go to or in a million subscribers this should effectively match everything again but we can get a a little bit more creative because what we can do is go back to here and maybe modify one of these to not have a full million subscribers so I'm going to change this last one that says Dev leader because it's different right I'm going to make this one be 1 two 3 and we'll do that many so if we're trying to look for a million subscribers this last one should be excluded so just again to revisit what we have we're anding this filter that's matching no match here on the name to empty then we're oring in this one which now should include all of the subscriber counts of 1 million and there we go we should be able to get just these two records back this is showing the or filter properly including these things right it's not including everything and it's not including everything because the subscriber count on the third record was 1 2 3 and I wanted to make a quick note about this because the filter syntax that we have here with and equals and or equals can get very complicated very fast depending on what you're trying to do you really need to think about the grouping of the things that you're combining together so for example once you start having ores included in here you need to be thinking about if you're trying to group them up like think about where you would put the parentheses if you were writing this out as an if statement cuz if you wanted this or this and something else but this or this group together you need to go create a filter that's those two things or and then take that filter and and the whole thing with another one so do try to keep that in mind as you're building these out if you're trying to just J exam it all into one filter so if not breaking it out into separate variables so that you can combine them more easily I think you're going to be in for a lot of trouble trying to debug you know why certain things are matching it's just this really easy opportunity to mess it up and finally I want to show you another variation of this fine method where we're not changing what the filter is so we're going to get the same set of results but we're going to use a cursor instead and what we're going to do with that cursor is change the batching size that we're pulling back and I think going back to what I said earlier I do need to Benchmark these to show you the differences for memory footprint and performance but based on what I'm seeing here having the batch size being something we can configure probably means that if you're not batching there's probably some upper limit to what we're pulling back into memory or maybe it's Unbound which seems a little bit unsafe but if you want to be able to control that I think this is the thing that you're going to want to use when we use a cursor so you can see down here we get this move next syntax so it kind of feels like using a for each Loop but the difference is that we're going to be getting sets of data in batches which is what this part controls is outer loop and then once we have a particular batch then we use a 4 each loop on that so you can think about it like if we had a million records and we were trying to pull them back in batches of a thousand we would go through move next a whole bunch and every time we did this for each Loop it would be on sort of like Pages or batches of 1,000 records I suspect that this probably controls the memory footprint but has some overhead in terms of round trip time but still to be benchmarked so I will follow up with that in a video so what we're going to do is run this I have a batch size of two and I'm going to put a break point in here as well because I want to see if we have three records if we end up hitting this move next part twice to get two pages right one will be a page of size two or a batch of size two and the next will be of size size one but that means that we should go adjust our filter as well so I'm just going to take off the filter so instead of seeing just those two results come back we should see all three and that means that we should see this hit twice okay so this is the first breakpoint hit so we're going into here this is the first page hopefully we see two come through that's the second iteration on that first page and if we go back up we should go in one more time and we do and this time this for each Loop should only go in once that's for our third item and final item so we go in okay write the third result out and this should not go in now because we've seen all three great and when I go back up to the top move next should be false so we should hopefully leave the loop right now and we do so the find options with a batch size allows you to control that and then with the cursor we can step through Page by page with a little bit more control and those are some of the options that we have to play with when querying documents for mongodb we got to see a handful of different ways where we could take things to a list to a numerable or using a cursor and we also got to see how we could combine filters using and equals and or equals or that more verose syntax where we ask the filter Builder to and two other filters together just a quick reminder that you're going to want to pay a lot of attention when you're building up complex filters with ANS and oras because once you start incorporating oras into there you really want to think about where your parentheses in theory would go right so try to draw it out on paper try to draw it out as an if statement first so you can see where you put the parentheses and think carefully about how you'd combine those things if you've enjoyed this and you want to learn more about using mongod DB and C when the next video is ready you can check it out here thanks and I'll see you next time