How To Use BenchmarkDotNet - A Beginner's Guide For C# Benchmarks

so a lot of you have probably seen your favorite c-sharp content creators creating videos that have benchmarks in them and you've been wondering how can I do that for myself well I recently had someone comment on a video and asked hey how can I set up benchmarks like you've shown in here and I figured I'd make this video to introduce you to benchmark.net and get you set up so that you can do it for yourself so let's jump over to visual studio alright so I'm here in visual studio and I've just created this new project to demo to you guys so I'm gonna go ahead and right click on the project because benchmark.net is a nuget package and I'll go to manage nuget packages and from there on the browse tab I'm going to search benchmark.net spelledout.net not with a DOT and you can see this package right here at the top currently has 19 million downloads I'm going to be installing version 0.13.5 if you're watching this video at a later date this version and these download numbers might be different but this is the logo you want to look for and you can see the price project URL here as well so I'm just going to press install and then I'm going to press OK for this dialog that pops up and then I accept for this other one that follows and just like that we now have the nuget package installed so I'll go ahead and close this package manager and go back to program.cs where we can get started alright so to get started with benchmark.net once we have the nuget package there's a couple things that we need to be able to do one is going to be actually defining the benchmarks that we want to create and then the second part is actually being able to select which ones we'd like to run so just to start us off I'm going to paste in the code here that is a benchmark switcher this is built right into benchmark.net and it's pretty awesome there's a couple of different options that we can look at here but I'm going to be using this Dot from assembly method that we have here and then I'll be saying that we want to get the current assembly so there's different ways that you can get the current assembly you could do something like assembly.get executing assembly as well but if you wanted to be able to select from different plugins or something more advanced you could always change this up if you have your benchmarks located somewhere else the final part that will attack onto this is the dot run and it does take in the command line arguments when you're not using top level statements that's just args and if you are using top level statements and you have a main method you also have args that can get passed into that main method as well so I'm going to go ahead and press run on this and of course we don't have any benchmarks so just to show you what it looks like we see that this outputs no benchmarks to choose from make sure you provided public non-sealed non-static types with public Benchmark methods on them so that's actually a nice little hint for us as we're going through so we're going to look at how we can actually create those benchmarks but once we do that the proper way again public non-sealed non-static types and then public methods that have this Benchmark attribute on them those are the requirements once we've provided those we'll actually be able to see them listed out here in the selector and then we can pick one and run it all right so let's go ahead and actually create our Benchmark class so it needs to be a public class that is not static and not sealed and that's because something about how the Benchmark Runner actually works is that it's going to create an instance of this and it actually needs to be able to inherit from it I'm not entirely sure how it works behind the scenes but my understanding is that if you made this sealed and you actually try to force it to run it will tell you that it's unable to do so because of something like that and if you see my autocomplete is actually suggesting exactly what we would want to do to be able to create a benchmark method and like the output of that console said we need something just like this where we have a benchmark attribute and we just need the proper using statement for that which I'll add at the top now and then a public void of our Benchmark so you can name this whatever you would like and it just has to be public and have this attribute on top of it so currently our Benchmark won't do anything at all but let's go ahead and run this again and we can see if the selector actually shows us what we can pick alright so you can see all of the green here this is a good sign it says available Benchmark it says number zero here our benchmarks which is the class that we created so we can press zero and it will go run this for us however it actually prints out an error for us because we are running this in debug mode and we need to be able to run it in release there are some options you can use to override this but again if you're trying to do actual benchmarking you do want to be using a release build and you actually do not want the debugger attached to the process either so instead of me pressing F5 to go run this or pressing the play button at the top I'm just going to show you that I can right click on the project it's self and I can go to the debug menu and then I can say start without debugging now the other thing that I need to do though is switch this debug in the top left here over to release so let's go ahead and do that then I'll right click again I will go to the debug menu and then start without debugging now I'll select Benchmark 0 which is the only Benchmark that we have press enter and it will go run them for us alright now this Benchmark is not going to be very exciting because we don't actually do anything inside of our method but we can see the output here where it says the method name and then the mean and then the error from it so we're going to look at a couple of other things and actually change the content of our benchmarks that we can see some actual useful information if you've been following along in visual studio and trying to do this at the same time that you're watching it you might have tried to run this exact code and said hey Nick that took way too long for something that actually didn't do anything so I've edited the video of course just to show you the output but without any attributes on our class here called our benchmarks it actually tried to go do a lot of warm-up and actually go do a bunch of iterations to go run this Benchmark it actually took way longer than I would have wanted it to for something that doesn't actually do anything so something that you can consider is these attributes called short run job and there's also medium run job and you can decide what kind of job that you want to go run but by picking short run job it's actually going to do a lot less work and one might argue that if your benchmarks are going to be something that you want to publish and have other users look at the data you probably want to spend a little bit more time running longer benchmarks so that you get more accurate data but for almost everything I'm doing and just exploring short run job does the trick for me so let's actually look at the code inside of our Benchmark and of course right now it's not actually doing anything at all so let's make it a little bit more interesting what would happen if we wanted to go sort the items in a list well we could go do something like list of ins and then make a new list here and then we could go fill this list with a bunch of stuff and thank you autocomplete let's go do that we'll add a bunch of things into this and then we can say list dot sort now if we wanted to go Benchmark the sort method here in particular something interesting to note is that this Benchmark is actually technically going to be testing all of this so if you want to just know how long the sort was taking and maybe some of the memory implications then you're not going to want to have this inside of the Benchmark method itself so we can go look at how to improve that and because I mentioned memory as well something else that we can do is actually put another attribute on here called memory diagnoser and when we do this in the output that we'll see when we do go to run this we'll actually see some columns related to memory allocations so just a quick pause to see where we're at we do have our public non-static non-sealed class called our benchmarks we have one Benchmark method we have it marked as a short run job so that it doesn't take as long to Benchmark and we have this memory diagnoser attribute on it so that we can see see some memory information now if we're interested in just profiling this sort method well we're going to be having to move this to a different location so that it's not included in the Benchmark process itself so let's go ahead and cut that out and we can go explore another spot that we can put this so something that we're able to do in benchmark.net is create a method and I don't think it actually matters what you name it as long as you have this Global setup attribute on it what we can do in here is actually paste in the code that we were interested in but the thing that we have to be considering now is that we're in different scopes for where our variable is declared so we actually need to go move this to a field and now you can see that this field is shared between this setup method and this Benchmark method I personally underscore all my Fields so that they're separate and they stand out so that's my naming convention that I use so I'm just going to go ahead and do that and then the other thing that you'll notice is just because I have some of these options turned on in my project it's suggesting that this list object is actually potentially going to be null and we might want to be concerned about where this is used just to make These Warnings go away I will mark this as a nullable list and then I'm going to say that when we go to run our Benchmark I'll put this bang operator here because I am confident that this will not be null when we go to run the Benchmark and that's of course because this setup method will get run before this Benchmark and just to elaborate a little bit further about what Global setup does this is going to be something that we mark this method with and this method will end up running before all of the Benchmark methods that we have so in our current example we only have one but if we had other benchmarks that were added in here let's see if autocomplete Works some magic there we go so if we had these two benchmarks then this Global setup will run once before both of these benchmarks get run and that way you can set up your data only one time and that way you're not having some extra overhead even though it's outside of your benchmark but you don't have this extra overhead of like setting things up per Benchmark for our purposes we're only going to be looking at this one Benchmark method but like I just Illustrated you can have multiple so what we want to look at now is if we go back to this Global setup we do have this 1000 number that was put in by the autocomplete thanks copilot I'm not exactly sure why it picked 1000 but if we were interested in benchmarking things we might want to pick particular ranges that we're interested in and actually there is something in benchmark.net that makes this very easy for us and I'm just going to paste it on my clipboard here so we have this public int list size so I have a public field here and then I have this params attribute and what we're able to do with that is actually specify the different values that we would like to have for our benchmarks so in this particular case I've just put in four different variations so we have one thousand ten thousand a hundred thousand and a million and that what we want to do from there is actually use this in our setup method and we can use list size right here so to actually explain what's going on when this happens we need to think about how this Global setup works and how the params work so you can have different params not just this one list size and you want to think about benchmark.net actually creating a matrix of the different combinations of params that you can work with in our particular example our Matrix is only one dimensional because we only have this one param what will actually happen is that per our one Benchmark that we have here our Global setup will run once before this Benchmark and it will use the first param and then once that's complete complete the global setup will run one more time and it will use the second variation and then it will use the third and then the fourth and again if you had multiple params it will actually do a combination creating a matrix of all of the possible combinations for your parameters now that we have the basis for our Benchmark set up let's go ahead and actually run this so remember that we want to keep it in release and then I'm going to run this without debugging so I will go to this context menu debug and then start without debugging all right then we got our selection so I'll press 0 and enter and then I will edit the video and we can look at the output all right now that the benchmarks have completed we can actually go look at the output table this will look a little bit different than we saw earlier because if you recall I did get us to include that memory diagnosis attribute so we're going to see that we have this allocated column now as well and this shows us the memory allocations so our benchmarks are actually listed out here and even though we have this one method as we discussed we actually have four different variations because as we saw with the params attribute we were able to specify one thousand ten thousand a hundred thousand and a million for the list size if we go over to the mean column we can see that for one thousand elements it took just under three microseconds to go run this when we went up an order of magnitude for the list size so that's now ten thousand elements we went up to 43 microseconds and then an order of magnitude up again we're at 511 microseconds and then one more order of magnitude up to one million for the list size we are now at 6108 microseconds interestingly the allocated column really didn't use much memory at all and even for the larger list sizes we only had one byte and five bytes and do recall that because we initialized the list in the global setup we are not seeing the allocation costs for the size of the list because because that's done outside of the Benchmark itself and that's just a quick crash course with benchmark.net and just a quick recap we were able to look at how we needed to define the class so it needed to be public not static and not sealed and then we actually had to label the public benchmarks inside of that class with a benchmark attribute we also discussed the job switcher that we had at the beginning of the program and that lets us actually switch between the different benchmarks and then we looked at the output and we could talk about the neem column we could see that we had the Benchmark method name and the different parameters as well as the allocated column that we got by having that other attribute on the top of our class finally if your benchmarks were taking a little bit too long for your own patience we did look at the short run job attribute and you can play around with that for your own purposes so hopefully you found this helpful thanks so much for watching and we'll see you next time foreign