Is This A Testing Crime or Calculated Risk? [ASP.NET Core Example]

look I'm guilty of a testing crime and I know that you are too many of you as software developers were always trying to move fast write more code get more things done deliver more features fix more bugs go go and if you've seen a lot of my content you'll know that I talk about testing an awful lot and I do take it seriously yet here I am telling you that I committed a testing crime you see it's not that I decided to skip writing tests it's that I decided to write a different kind of test so that I could get myself some confidence but save myself some time so this was a calculated decision I decided that the risk versus reward so the amount of time that I'd have to put in to get the confidence back out was worth it and that means that I was writing a style of tests that I thought would be beneficial to allow me to keep moving but still give me confidence over the code that I was changing so yes a testing crime in this case but it wasn't that I went all the way to just skip writing tests all together and I know that some of you do that and you shouldn't you should write tests for your code so what happened in my situation well in my previous videos which I'll link above I've been talking about that I've been writing these tests that would give me confidence over these asp.net core apis that I've been writing and because we're iterating really fast on our project I didn't have time to go dissect everything go and write really granular unit tests and a lot of the stuff that I have is interfacing with mySQL databases and quite frankly I don't think that writing unit tests over a boundary that's going to execute SQL queries is very valuable at all what I did write were functional tests or integration tests that would go launch containers go actually exercise the routes of my application so from the top level of the app all the way down to the database and back up to return an HTTP result however when you write tests like this that cover so much the exercise so much of your code base it's basically impossible to be able to write really granular assertions because if you had to test everything thing in detail along the way and actually assert it your number of assertions would be ridiculous so I traded off having granular assertions for more code that I could exercise in a single test and guess what it worked for a while but it did end up catching me off guard the other day and I wanted to share that experience with you so before we jump over to visual studio please consider subscribing to my newsletter I send it out on the weekend it will have updates from the week including software engineering topics c-sharp topics and some of the content that I've been putting out it'll have Community spotlights as well and some learning resources that aren't just for me it's only a quick five minute read so I hope you enjoy it I hope you consider subscribing with that said let's jump over to visual studio and check out where I messed up alright so I'm here in visual studio and if you've seen my other videos I've been talking about how I set up some functional tests to be able to kick off some testing infrastructure that kind of simulates a production environment except using test containers if we look at the construct of this test class we can see that we have some fixtures passed in so this is an X unit concept and then these are details specific to my application but I've created this Builder pattern to be able to set up some of the test data and the test containers so that I have some infrastructure to work with and with this Builder pattern it does seed some data so that I can pretend that I'm working with something real in a real production environment we might be talking about millions of Records so instead of actually using that I just have a small subset of the data that I populate now I'm going to jump down to a test that's down here at the bottom that talks about getting a recipe through my route so this line here on line 119 is actually calling the route through an API client and by the time we're running this line here we actually have an entire asp.net core application up and running and connected to the MySQL test container and yes this route name is a little bit ridiculous I kind of realized in hindsight that I should not have a get in here because it is a get method so I will change that with that said though this line here actually hits the server and pulls back a result that we would go parse with this assertion here and we're looking for this dto which has get recipe details response defined and that will have the recipe information that we're pulling back now this particular test method actually says in the name that the basic details are asserted so while I'm not actually validating every single thing that's coming back as the response here I am looking at a handful of things the other thing worth mentioning is that the actual test data I'm using I'm not actually defining something very specific I'm actually just asking for any recipe that this user can see and then these variables here are actually just the paging and offset so I'm saying start at zero and pull one record so these lines of code right here actually just filter for a single recipe any recipe that this user can see and it checks that we have one single recipe so you can already see that by not being very specific at all with my test setup I'm just saying I want to go pull any recipe back that I can see and I'm going to double check that there's some basic information that comes back so yes there are assertions but these are very simplistic they are not going to be very brittle and because I'm not looking for anything specific because of these lines of code right here this is just really going to be a test that exercises a bunch of code and not actually assert on a lot of the details now the risk that I want to call out with doing this is that this can give you a false sense of confidence if you're assuming it's doing too much I've already stated a couple of times that I wanted to write tests like this just so that I could actually exercise a lot of code so this did fulfill the purpose that I was going after however it only gets you so far because when you start making some other types of changes and you don't have other test coverage say like unit tests or other functional Tech s with more test data you will run into situations where you're just not covered and that's exactly what happened to me let's go have a look at what this dto actually looks like so this is the model for get recipe details response it has a single recipe details on it so if I go into that we can see that we have a bunch of information on it and we're going to have this ingredient range so to talk about the change that actually messed me up I was trying to do some SQL queries in bulk instead of doing them one at a time and that for me was going to alter that I had way down in my repository that's pulling code back from the database a method that would Loop over more records and return an aggregate of them instead of just a single record so if we look at ingredient range this is just going to Define some of the food IDs and the range that these foods can be generated on for a recipe but if we go back we can see that we're talking about having an entire array of ingredients and that's because if you think about it a recipe has ingredients it's made up of multiple things and some recipes You could argue are just made up of a single ingredient but the point is this can be multiple things and most of the time it is multiple things Okay cool so this is right at the level of the route so at the top most part of the application if you were to think about it like a stack now I'm going to go all the way to the bottom we're going to go look at the repository that's responsible for pulling this information back okay so we're in the repository and I'm going to explain the change that I made that actually caused this funny bug to show up in my code and not get caught by my test so I had just mentioned that I wanted to change some of my queries to actually pull some stuff back in bulk and the reason for this is that I can make fewer trips to the database to get the same information when I was examining my code I saw that this method here get by recipe definition IDs unordered async was actually called many times and it was actually just using a single ID passed in which meant that if we had to call it 10 100 100 a thousand times we were actually just pulling back one set per ID and that meant that we had to go do many round trips to the database instead I said well instead of doing this in a loop and making multiple queries why don't I just go run one query and pull back all the information at once however it's going to be way faster than making multiple round trips for each time that I'd want to go look up by a single ID that you'd see on line 127. now to go make this actually work wasn't actually too difficult but there was a bug in my code which I will touch on but the result that I ended up seeing was that every single call to this and I noticed it on the API when I was actually looking at my app and looking at the results was that every recipe was coming back with just one ingredient now it caught me off guard because I do have a couple of recipes that are things like boiled eggs and boiled eggs do just have one ingredient but the more I was clicking around in my application I realized wait a second it's impossible that all of these recipes just have a single ingredient and then it dawned upon me that I Am My Own Worst Enemy because I realized I just made this change recently and I hadn't seen this problem before I knew that I had to go back through my commits and check to see the code that was altered in this particular spot now the bug was actually one single character that was incorrect and I had to go figure out why I was only getting one ingredient per recipe the actual logic to go pull back ingredients across many recipes was working but the many recipes only ever maxed out at one single ingredient now it might not be obvious from looking at this and this is the fixed code but this is where the problem was I had mentioned before if you look at the pattern that I'm using here that this is technically the offset and this is actually the page size what I had in my code before was this I was saying go to offset zero take a page size of one and go get me the things that match this recipe definition ID and this was totally due to a copy and paste error so I feel a little bit silly about that but the impact of that was that I was only ever pulling back one ingredient because I said cap the page size at one so the fix was quite literally this rate here by making in.max value and the interesting thing is that if I made this change no test broke which tells me no tests were actually covering this which is exactly why it escaped in the first place now this is a good point to pause and think about what I was saying earlier about my test coverage I did walk us through this example from before where we had this get recipe API test and actually pull back a specific recipe this particular route will in fact go exercise that code however none of the assertions actually make sure that there are more than one ingredient and if we double check here we can actually see that we're ensuring that we have at least one ingredient that's why it says not empty but it never checked for more than one ingredient the really funny part is that after I made the change if I go back here and I put int.max value I said okay I'm gonna go make sure that I can have multiple ingredients pulled back and I want to verify that in my test so if I go back to the test I actually did try making this look something like this code here that uses a method that I have that just to kind of extends some X unit that does some greater than checks while I made this change here and I corrected the code now this test was failing it was still saying that nothing was coming back with more than one ingredient and I said that's impossible I know that I actually fixed the page sizing issue why is this test not passing and that leads us to one more interesting thing about writing functional tests with dummy data I'm going to show you some of the code that runs as part of that Builder pattern that I have that seeds some of the data into my test data for running tests and this particular one is going to be responsible for populating the recipes that I have in my test data what was interesting about that is that and this is a bit of a spoiler alert this code wasn't actually here here and we had a single food that was getting pulled off of this again you can see this pattern with the offset and the page size this was like this and we had a single food coming off so not two and then we had this here which this is going to be hard for folks to read and I don't expect you to know the structure of my uh my record types here but I essentially had the ingredients here with one single ingredient so of course my test was still going to fail even though I had fixed the bug all of my test data being generated only ever had one ingredient in it and that means that even if I did go write those tests early on and try to make sure that I could go assert the exact ingredients coming back from my recipes I would still be testing that there is this exact one ingredient so the lesson here is that my test data also needs to be more diverse so that I can cover more scenarios and that way I can write tests on those scenarios as well if I undo all the changes that I just did on there this is actually what the new setup looks like and actually has two ingredients I use some different units because I told myself hey maybe in the future you're going to want to look at some different units here as well and that way I have a little bit more variety being generated in my test data okay so now that I've made a change to the test data being set up I've actually fixed the bug in the repository and I could go back and change that functional test that actually hits the route I said well I'm still not satisfied because I caught myself in a trap and I committed a testing crime so so what's the punishment for the testing crime well it's going to write the proper tests in this particular case I decided I'm still going to stick with some functional tests but I'm actually going to write functional tests over the recipe definition food repository so that means I am going to be explicitly exercising this repository with some functional tests and not just at the route level where I have a lot less control over what's going on you'll see that I still use this Builder pattern and now that I actually seeded some dummy recipe data with multiple ingredients I knew that I could go write the needed tests to ensure that it's doing what's expected if I scroll down a little bit more we can see what some of my tests look like and you can see that I have looking up with a single ID and we're going to get multiple ingredients back and looking up by multiple IDs and getting multiple ingredients back as well so if I expand this we can see some similar patterns that we saw earlier in this video where I am going to go look up two recipes that will exist in the data set and then when I actually go do the lookup for those recipes so I'm passing in that collection of IDs here which should be two because this assertion says so I'm expecting that I get two results back now the results are per recipe that we're looking up and each result per recipe is going to have the set of ingredients that are part of that recipe I told myself it's not just enough to make sure that we got the two recipes back because the bug was actually that each recipe only had a single ingredient you can see right here I have this dedicated assertion that says we are looking for more than one ingredient and then I'm just doing a quick sanity check on every ingredient that comes back that it's actually matching the recipe definition ID and that way if some of my grouping at the SQL level was actually a little bit messed up and while I was looking up multiple recipes I wasn't actually accidentally going to be mixing the ingredients cross the recipes this part of the test here will prove that for me and there we have it that was the testing crime that I committed and just to quickly summarize what happened I had written some functional tests on purpose that would actually go exercise a lot of code but not assert a lot of the details this was so that I could get enough confidence to keep iterating but I did end up catching myself off guard by introducing a bug that was not caught even though that code was being exercised and that's because it was not being asserted on I also mentioned that it's really important that when you're simulating some test data that you're actually making it representative of the real scenario that you're trying to look at again otherwise this might give you some false confidence depending on what you're testing and what you're verifying and finally my punishment for committing this testing crime was actually going to write proper tests that would assert this particular situation so I hope that you found this insightful please remember to write tests along the way it's okay if you're going to have some escapes in some of your test coverage but remember remember you need to go back and pay the penalty for not having the right test coverage so go add those tests if you mess up thanks for watching happy testing and we'll see you next time