BRACE YOURSELVES! We're Testing Implementation Details

everyone has really strong opinions about testing in software engineering including you should never test implementation details but I strongly disagree with this hi my name is Nick centino and I'm a principal software engineering manager at Microsoft I disagree with the statement that you should never test implementation details because I think testing and software engineering is all about building confidence and that means that you might be testing things in different ways depending on how you need to build confidence in this video I'm going to walk you through a previous video's code where I was looking at different ways we can handle exceptions and I'm going to illustrate to you different ways where we may want to test implementation details to build confidence a quick reminder that if you like this kind of content to subscribe to the channel and check out that pin comment for my courses on dome train let's head over to visual studio and look at the code in the previous videos I had an example set up where we had different ways where we might want to fail fast or catch exceptions such that we could retry or allow code to continue on when we're looking at this particular system it's pretty contrived but it has a piece of code that has a client that we want to call a login method on so I have that highlighted here on my screen and you can see that what we want to be able to do is have some retry logic wrapped around it now you could build other things like this like Paulie to give you that retry mechanism but I'm just kind of doing a really basic implementation here to show you and we can play with this as we go through the test that it's really just a for Loop and a TR catch to see if we can make this work on subsequent attempts in this particular case we're dealing with a client that we don't have control over I mean if you're paying close attention yeah that client is defined here in the left side in my solution but the point of this example was to illustrate that we have some thirdparty code interacting with some external service and we have no control over this what I want you to do is imagine that you're working in your software engineering job and you needed to be able to build in retry logic around something in this particular case it's a login API to an external service but let's pretend it's some other service that you might be dealing with and you need some retry logic or something else where you can make attempts again generally with this kind of thing if you needed to have some type of counter with retries a really common mistake that we can have is off by one errors right because attempts and retries are not the exact same thing generally you attempt one more time than you were retrying right so you could have zero retries and still make one attempt even though this is really simple it can still be very error prone even with an off by one kind of error so it's important that you're paying close attention to this kind of stuff now when we want to go look at this code and build confidence over it I mean you might be able to sit down and reason over it maybe you're going through a poll request with someone and they're reviewing your code they're checking it out and everyone feels good about it but in my opinion if you are trying to build confidence not only in the changes that you're rolling out but if you'd like to ensure that this Behavior does not break then you may want to consider adding tests over this particular logic someone may look at this and say well this is certainly an implementation detail if we want to go exercise this code here and that's because if you're looking at what this API might do this is really on a private method but even if you assume that we pulled this out onto a public method we do have a couple others that we'll be writing tests over and they will be exercising this but ignoring that this is private for now the implementation detail that we see here is some retry logic and in particular what we do when we catch the exception someone might say hey look if you're going to go write tests over this code you shouldn't be asserting anything in here because that's implementation details we just care about what the outcome is going to be when we pass in some inputs the reason that I disagree with saying you should never do that is that if we wanted to build confidence that the retry mechanism is working then we might want to go test some implementation details furthermore we have other people saying when it comes to mocking and tests and unit tests and things like that that mocking is bad it gives you false confidence and I wanted to pause here to talk about mocking as well because we're going to be using mocking heavily for these test examples now one of the challenges with mocking is that people will say when you're leveraging mocks you are sort of making up the behavior that the object is having and that's not going to necessarily represent the real world behavior in this case here we're going to be looking at writing tests over this login call on this client so this method login a sync that we have here we're going to be writing tests over this and we want to make sure when the login fails that we can basically handle it in particular ways I leave it to you to decide if you have other ways that you can go test a failing login API that you have no control over one way might be to give it wrong credentials so certainly that could result in something throwing exceptions if that's what your API was doing so that might be totally viable but what about other errors what happens if the network is down or the service isn't available that you're trying to communicate with how can you simulate that there a situ ation where if you're dealing with transient errors especially in big distributed systems you might want to have retry logic and if you're not really able to simulate that easily you're going to have a hard time writing tests over it in my opinion using mocks in this particular case can help out a lot and I'm not saying that this is the only way to do it but I want to head over to the test now and show you how we can go make sure that this code is doing what we want I'm going to start us off with what we have in the setup for our test here so I am going to be using mock so mq and xunit and these are the two things that I like to use together there's tons of other mocking tools that you can use and different test Frameworks aside from xunit so the concepts here are very much the same so you can use whatever you want to kind of get through this right so what we're doing here is creating a mock repository and this is the object that's going to allow us to create our other mocks that we can go Mock and we have those two right here sorry online 23 and 24 and then I'm going to make a config here and this config if we go check it out is going to have things like a username and password so that's what we're going to be authenticating with what I'm going to be showing here is I'm using goids for this but you can of course use something like bogus to go make fake data or just put in random strings that you don't care about but the point that we're going to be looking at here is that we don't actually care what the values are in the config we're just going to make sure that whatever they are set to that we're verifying that they're passed correctly to the calling site that's why these are goids it's not really going to matter and it shouldn't matter in our test next we're going to make the actual system that's under test in this case it's called NX system I have this all set up in the Constructor xunit will go run this Constructor before every test so that's why it's okay for me to do here even though we have multiple tests with that out of the way the first thing I want to look at is that we can retry when we throw exceptions in that method we were looking at and we are going to be exercising the method called do nice to have work with oth so you can see on line 91 that's the method we're calling and it is one of the public methods that we had on our class you can see right here and I'm going to be calling the get off token a sync so that's an internal implementation detail right and what I'm going through these you could imagine and I didn't do this for this video but you could imagine for basically everything we're going to be looking at for the most part that you could go pull this out and put it on its own dedicated class make it a public method because we are going to be focused on the logic inside here I'm kind of going two levels deep but that's kind of not the point I'm just being lazy and not trying to refactor this whole example building on the previous videos so it's going to be this method we're looking at so that's the one we're exercising when we go to look at our mocks you can see that I want to be able to setting up our cool client here with login async and like I was saying I'm not using those goids directly I'm basically just pulling them off of the config that we made and that's why when we go to look up here it doesn't matter what these are set to because I'm just going to take the Val directly off of them that way when we're using mock it's saying look I'm setting up login async but you must be using whatever username and password are on Nick's config doesn't matter what the actual values are it just matters that we're connecting them together properly and before continuing on just a quick note from this video sponsor which is packed publishing packed has lots of awesome net and C development books and in particular I wanted to talk about this one software architecture with C2 and net 8 this book you'll be able to learn about microservices how devops Works Entity framework core which is super popular of course and how you can integrate with Azure if you're interested in software architecture microservices and Azure this will be a great read for you you can check it out in the links Below in the description and the comments thanks and now back to the video when I set this up I'm going to simulate that we're throwing an exception this is why a mock is really powerful here I'm able to basically make this up this isn't a real service it's not actually calling a login method on some service I'm just using a mock and then simulating some behavior that would otherwise be hard to recreate and this test is actually pretty simple because under the assert portion here I'm just going to call verify on login async and check that we did it 3 times that's important because part of the confidence we're trying to build in this particular case is that our retry works the number of times that we expect just to illustrate this I'm going to change this to another number so we're going to put 333 here instead and when I go to run this test we should see that it's going to fail the other thing that I wanted to call out is that I am using mock repository verify all this is going to ensure that we don't have extra things set up so other mocks that we might have set up here that were never actually called so if we've set it up it must get called and that's what verify all is going to do for us so let's go ahead and run this one and if we check the output here we can see in the bottom right portion of my screen that we do have expected invocation on the mock exactly 333 times but it was only three times there's a good indicator that we have some coverage over the retry logic right so if I go to put this back to three like it was saying it's going to technically do it the proper number of times here if I go to run this test once again we do now have coverage over this bit of code that ensures that we have retry three times and to prove it if I go back into the code and we scroll down to the number of retries right remember I was saying off by one things are pretty easy easy to do here so maybe you call this attempts right this is a really easy thing to mess up if I go run this now we get an error right it's a expected invocation on the mock three times because we were saying hey we want to make it go three times but it's four times so you need to kind of rationalize what you're technically trying to do here and this is a good way to catch really simple errors again this is just illustrating that you can go test implementation details if you want to go build confidence over something there's nothing wrong with it you just have options when you're writing tests now that we've looked at a really simple case here I want to start building on this example and showing a little bit more complication and different scenarios that we can work through so what we saw here was that we basically call something three times it's not really fancy but we were simulating throwing an exception and then being able to handle that the proper number of times so from there well what happens if we want to look at logging or Telemetry right what happens if want to handle exceptions a little bit differently what I want to do at this point is say let's start with a different type of exception because I think this is a pretty important way to to look at this particular scenario with login there's one situation in particular that I think we should not retry that would be that if we tried to log in with the wrong credentials and we got an unauthorized access exception we probably shouldn't retry right so if we have something that might be transient we'd say hey let's go retry and I mean yeah in this particular case too I probably should have some type of delay or back off but that's what Paulie's for so you probably want to use something like that instead of building your own but let's say that we want to handle unauthorized access exceptions and when I say handle I mean let's catch them and then throw them again because what we don't want to do is retry if we get unauthorized access this has technically broken some expected Behavior we have if we go to throw this exception so I do have another test for that if we go look right here if we do throw an unauthorized access exception and we want to make sure that the code path completes because this method do nice to have work with oth is going to complete but we should not have had any retries because it was unauthorized access let's look at the setup that we have we're going to make sure that we call login async and we're going to throw the exception just like we saw in the first test this is going to be the method that we're calling and then here we're going to look at the number of times that we try to log in and it's just one time so let's go see if we run this what happens we can see that this test passed and it's a green check for us but what about the other test that we had if we go run that one now it also passes so that's good news it means that we do have test coverage in these two cases here that's great continuing to build on this set of examples here if we look at this test that we looked at originally that had the expected exception it was just some invalid operation exception if I jump back into the code I'm using an exception Handler that's just catching everything right it's catching everything well except this one here unauthorized access because we catch that one beforehand everything else though we're just catching and allowing to happen but our test doesn't actually check for that and it's only checking for this one type of exception I think that it's probably Overkill if you wanted to go guarantee that every other type of exception is going to be caught right that might be just a little bit too much but we could do something if you again wanted to build a a little bit more confidence instead of just having this one type of exception here we could change this from an xunit fact to a theory and have some other exceptions that we could try throwing just to make sure that no one accidentally hardcoded something like this right if they only did this and this again is kind of tying back into the previous video where I was saying sometimes people will suggest not using just a general exception Handler and trying to have very specific exceptions so I just trying to walk you through what that could look like here right so let's assume that we have this to start it should still pass so both of these should work great they still do let's go make this a theory okay so a theory in xunit is going to allow us to parameterize our test so I'm just passing in the expected exception and I have this member data on top of the method and Theory instead of fact but if we go and see what exceptions to retry test data looks like we can see that I have these three exceptions and you can could technically go add more to this list I already think three is totally fine but this just allows you to kind of do whatever you want to build confidence so I still have invalid operation exception but I had a couple of others that you might expect to see if you were calling some web API maybe there's a timeout maybe there's some other type of exception when the Network's down or something like that right you might want to go simulate what those look like and make sure that your code on the handling side is able to do what you expect because again we can't actually control this system in real life if we don't own it but we might be able to mock it out and see how our system responds on our side let's go run these tests now and see what happens and if we have a close look here we can see that our Theory did not pass we actually had one of the three pass and that's because invalid operation exception was the only one that we're handling now right so if I go put this back to make sure it is one of these Pokemon handlers because we got to catch them all right if I go run this test now and put this back there we go we get green lights again the idea with this is I'm not saying hey this is the best idea to do all of the time I'm just letting you know that you could use this and have better protection if you're not sure what other types of exceptions that login async could throw in this case and then you're able to properly test that behavior again this is all implementation details right you can go add other exceptions into here to make sure that your part of the system is responding as expected the last part I want to talk about is logging and if we go through some of this code and if you watch the first video on this you'll know that I left some comments in here saying hey you might want to do logging or Telemetry because if you are going to be catching exceptions and just continuing on you might want to have some insights within your system about when things aren't going as expected let's say there were some transient errors when we were trying to use cool client and connect out to some external service we might want to know if there's statis ially more errors coming up when people are trying to log in and again maybe we don't care about the unauthorized access exception part but we are saying hey look I do kind of care if maybe there's some transient network issues that might be telling you about some other type of issue or maybe this service is overloaded and it's not responding properly so maybe we want to build in better resiliency on our side we do want to put some you know back off policy in place when we're retrying so there's all sorts of reasons why you might want to do that logging into Telemetry could help you with that so in this case I'm not going to have Telemetry I'm just going to add in some very simple logging and what I'm going to do is just call log error with the exception and then we're going to say fail to get the O token now I want to back up a little bit here because after working at Microsoft with big distributed systems where we have tons of things running in the data centers and on top of that working in a digital forensics company where we have law enforcement government agencies people running searches on hard drives in their labs and we don't have any access to them it's really important for us to have confidence that we have the right logging and Telemetry in place I want to call out that for some people they just don't care you might be one of those people you might say Nick I'm just logging because it's extra helpful if something happens to come up and that's cool no worries what I want to show you here is that you could really care about logging in Telemetry because that's a really core part about how you're going to debug and basically work with technical support to make sure that you can make things better I'm thinking at Microsoft dealing with the deployment teams dealing with the current team that I'm on now for routing we really do care when things aren't going as expected if you're on call or working with service Engineers you bet they're going to be looking at logs to try and see what's up it's important that you're designing features and making sure that you have logging and Telemetry in place and you may want to go test it for proof and have confidence that it's working I've added this one simple log error line in and people are going to say hey look this is a perfect example of tests for implementation details are brittle and you bet they are in this particular case our test broke because we changed the expected Behavior I do want to make sure that we can go have logging that works as we expect so when we go to do this the problem that we're encountering is that we're not doing the mock for the logger now let's go set that up and this means that we're writing tests that are checking for specific behavior so what I've done here is I've added the logger setup for log error and it's going to check that we're getting this particular exception and this message you might not care about the message maybe in your case you're saying I just want to make sure that the exception is coming through sometimes people don't care about this stuff at all like I said but if you do if you're someone like me We logging in Telemetry is important for some applications this is going to be helpful again we run the test they all pass because we are setting up log error on the logger you could go one step further and you could check to make sure that we're logging it the right number of times this is extra important if you're being careful about which exceptions we're logging if I go back and I guess when we're logging them too and to illustrate what I mean by that so we did just check to make sure that log error was called three times what you may want to look at is back in here maybe what you want to do and I'm not saying this is good or bad it's up to you to think about how you want to do it maybe you just want to track the last exception you might retry X number of times and you're like look I don't care what happened along the way but the last thing that happened I just want to track that everything else is noise to me I don't know up to you it's not my system you might have different design choices if you wanted to do that if I go back here you might say that you want to make sure that it's logging an error exactly once right so you could do anything you want to there to make it a little bit more robust and suitable for your situation something else that you might care about is if we consider the unauthorized exception path again this is one of the scenarios that we looked at earlier we were saying when we have an unauthorized access exception from Bad credentials that might be a scenario where we don't want to retry right and this is the test where we had one attempt we only tried to log in once because we get this particular exception you may have restrictions in place where if you're doing something like unauthorized access and you want to make sure that you're not potentially logging something bad so say username and password are wrong and you're logging with the password or the username or something like that depending on the environment you're working in you definitely don't want to log someone's password but maybe the username or something is user identifiable and that way you don't want to include it in any logs for that matter what you could do is get extra save here and you could say that when this happens you genuinely want to assert I know this says verify here but this is technically performing an assert that something has not happened I know it's using a different word but conceptually that's what it's doing you could say hey look I want to explicitly state that we're not logging anything for errors technically the strict mock enforces that for us already but if you wanted to be verbose about what you're trying to cover in the test this would also work for you so I've just run it again and when we throw this particular exception we're guaranteeing that we're not logging any errors and something else that you'll notice is that in here I'm using it do is any generally inside of setups for mocks like here I don't like using it do as any and this is going to be the last little part of the video for testing implementation details and one final scenario where this is really important let's assume that I didn't have this here and I wanted to use it do as any because we were saying hey remember what Nick said at the beginning of this video we don't actually care what's being passed in here right we really put goids up here so Nick was saying he doesn't care and that's partially right but I do care what's being passed into this method because I do want it to be exactly whatever these values were if you were to set up a strict mock with it is any and this is different from the verify call and I'll get to that at the end if we were to set this up if I go run this now it should pass here's something that you might want to cover so if I go back into the code and we see where this is used here maybe you're doing some testing or you're doing I don't know something else and let's say that you hardcoded in your username and password into here so now if we go to run our test right here if I go run test we will see right unauthorized access exception this one is still passing and it shouldn't it shouldn't pass because we put the wrong things in there you don't want to have these hardcoded values in here and if we go to put this back to what it should be so again testing implementation details this is inside the method right I put this part back but I still left the hardcoded values in if we go to run this test it should fail this should help us catch bugs for the implementation details we have so I think that's important again let's go put this back and again if we go run this now it should be corrected you may have situations where you're doing functional or integration tests and you can guarantee that because you're working with the external service or something and it's set up in an environment you can genuinely kind of pipe through your proper credentials or configurations and you're like hey that's communicating with the service that's great but in this particular case if log in a sync on this cool client with is not something we could call reliably in a test environment because we don't control that service this is a situation or you may want to consider mocking this kind of stuff out and yes those are implementation details the final part that I just wanted to wrap up with was on this verify call you will see it is any here and this is just for the exact opposite reasoning from the strict mock at the end here I do have situations where I just said look I want to make sure login async is only called one time it would wouldn't hurt if I just put in the same things the username and password and the cancellation token directly but what I'm doing here is saying I only want it to be called one time and I don't care who called it with what as long as it only happened one time because when you couple this verify with this logic up here this is ensuring that you have the right parameters passed in and this is ensuring that in total it was only called once at and that's the same idea with the logging part here for log error I'm saying hey look I don't want to check to make sure specific messages were only put in once I just want to make sure that we never logged anything in error and you could do it with log warning cuz I have that on the logger as well this is really just about how specific you want to be with your tests again because you want to build confidence in a particular way so in this video we went through a handful of different ways where it may make a lot of sense for you in your situations to be able to test implementation details what I went through were some cases where we don't have control over a third party system I had a login example and we wanted to put retries over it so we needed to be able to simulate that that login was failing but we needed to simulate different types of errors that could come back from that we also looked at something like logging right logging or Telemetry you can kind of use them interchangeably for this example that meant that if you had very specific situations that you wanted to log or not maybe certain data that you do or do not want to show you could go write tests on the implementation details to make sure that that's not happening I also wanted to call out that yes in these particular examples that we look through I was looking uh more specifically at a method that was inside of like basically it was a private method so you'd have to do a public method call to the private one I know that we're really focused on that private one yes we could have you know pulled that out into a public method on another class that would have been accept able as well but we were still looking truly at implementation details and not just you know testing against the API could we still write tests like that absolutely would they offer value absolutely but to say that these types of tests offer no value I just walked you through a handful of situations where they absolutely have in situations that I've done professionally thank you so much for watching and if you want to see some interesting stuff with building plugins in asp.net core because that's super cool you can check that out here thanks and I'll see you next time