WHO WINS? - C# Head to Head With Unit Testing VS Functional Testing

So, think about it, right? You're an upand cominging software developer. You've been told that you need to write tests and you know it, you believe it, you see the value in it. But the problem is that you're trying to get better at implementing them. You start watching tutorials about how to unit test things. And then the more you start reading about that, the more you start realizing that people are saying, "Hey, you should be functionally testing things. Drop the mocks. That's not real. You need to have real systems working together to test them properly." But then you run into some situations where this doesn't work. So, you're back to the unit test with the mocks and all of a sudden you're going, I actually don't know which way is the best way forward. Well, spoiler alert in this video, I'm going to show you that both of them are important and you should be using both of them in different scenarios. Now, before I kick this off, just a quick friendly reminder to subscribe to my free software engineering newsletter that I send out every week. I will link to that in the pin comments below. Okay, so the first thing that I think is really important is to get some definitions out of the way. And this is one of the biggest complications when people are sharing information online with their really strong opinions. Now, for this video and effectively any content that I put out there, when I talk about unit testing, I'm talking about being able to test a single function. And I don't mean that you can't call private functions or anything else like that, but effectively, if you have a class with a public method on it, you're testing that method. If that happens to call some other private methods and things like that, those are going to get exercised as well. But the point is that you're covering that method. It will also mean in a unit test that from my perspective you're not touching any external systems. That means no internet, no hard drive and ideally if you have other dependencies they are through interfaces that are mocked out. And personally if you have other mechanisms to be able to mock without using interfaces I mean do what you got to do but the point is that you're not interacting with the real dependency because we need to be able to control that for a unit test. My unit tests that I write are brittle. That sounds really funny, but when you go changing the code that's inside of the method that you're testing, your unit test should break. And that's because they are testing the internals of your function. And if you're changing the code inside that function, the unit tests are going to break. So you might be thinking, well, that sounds really stupid. Why would you ever write those? We'll see. So functional tests, on the other hand, are looking at your system as a black box. So they are going to be testing the function and not the internals of it. So think about it. You have an input or a condition that you're setting up. You run your code and then you're looking at some output or looking at some state at the end. These aren't as brittle. The idea is that if you're changing the code inside and aiming to keep the functionality, then a functional test is perfect for you because if you have the inputs and you can verify the outputs, that should be the same before and after. Now, because you don't have full control over the internals in your functional test, you are losing out on a little bit of control and granularity. And that will mean that when a functional test fails, it might not be exactly obvious where the test is failing versus a unit test where you'll probably notice that a mock is not being executed properly. So you have a verify call or an assert call failing. So these are my working definitions of unit tests and functional tests. Again, very quickly, a unit test is going to be using mocks to be able to isolate the systems that we're working with so that we can focus on our system under test. And functional tests will treat that as a black box. And that means we're going to be using real systems connected in there. We don't care about the internals of the code. We're just going to be exercising it. So my goal in this video is to show you that both of these types of tests and other tests that you can think up will have a time and a place because they're just tools to help you build confidence in your ability to deliver software. And if you're not convinced, let's go jump over to this code in Visual Studio and I'll try and prove it to you. Okay, so I have a little program on my screen right now in Visual Studio. We just have a class. It's being created called Nick's cool system to test. This is the one that we're going to be spending all of our time looking at. But I just wanted to show you the calling code. And all that we're going to be doing is asking the system to get some titles. I'll explain that in a moment. And then we're going to print out all the titles. So this is the calling code, but we are going to focus on this class in particular in order to be able to cover it with tests. All right, I have the class pulled up on screen. And some of the formatting that you see here might be a little bit weird, but I'm just trying to get it all on the screen for you to see. So, what we have is this method that I said get titles async. And we can see that we're creating an HTTP client here. This is not necessarily what I would recommend doing, but we're just doing this to start off this demo. From there, we're going to be asking that HTTP client to get a string that is the feed of my website. And from there, we're going to be trying to extract all of the titles that we see between these HTML elements. So the core part of what we're doing is going to be downloading that feed. And then when we go to look at this code, we have a loop that's going to be looking for the start index. So that's going to be the first part of the title element that we're looking for. And if we do manage to find that, let's go find an end element. And again, if we do have that, we're going to pull out the title by getting the text in between. So we're going to use a substring and then use the indices that we just found. Now, this is a bit of a special treat for later that we'll come back to, but for now, what we're going to be doing is adding every single title that we come across into this collection, ensuring that we set this index properly and then returning all the titles. So, at this current point in time, we don't have any test coverage on this class. So, what should we do? Should we go write unit tests on this or should we go write functional tests? Well, a functional test would be pretty easy for us because all that we need to do is be able to call this method on an instance of this class. there's no dependencies that we have to set up. So we'd call this method and then just go check the list of the titles that come back. But one of the challenges that we face is that I mean this is hitting the internet. This is going to go out to the internet, hit my website and get the feed. And if I'm publishing blog posts at the rate that I currently am, this test is probably going to fail for you at some point if you're looking for specific blog names. So the titles aren't going to be consistent. So if you're thinking about running this test in any type of automation, it's going to be problematic if the titles are fixed. And the other thing is that it requires internet connection and that might be totally fine in your environment that you're working in. So I don't want to rule that out as hey look you can never have internet for your tests but I do want to call out that that's going to be a requirement to be able to write a functional test for this. If you're running this on build systems or VMs or other environments that cannot guarantee an internet connection then you're going to have flaky tests if the internet is ever down or you can't actually access the internet. Okay. So we could write a functional test, but if we wanted to unit test this, how would we do that? Well, I was saying that my definition of a unit test is that we cannot go out to the internet. So if we wanted to be able to refactor this code, we'd have to go look at pulling out this HTTP client and putting it behind an interface or some other way that we could inject the behavior to control the HTTP client. If we have that control and we're going to write a unit test, that means we can return the string that we're interested in from hitting this URL without ever having to go to the internet. All right, so I did say that I was going to prove to you that both unit tests and functional tests have a time and a place and that you need to be using both of them. But in this particular scenario, it seems pretty crappy to use either of them. The functional test means we have to be hitting the internet and we have to expect that my data on my blog is changing. So that's going to be kind of problematic to test with. and the unit test that we'd want to set up. We have to go refactor this code to pull out the HTTP client part. And that means if we have to refactor this code and we don't have any functional tests in place, we technically risk breaking this. So neither option is super stellar here. But I do think the path forward that's least terrible is that we could start with a functional test, right? We could say, let's go write the functional test, prove the code is working. And that might be something in your own development you're like, look, I'm going to try it. We don't have to check it in. But in order for me to refactor this properly, I need something to test with. So I'm going to go ahead and do that. And then we're going to refactor the code to pull out the HTTP client part and have an interface that we can pass in. From there, we're going to add unit tests. And once we have those unit tests, we can blow away that functional test because we don't want to have to rely on a test that's hitting the internet and a test that's going to have a data source that's changing that we don't control. Let's go see how that works. All right. So on my screen right now, I have a functional test that is set up with arrange act and assert. And this is a functional test still because we haven't extracted anything to do with the HTTP client. We're simply just going to be asking that class to get the titles asynchronously and then check them out. But I haven't gone ahead and fixed up these asserts yet. And I just wanted to explain why this test is already a little bit complicated and weird. I've confirmed that currently there's 12 titles that come back off the RSS feed. And I know that at the beginning of this list, I have a handful of them that have dev leader in them. However, at some point, the titles start changing because those are going to be the titles of the blog articles themselves. The other title elements that we end up getting just have dev leader inside of them. So, in order to speed up this process, I'm just going to sample a handful of them. And I'm going to do it something like this where I go ahead and comment these out just to make this go a little bit quicker. All right, I've gone ahead and populated some sample data for this test. And this is an interesting stopping point because I always say that tests for me are a way of building confidence. And because we're just trying to put together a quick functional test in this case to allow us to refactor, in my opinion, this is going to get me the confidence that I need. I just want to sample a couple of the titles instead of populating every single one of them. And in my opinion, if I grab a couple from the beginning that have Dev Leader, a single title that is of a blog right at the beginning, and then one from the end of that list of 12, I feel pretty content with that. If I have this in place, I feel that I should be able to go pull out that HTTP client with a quick refactor and then add the unit test after. A quick run of this test shows that we get a green check mark. So, this is good enough for us to start looking at refactoring. Now, I've gone ahead at this point and I've created an HTTP client factory that I'm now injecting into this class via the constructor. But, we had to do a little bit of extra work here. And this is a good call out because when we're thinking about writing unit testable code, the way I've defined it at least, this means that we need to be able to have things that we fully control when it comes to dependencies. Now, if I scroll up and we check out the HTTP client stuff, you're going to see that it was a little bit of extra work and there's two classes and two interfaces. Starting from the top, we have this IHTTP client factory. This is going to allow us to create a client with a name. And the only reason I've left this in place is just because the built-in one that we have from net allows us to create one with the name as well. So we're not using in this example. I just wanted to keep it consistent with what you probably see already. From there, I've gone ahead and created this dummy HTTP client factory because we're not using the built-in net stuff. So you can see that it's inheriting from the interface that I've created. And then the method that we're using here is to create a client with a wrapper. And the reason that we need to create a wrapper is because we need to control that HTTP client that comes back. If we simply just returned the HTTP client itself, it would mean that we don't have control over it. This one is still going to go out to the internet. With this wrapper, though, if we scroll down, because it implements this interface I HTTP client, which I've simply stripped down to have this single method that we use by doing this, we can now mock out this behavior. And again, I'm just illustrating that this is how I end up having to create unit tests when I'm dealing with systems that have concrete dependencies on resources like the internet or going out to disk. If you're thinking, "Hey, that's a ton of extra work. I don't like that." I hear you. It's a lot of extra work. I totally agree. But this gives us the isolation to write those tests and be able to fully control results that come back from things like get string. Now, before I go write the unit test, let's double check that our functional tests are passing. Of course, we have to make a quick change to our functional test to be able to now pass in that factory. So, I'm passing in a real HTTP client factory. I'm not using a mock in this case. This is the real system that I would be using in the application. If I go ahead and run this, we still get a nice little green check mark. So, awesome news. Our stuff is refactored and it's passing the functional test, which means let's go ahead and write those unit tests to make sure we're getting coverage without having to go out to the internet. All right, let's step through the unit test that I've created for this class now. So the difference that we're going to see in this versus the functional test is that there is a little bit more setup code here. And this is because we're going to be using mocks to control the input and output that we want to have when we're calling our external dependencies. And our external dependencies in this case are going to be the HTTP client factory as well as the HTTP client itself. So, we need to be able to set up the create client call that's going to give us that client back and then set up a call to dev leader to get the feed for my website. And then I'm able to fully control the text that comes back. This means we never have to hit the internet. We have full isolation like I would expect to have in a unit test. And all I've done is created some text that I expect that our class will be able to work and operate on. Therefore, I can check what's returned. Another really cool thing that we can do with unit tests is guarantee that we are calling dispose as we expect. In my opinion, I think this can be super helpful for catching bugs. Next, we create the system that we want to be testing. We call the method to get the titles and then check for the three that we ended up creating. At the end, we just ask our mock repository to verify all of the setups that we had up here. With this green check mark and the test now passing, that means that we have unit test coverage on our newly refactored system. Okay, now that we have the unit tests in place, technically we can go ahead and blow away those functional tests because we were saying that we probably don't want to have internet connectivity required for our tests. And realistically, the way that this system works, my blog is going to be changing and these tests aren't going to be working and passing forever. You'd have to be constantly updating them unless I give up on writing articles. And I'm going to pause for a moment because this is a good opportunity to mention that I do have a course on refactoring that's available on Dome Train. So, if you're interested in seeing more examples where I'm able to extract code, make it more testable, perhaps you want to make code more readable, more maintainable, or even walk through a really big example that shows the strangler fig approach, which is really cool for bigger, more complex systems, then I highly recommend checking it out. It's available on dome train right now. And that would be super helpful for this part of the video coming up because we're going to talk about what happens when we want to go refactor this. Now, we have this system that we just did a little bit of refactoring for. We made it unit testable. Got rid of the functional test that was temporary for us. But now we want to go back and refactor it. The focus of this video, right, is comparing unit test and functional test. Which one's better? I'm here to prove to you that both are needed. So let's go see what happens. We have unit tests in place. Let's go refactor this code. So now that we've pulled out that HTTP client factory and we're using it in here, the refactor that I want to look at is pulling out all of this code here into something a little bit more generic. This code is supposed to be going through the different HTML elements to look for titles. But technically, we could pull this code out, refactor it, and have it be a little bit more generic. Instead of just looking for only titles, maybe we want to repurpose this code to look for different HTML elements. So, what I would like to do is go create an HTML element finder. So, I'm going to go ahead and pull this out and do a little refactoring to make this work. So now that this code has been pulled out and refactored into this HTML element finder class, you can see that I'm passing that into the constructor here. And then when we look at get titles async, we're asking that HTML element finder to get all of those element values. The only difference now is that we're passing in the type of the element that we're looking for, which is title. And if I scroll down to see the implementation of this class, effectively all I've done is lifted the code out. And instead of using constants in here, we're able to just have any marker name that we're looking for. The rest of this code is the same. And one more refactor that we're going to look at doing here is pulling out this HTTP client as well as the part that gets the string. I'm going to make a dedicated class that's just the dev leader feed fetcher. All right, with both of these classes pulled out, now we can see that I've changed Nick's cool system to test to pass in both of these classes that are newly extracted. If we look at the method here, we can see that we're just calling these new classes. But if we want to go see where that code moved to, we can see that this dev leader feed fetcher class is effectively just the code that was directly pulled out with a bit of dependency injection that we needed to set it up. And if we go to the HTML element finder, it's down here below. So awesome. We've successfully refactored this class. So let's go run our unit test to prove that our refactor worked. But one slight problem. It looks like our tests aren't compiling anymore. But that's okay, right? We just made two new dependencies we can pass in instead. But once we're passing in these two new instances, you can see that I have to go ahead and pass in the HTTP client factory here as well. And to prove our refactor was successful, I can go ahead and run these tests and see if they all pass. And a green check mark for us means that we've successfully refactored this even further and still have unit tests in place. But there's one more thing I want to call out about this, and that's that we now have this system that we're trying to test, as well as two other dependencies that we're passing in here. So, this is yet again another really good opportunity to pause and think about the purpose of our unit tests. It's true right now the unit tests are absolutely passing. We have three systems technically that we're testing and there's only three because we pulled out two from the original one to make three. But if the goal of unit test is to be testing things in isolation, we're no longer doing that. This test that we have is effectively a functional test except that we're mocking the dependencies that are external. Now, is there anything truly wrong with this? I mean, realistically, if you wrote this test and you found that it was giving you the confidence that you need, I'd say leave it. I'd say this is perfect. It's doing its job if it gives you the confidence you need. But what I find really interesting about this test now is that it's a bit of a hybrid of a unit test and a functional test. Another thing that I want to mention is that we got a little bit lucky with this because I ended up structuring the code such that it had dependency injection the whole way through. And because we refactored it in this order, it allowed this unit test to pass. If instead we would have refactored it in a different order and these two new classes that we made did not have dependency injection for the external systems, we'd be pretty screwed here. And what I mean by that is the entire unit test that we had written no longer would apply. We basically have to go delete all of the code in the unit test and now go look at testing what's left. I'm going to jump back to the code, but I'm not going to waste your time showing you all of the tests that we could go write. I just want to show you the different spots that we could go test and what those tests might look like. So again, this is our bit of a hybrid test that's left over from our original unit test. But if I jump back to the original code that we were looking at and trying to test the function here, get titles async. If we wanted to unit test this, now truly the only things that we would want to do are test these lines of code. The definition of unit test that I had is that I'm not interacting with the other dependencies that are embedded. And that means this class here on line 47, dev leader, feed fetcher, as well as HTML element finder for me as a unit test, I would want to go mock out the behavior of these. The reality is for the current state of this code that if we needed to go mock these things out, it's really not going to offer us a lot of value. These two classes that we're calling into are extremely simple right now. And we've already seen that I have a unit test covering the whole thing. Now what that would mean is that every place that's going to be using dev leader feed fetcher is going to have to set up the mocks in a similar way. So you are going to be having a bit of extra redundant code for setting up mocks when instead if we were just looking at truly unit testing what's in here. What we're trying to prove is that we're calling this method and getting a string back and that we're taking this feed content the result of this method and passing it into this other method here. It's really that simple. The unit test for this would be a couple of setups and just proving that you're passing variables between a couple of function calls. It almost makes the unit tests over this part of the code have very little value at all. And that's because we moved the logic out. But what we could do instead is look at the other spots that we pulled out and consider writing tests on these. Again, this method here looks incredibly simple. There's not a ton of logic happening. But what's interesting and what might be worth testing is the fact that we're using this HTTP client. Now, I know that there's proper patterns for this HTTP client. This is probably a bad example with the using statement here. However, the point that I want to get here is that if you wanted to prove that you had using somewhere in your code, having something like a unit test and mocking that out could be really helpful for ensuring that you're disposing things the way you expect. Otherwise, in this method, there's not a whole lot of extra value for writing unit tests, but we could. Functional tests would not be a great idea because you would be going out to the internet. That's why we did this in the first place and had this dependency passed in. And if we go look at the HTML element finder, this one's interesting because do we write unit tests on this or functional tests? There aren't really any dependencies that we pass into this and therefore there's nothing that we really want to have control over on the inside of this method to prove that it's going down different logical branches. We can control that completely by the parameters that we pass into it. So I think by my definition, this would be a unit test, but that's only because we don't have to consider any dependencies in the first place. Now, if you were paying attention towards the beginning of this video, you probably saw this block of code, and this is just a bit of a bonus round here. If we had unit tests that were covering this and we had some scenarios where we were passing in different content and marker names and looking for the things that we expect to validate that, what we could do if we want to go fix this is we could go write a failing test first. We could go write a test that has dev leader passed in and that we are expecting it to be removed from the results. That test would be failing initially and we're not going to go commit that. But we'd write that test first and then we would come back here and implement the fix which in this case is really this simple where we're just uncommenting this except when you have a bit of an embarrassing demo situation and you have a typo here and you also rename this to something else. And even more embarrassing when you realize that you have two S's and you've been recording this whole video with those two S's. But anyway, it's that simple. We just have to implement this and then once that's in place, that failing test that we talked about would now be passing. So, this is an example of some type of fix that you could go create and a unit test that we would have on this code would not necessarily break. All right, bringing it all back together. Which one wins? The unit test or the functional test? Well, I stand by my original answer. Both. Both are very useful for different situations. Unit tests are really valuable for white box testing. If you're trying to test the internals of what's happening in your function, you can get a lot of value out of unit tests. The trade-off is that you have tests that are brittle. If you have to go refactor code and the internals that you're touching are changing, odds are your unit test is probably going to be busted. This is especially the case if you have systems that you're controlling with mocks. Conversely, the functional test really stood out for allowing us to refactor. We could change the entire guts of the method and as long as the behavior was as we expect, the functional test will pass. So, the main takeaway that I want you to have from this video is that when people tell you a certain type of test has no value or a certain type of test is always the best way, they're wrong. And they're wrong because these different test types are just tools. They're tools to help build confidence for us to deliver changes in our software. Use the right tool for your situation. So, I hope that you found this video insightful and got some takeaway about how you can leverage both unit tests and functional tests. And if you thought the refactoring in this was interesting and you're thinking, hey, next time I want to go refactor code, what's the best way for me to go approach this? Well, if you want to figure out how you can go address and schedule your tech debt and refactoring, you can watch this video next. Thanks, and we'll see you next time.