Microsoft Testing Platform CHANGES EVERYTHING for Your Tests!

In this video, we're going to have an intro to the Microsoft testing platform. My name is Nick Cosantino and I'm a principal software engineering manager at Microsoft. This is going to be the first video in a set of videos and part of a playlist where I'm going to be talking about Microsoft testing platform and then comparing and contrasting some of the different testing frameworks that can leverage Microsoft Testing Platform. I figured that it would be a good start and really important to look at what Microsoft testing platform is so we can have a better understanding of that before diving into some of the specifics. And because if you're like me and you didn't really realize that this was happening behind the scenes, there's a lot of really cool stuff that's come up with Microsoft testing platform. And for me, that's probably going to mean that I'm starting to shift which testing frameworks I'm using based on the information that I've seen presented. So with that said, let's jump over to this Microsoft Learn article, walk through it. We'll talk about some of the details. And I just want to be transparent in that in this video, we're not going to be writing code specifically because I want this to be an intro video to what Microsoft testing platform is. In the other videos that will follow in this playlist, we will be looking at more code examples comparing and contrasting. So just a quick note on that. So if you do head over to the Microsoft learn website, you can see that in the leftnav there is an entire testing section. There's testing platforms and you can see Microsoft testing platform in the left nav and we're going to walk through this overview. There's a lot of really interesting information here. So I just wanted to kind of start by clarifying what Microsoft testing platform actually is. Right? So we can see it's a lightweight and portable alternative to VS test. And for those of you that have written tests in C and worked in Visual Studio, probably familiar with VS Test over the last x number of years. This is really what we've been using and we're familiar with. But the important part is the Microsoft testing platform is embedded directly in your test projects and there's no other app dependencies. This is super cool for us. And if you're familiar with running those other types of tests, right, we have VS test console. You'd be running .NET test to go run the tests and we don't have to do that anymore. So we'll see more details of that coming up. It is open source as you can see here. Then you can actually check out the code on GitHub. So, if you want to go see what it looks like, it's all available for you to check out. Now, there's some core things that the developers wanted to make sure they had in the Microsoft testing platform. So, we're going to read through what all of these are down here cuz I think that there's a lot of things that if you're glancing over this, you might not really realize, but there's some powerful things that we should call out. So, um they did try to maintain compatibility as much as possible. So, I think that that's really cool that if you're familiar with doing things a particular way, Microsoft testing platform should for the most part allow you to start using the new way without breaking all of your old stuff and you can sort of transition over to that. So, for me personally, I'm going to be leveraging that kind of thing so that I don't have to go start everything from scratch, throw out all my test, nothing like that, but I can start migrating over. Um, I think the first thing on this list is this word determinism. And if you are someone who's writing tests, I think if you have a lot of really basic simple tests that the idea of determinism might not even cross your mind because you're just like, you know, I wrote a calculator and I'm writing unit tests on it and obviously it's going to be deterministic because I put number in, I get number out. Like by definition, it should be deterministic, right? I think that one of the things that comes up though is when you start having more complicated tests or you have more complications or complexity with your, you know, continuous integration, continuous deployment pipelines, you start to have scenarios where all of a sudden things look pretty similar in your dev environment, maybe slightly different in CI/CD and all of a sudden you get different testing behavior and that kind of thing. So they're saying that ensuring that running the same test in different contexts, so local continuous integration will produce the same result, right? The new runtime does not rely on reflection or other dynamic .NET runtime features to coordinate a test run. And we'll see this kind of theme coming up throughout the rest of this, right? But not relying on reflection, we'll see a lot of stuff that they're saying, hey, this happens at compile time. Now, I am personally someone that enjoys using reflection. I'm very comfortable with it. But it does mean that you have things that happen at runtime. Like by definition, using reflection means that we're inspecting different types and other information at runtime, which means you don't have the benefit of being able to figure out issues or any differences ahead of time at compile time. So if we can move stuff closer and closer to being at compile time, we can catch stuff sooner, which I think is really powerful. Runtime transparency. So the test runtime does not interfere with the test framework code. It does not create isolated context like app domain or assembly load context and does not use reflection or custom assembly resolvers. So again this part here not using reflection or custom assembly resolvers. I think that's uh tying into the same points that I was just making. So I think that like I said you will see more of this right the next point compile time registration of extensions extensions such as test frameworks and in out of process extensions are registered during compile time to ensure determinism to facilitate detection of inconsistency. So this point really gives us this point up top. So that's how it's being done. Zero dependencies. I think more and more people are at least from my perspective when I'm like looking at social media or developers talking about how they're building things. Dependencies come up more and more where people are concerned about it. Whether that's package references, whether that's just maintaining dependencies in a more complex application. Zero dependencies. The core of the platform is a single net assembly. So it's Microsoft testing platform which has no dependencies other than supported runtimes. So that's nice and contained in one spot. Hostable the test runtime can be hosted in anynet application while a console application is commonly used to run tests. You can create a test application in any type of net application. Interesting, right? So they're saying this allows you to run the test within special context. So what would those be? Devices, browsers where there may be limitations. So if you needed to run a set of tests in a different context, then you are able to do that. And I think that's really coming down to the fact that they've moved or sort of isolated all the testing logic into one assembly and then have they have sort of a different way that they're invoking the tests versus just having reflection go do it for you. Um support all the net form factors. Support current and future net form factors including native AOT. This is becoming more and more talked about. So I think that's really important. I think when you start going down this direction, you're not getting a lot of reflection support. Oh man. Oh, she ran away. My dog just came in and she's covered in mud. I know you're happy, but you shouldn't be covered in mud. You got to go. Okay, you got to go. No one's going to be upset with you. Performant. Finding the right balance between features and extension points to avoid bloating the runtime with nonfunddamental code. This is another thing that I think is really common with any type of packages that we're used to using as developers where we want to get some type of functionality and as things are getting built up over time it's like yes one more feature one more capability and over time things start to bloat. So I think that if they're keeping this in mind from the beginning where they are saying like hey look we want just enough in terms of the core functionality that they're trying to keep that in mind and hopefully this is something that they can stay on top of if they've architected and designed things with that in mind from the beginning hopefully that's something that can stay consistent. So allows you to configure the test process host observe the test process and consume information from the test framework within the test host process. So again, just enough single module deploy hostability features enable a single module deploy model where a single compilation result can be used to support all extensibility points both out of process and in process without the need to ship different executable modules. I think a lot of interesting points here. I think my primary takeaway that I'm seeing here is I really like the idea that things are moving away from reflection. Even as someone that likes reflection, I think from a determinism perspective, I think from being able to have consistency that moving away from reflection is going to be a really powerful thing when we start looking at more of these implementations. So, I just wanted to call out that in this playlist that I'll be creating. So, again, these are future videos. It's not all in one video. I need to break this stuff up. What we're going to be doing is going through this list of supported test frameworks. I will be starting with MS test. I've played around with this a little bit. I'm super excited. Historically, I use XUnit. I've been using XUnit for like since I was born, basically, right? So, XUnit is my go-to. I've never even considered switching anything away from XUnit. And honestly, I tried MS test the other day and went, "Oh man, maybe I have a reason to move over." The reason for that is that and we'll see this right but XUnit historically I go to add you know test project I add the reference to XUnit then I need to add the runner so I'm basically adding a bunch of stuff to be able to use XUnit but I'm super comfortable using XUnit like I said I've been using it for ages but when I got to see like how I can just get up and running with MS test one package you know start labeling my tests and stuff and it just works immediately so I'm very excited. I have not committed to anything but I suspect that I will be slowly transitioning to just using MS test. In follow-up videos, we will be going through code examples for all of these. I will be doing individual videos on each of them. Then I will try doing comparison and contrast of different features. So for example, we'll see like how you can get set up with MS test, how you can get set up with nunit, xunit, tunit. But then later we'll talk about things like parameterized tests like what does that look like? So we will be focusing on the uh the Microsoft testing framework part of all of this in the beginning and then we will start to branch off into the different feature sets to compare and contrast against the testing frameworks not necessarily exactly specific for MS test and or uh the Microsoft testing platform. Let's keep going a little bit. I think that there's one more spot that I wanted to call out, which was comparing with VS Test because I think this is uh sort of what most people are used to, right? As we're going through this, we were just talking about all of the Microsoft testing platform stuff. Here's why they're doing it, all that kind of thing. But what does that mean when we're contrasting with what we're used to already? Tests are executed in different ways depending on the runner. Okay, so if we're using VS test, this is what ships with Visual Studio, the NET SDK in a standalone tool. VS test uses a runner executable to run tests. And if you've run tests and you've had your task manager up, you've probably seen VS test console executable, right? So it's like the hosting process for your tests and you can see that through running .NET test. If you're not running necessarily in Visual Studio, but running it on the command line. This point here, execute Microsoft testing platform test. We kind of saw on the previous page, right, that the Microsoft testing platform is embedded directly in your test project. We don't need to ship other executables, which is really cool, right? You don't need to have anything extra to go run your test. You literally just run the executable. It runs the test for you. So, I think that that's a really nice value ad. Again, when I was talking about XUnit, just in order to have stuff uh get recognized in Visual Studio for the runner, I have to add stuff. If I want to go run it on, you know, my command line, I have to make sure that I have stuff available. With Microsoft testing platform, we just run the executable that's made that has our tests and it will work. But let's look a little bit more specifically about this. This page, in my opinion, is a little bit disorganized, but I think there's some good information here. For executables, VS test ships multiple executables. So, we already talked about VS test console. You have test host. You have the data collector. This is what we're used to, right? And this says MS test is embedded directly into your test project and doesn't ship other executables. the executable your test project compiles to is used to host all the testing tools. That's kind of what I was just saying. I feel like this little paragraph does a much more succinct job than some of the other points that are above. There is a path forward to migration and I think that this is uh in particular a lot more beneficial for folks with CI/CD pipelines where you're like, hey, we got this working, our tests are running for the most part, right? we're happy enough, but if we're going to start migrating, what the heck is that going to look like? So, in addition to the steps specific to your test framework, you need to update your test infrastructure to accommodate this, right? So, net test, this is the command line that we're used to running, but it says uh the options are divided into two categories, build related arguments and test related ones. The build arguments are passed a net build command and as such don't need to be updated for the new platform. So, that's good. Half the set of arguments we don't have to worry about. build related arguments are these ones listed below. So like I said, you know, you can check out the Microsoft learn page, navigate to here, try to figure this stuff out. If people have questions, comments about this kind of stuff and need video followup, I personally have not had to go deal with uh building out CI/CD pipelines with all this stuff in a while. But if people need help with it, we need some extra tutorial coverage, leave comments. I will try my best to try tackling specific scenarios with tutorials. The test related arguments are VS test specific and need to be transformed to match the new platform. So I think for most folks this is where we need to pay more attention in terms of migration. The following table shows the mapping between the two. Right? So there are going to be some things that aren't supported. I don't know if that means that in the future uh they will be coming. I suspect that there are some of these things that don't make sense that they're included, you know, rate built in because of the uh sort of pillars that they were talking about on the previous page. They want to have like just enough, right? They don't want to bloat things, but that's I think TBD. So, if you're using blame features in particular, I think that some of these are not supported at all, right? So, the d-lame is not supported. Blame crash collect always not supported. There are some things that they do have that seem to be uh supported with the you know the dash blame but uh seems like it's more specifically with crash dumps and uh taking dumps in general. So I would you know encourage you to go pay attention to these specific things where they're not supported. And then the other thing is that depending on data collectors, depending on loggers, depending on test frameworks, I think that you're going to be probably experiencing some gaps. And I think that's just something to pay attention to when it comes to these other things because they're not built into this uh Microsoft testing platform. That's going to mean like for something like a testing framework dependency, that testing framework is going to have to make sure that they're starting to build in support functionality for Microsoft testing platform. So that's why it is going to be dependent on those things. The test explorer, right? So this is for me a really big thing because I use Visual Studio. I like running my tests right in Visual Studio. I just want to rightclick, you know, run tests or go to the testing panel, run all my tests. So, when using Visual Studio or VS Code Test Explorer, you might need to enable the support for the new testing platform. In my experience so far with um using MS test with the Microsoft testing platform, this just happened automatically. I don't remember toggling anything. We'll make the video. We'll walk through it. I think for XUnit because I was poking around with the XUnit support for Microsoft testing platform. I think there's a couple of uh attributes that we have to set up, but again that will be covered in a follow-up video. So depending on what you're doing, you might have to tweak a couple of things, but so far I had MS test and XUnit easily showing up in the test explorer. I'm not too concerned about that. If that was going to be something that was broken for some reason and like you could only do this kind of stuff on the command line, that would be a deal breakaker for me. I just don't run my tests on the command line. I like having the UI for it. So, Visual Studio Test Explorer supports a new test platform starting with 17.14. So, this is going to be important, right? Like if you're on older versions and you're like, hey, it's not working, you know, start with the supported version. So, you might need to update VS Code. Test Explorer supports a new test platform starting with version X. Version X. Did someone forget to update a version number? I'm not sure. Um, Azure DevOps. Uh, you might need to update your pipeline to use the new test platform. So, I think that this page probably needs a little bit of uh more detail, but um I think that's an okay start. Like you might need to update your pipeline. Like thank you. That's uh helpful, but like to what and like under what conditions? Same thing when you're using VS test and Azure DevOps, you can replace it with net core task. So again, if you're switching over to using Microsoft testing platform, there's a bit of a migration path that you'll have to consider. But just a quick recap, right, the build arguments you don't have to worry about for .NET test and the testing ones. Yes. However, remember that you can in fact just run the test assembly with the Microsoft testing platform. So literally net run on the test assembly will run your tests. So, I think that that's pretty cool. Overall, I'm personally pretty excited to start switching over to Microsoft testing platform. This was the type of thing that I wasn't even aware that was happening. I think probably at the end of 2024 at some point on Twitter there was a conversation and someone uh from Microsoft I can't remember the individual they had commented on a tweet and they said something along the lines it might have even been Scott Hansselman I'm not sure but commented something about MS test and I was like why are we like who's talking about MS test right like why like we have xunit why is anyone talking about anything different even if people were talking about nunit I'm like why are you talking about NUnit, we have XUnit. And I remember at that point it was like, pay attention because there's going to be some changes. I'm like, okay, we'll see about that. And uh then I had someone from Microsoft reach out recently and and mention that, you know, all of this stuff was basically coming to fruition and I went, oh crap, I should really pay attention. So, I'm excited. I am personally thinking that I will move from XUnit to MS test in some of my projects because I like the idea of a lot of this being uh you know packaged more tightly and just the extra complexity and oddness around having the test runners and stuff. Just having it all in one spot is going to be great. So I think this will be cool. Like I said, we'll have some follow-up videos after this one where we're walking through the different testing frameworks, and that's actually going to give me an opportunity to work with TUnit, which if I scroll, uh, where was it? T-Unit here. Right. So, I haven't actually used TUNT. I know that some folks have already made a bunch of videos on TUnit, but I use XUnit. I will likely switch over to MS test. I'll make some videos on NUnit as well, and then TUnit will be an interesting learning for me to share with all of you. So hopefully you found this interesting. If you did, remember to leave a like on the video. Comment below if you're thinking about switching to a different testing framework or at least moving over to Microsoft testing platform. Would be interested to hear your thoughts. And just a reminder, subscribe to the channel for more videos like this. And as mentioned, I will have more videos for this series coming up soon. So check them out up here. Thanks, and I'll see you next time.