The Secret Trick To Keep Copilot On Track With Your C# Code

I find a lot of the time you can see a very drastic change in it staying on track. It's not to say that it can't get lucky and do a good job without it. And it's not to say that with a plan it does it perfectly despite what lots of people on the internet will tell you because they're all AI experts now. All right, folks. I'm not roleplaying as a pirate, I promise you. But we're going to be looking at Roslin analyzers inside of Visual Studio here. This is a continuation of a previous video which if you haven't checked out, I'll link it right up here. We're vibe coding Roslin analyzers to help keep our agents on track. So, it's a little bit meta because the whole reason that we're trying to put these together is because if our agents are doing things or they're not necessarily following some of the instructions and the prompts that we're putting together, we need to put some Roslin analyzers. This is just one example of something you can do to try and keep them back on track. The funny thing is that we're going to be vibe coding them. So, it's kind of interesting that we have to trust the agent to go do things. We have to get it right to put a rule in place to keep the agent from messing up in the future. If that's not super confusing, let's jump back over to Visual Studio here. I have since the last video updated these tests just slightly. And that's because the way C-Pilot put these together for us, it did actually what I would prefer. It basically had all of these theories put together and it was actually checking uh true and false. So it had the the condition for the expected value on the assertion passed in as a theory parameter as well. But what I wanted to focus on was assert true and assert false being forced to have messages. That's one of the coding standards that I like to have inside of my tests. And I just wanted to show you how that all works. But the way it wrote it originally, we don't get that unless we have assert true and assert false instead of assert equals. So, I split these tests up. So, the true tests are up here, the false ones are here, and then we have that continued throughout the rest of these tests. What is the class that we're testing? What are the tests do? Not super important for this video, but I wanted to show you why this is one of the things that I like to improve. So, I've just broken this test, right? So, I have 42. 42 is an even number. I've made it 43. 43 is definitely not even. And so if I go run this and we look in the test explorer, I just want to show you well spoiler alert before it finishes. I ran this earlier and uh you can see that we'll get something like this. In a really simple test like this, this isn't so bad because the actual test name is really is number even returns true. Right? Like there's almost nothing here. If this was a more complicated test that had a little bit more setup, it wasn't totally obvious. Some people will have a rule like only a single assertion or something in your test. I just feel like that's not representative of most code that I've seen. And personally, I don't write tests like that. I'm going to assert perhaps a couple of things. So when I see assert true and expected, sorry, expected true and we get false. It's like, well, what were you actually checking? We have this opportunity in XUnit to have a message here. I make it a standard in my code that we must always have a message. The tricky part is that works totally fine when I'm writing the code. it works less fine if I'm trying to remind AI to do that every single time it's writing tests. And the theme in this series that I'm doing is really that regardless of how hard you're trying to prompt, right? like you got it in your prompt, you got it in your co-pilot instructions or your agent MD file, you have it in your custom agent instructions, you have it in your documentation, you have it in the rest of all of your example code and still you're going to run into a situation where the LLM is just not listening and it basically leaves it without a message. So, we can use a Roslin analyzer to enforce this. And we're going to do this the same way we did the last one. We're going to vibe code it. If you haven't seen the last video, even though I linked it earlier, what we got out of that was an analyzer. So, I'll jump over to it. This is for having a range act assert comments filtered out for us. It's actually a compilation error. So, we got an analyzer and we even got tests associated with that analyzer. We're going to do the exact same thing here except we're going to do that for assert true and assert false. So, I'm going to go over to chat here. I have it in agent mode. I'm using GPT5 as the model. Um, I recommend that if you're trying things out and you're using like the latest model that's GBT, if you're going down a path and you're like, ah, it's not really giving me good results, try a different model out. So, if I just expand this, I like to go between GBT5 and what I have access to here for the latest sonnet models. So, 4.5. If I'm using cursor or something else, if I'm using Claude, like I like using the Opus models. Of course, I don't personally use Gemini models that much, not because I have anything against them, but usually if I'm flipping between GPT and like Claude, I'm doing pretty good and I feel like I have good results. So, let's try this out. I'm just going to speak into my mic and have the prompt get typed out for me. I would like you to create a new analyzer that enforces that our assert true and assert false exunit calls must always have a message parameter in them. That means that the analyzer will report a failure an error case when we simply have assert true or assert false with only the variable provided. We should add tests to cover these scenarios to demonstrate that the analyzer works as expected. And the message that should be provided is that we must have a helpful message in the assert true and assert false calls to ensure that the person or LLM that needs to fix the analyzer error puts in a helpful message. That was probably the wordiest prompt I've ever said. I feel like I was kind of repeating myself and going in circles. I'm just going to go back over it. I'm just going to space this out a little bit. So, I'd like to create Okay, I'm just going to kind of space out a little bit more so I can read it. And I only got one eye right now. So, we should add test to cover these scenarios, demonstrate the analyzer works as expected. I'm going to split that out. The message that should be provided when the analyzer reports an error, this is really confusing, is is that we must have a helpful message in the assert true helpful descriptive helpful/descriptive string in the assert true assert false call to ensure that the person I don't know if that's totally necessary. I think sometimes if you're just saying more and more stuff, it's like not necessarily helpful. I have this habit of doing that even when I speak, which is what you just saw, right? So sometimes if I'm doing this kind of thing, I like being able to speak out my prompts because I find that I do it more naturally, but especially when it comes to code, I need to refine things. So um in this case, like I put assert true and assert false like as expected. So that's kind of nice that it did that. But sometimes I if I have method names, uh it will write it like a sentence that looks kind of funny instead of an actual method name. So I think I feel pretty good about this. Um, that means we should get an analyzer out of this and we should get some test scenario. So, let's go ahead and run it. Like I said, it's in agent mode and I'm going to keep talking as co-pilot's doing its thing here. What's nice in some more recent versions, I guess this is in Visual Studio. It's probably in VS Code if we're seeing it in Visual Studio, uh, which sounds kind of funny to to say that cuz I would expect things to be the opposite way, but here we are. Um, we get this planning mode, which is kind of nice. So before it goes and executes things, it will put together a plan. This is something we see with GitHub C-Pilot. This is something that if you're using LLMs and agents to build things, if it doesn't do something like this, by default, you should be doing this for the agent. This makes a tremendous amount of difference when it has uh sort of like a step-by-step plan to execute. It makes honestly if you have time to go compare it um if you're using something that does not plan steps out like this, try it out and look at the difference between giving it a plan where it can go step by step versus just a description. If it's something that's very simple, a description might be totally fine, right? It was able to take my description and put it into a plan. It's usually not so bad. But if you're building more complicated features, I find a lot of the time you can see a very drastic change in it staying on track. It's not to say that it can't get lucky and do a good job without it. And it's not to say that with a plan it does it perfectly. Um, despite what lots of people on the internet will tell you because they're all AI experts now. Uh, I think we just have to be a little bit more pragmatic and realistic. I think it might be done. So I blabbed for long enough here. Says it will add a role analyzer. And now it says create the new analyzer. Add the tests. Update these tests to include descriptive messages. So, it already did this, which is cool. Um, I'm going to undo this and see like just on this file and see if it complains. And it does, right? It's pretty cool that we haven't even looked at the analyzer. We haven't even run the analyzer test to see if it passes. But we're already seeing on my screen right now. All the assert and uh assert true and assert false method calls are already complaining at us, right? What do they say? So if I hover over this, so it says vca2 provide a helpful/escriptive string message in the assert true call. And this says provide a helpful/escriptive string message in the assert false call. So for both of these, we ended up getting something helpful. what we have the opportunity to do. I don't think that it built it for us because um I didn't ask it to, but you might have seen this these little uh this little light bulb or sometimes if you do alt enter in Visual Studio, you'll get like a uh code fixup menu. So, you can actually build code fixup menus. What's really cool about that is you can basically say fix in just this spot. You can say fix in the file, the project or the solution. So when you add in an analyzer like this and you're like, "Oh crap, I have a thousand spots that do assert true and assert false, you can literally put the analyzer in place. You can have it vibe code a code fixup and then uh the reason I'm not showing that is you have to go restart Visual Studio. It's a bit of a pain in the butt. I can do that in another video." And what's nice is once you have that in place, instead of you going through manually fixing things up or asking C-Pilot, Claude, whatever to go through and fix all the compilation errors, you can literally use just the menu and have it go fix everything for you. So, I think that's super cool. But we do see that this works as expected. I'm just going to rebuild just to kind of show you the latest of what's happening, right? If I expand this a little bit. So, every single one of these spots says that it needs to have a helpful message. Let's see if uh C-Pilot can do something with this. Right? I purposefully wanted it to say put a helpful descriptive string here because in the past when I have done this in my own codebase because this is literally a similar analyzer to what I have at least the same behavior. Uh, I found that when the description or the the error message was not descriptive, what would happen is that the agent would go, "Oh, I need to have a message here. No problem." And it would say like the dumbest thing, something like assert true and it was false. And then the message would be like um expected true. [laughter] So it's like the agent is doing what I said, but it's also not what I want. So this is the thing, right? We have to give agents more context. We have to be thinking about them. Like if we had to communicate to someone and we were willing to leave out any detail, we should expect that at some point that person will not do what we expect. That's the unfortunate reality. At this point, maybe in the future, these things get so good that we don't have to worry about that and they just read our minds. But for now, that's not the case. So, the final part of this video, I'm just going to ask Co-Pilot to go fix this up. Let's try. We have compilation errors in the fun with numbers tests. It looks like all the assert true and assert false calls are failing to compile. Address the compilation errors to fix the tests so that we can run them again. Is that prompt excellent? Not really. But I think that we don't have to worry too much in this case. I think this should be a pretty simple lift. You can see right skipping planning because it's less than two files, less than uh 40 lines of code. So it has some conditions around that which is okay and it did it already. So it fix compilation errors by adding assertion messages. Align it with your analyzer requirements. It didn't go into detail about how descriptive it needs to be. But if we read right assert true expected input to be even. So that's pretty cool. This is funny because it added even more. It's it's the inverse right? But it just added a little bit more detail. Same thing on here. Expected the input to be prime. This is what I would hope to see, right? So, let me let me take this. I still have 43 left here from the beginning. If I go run this, this is what I mean by making a difference when with test readability. Again, I know these tests are super simple, right? We'll go to this failure. The message when this fails is expected 43 to be even. So, this is in my opinion significantly more helpful than like assert true got false fail. Um, yes, I know in this test it's named very simply. I get it. But hopefully you can see the benefit of doing this. We still haven't even looked at the analyzer though, right? So, assert message requirements analyzer. This is vibe coded by us, right? Like we didn't even look at this. This is the first time I'm opening up this code file for us to go through. I will make videos in the future about how to write analyzers and do more with them. But I've been very transparent in my prior videos and my live stream even even where I did this that I don't have a lot of experience building analyzers. I started by vibe coding them because I wanted to put more rules in place. So is this analyzer done very well? Probably not to be totally honest, but I don't have enough experience personally to go through and critique them. So, at this point, if I were to go through this, I might be looking for like, you know, uh, type symbol name, like, okay, do we want to do string comparisons like this? I don't really know. Is this an effective way to kind of like go through the expression tree and look for symbols? I don't really know. There's probably a whole lot more to be, you know, uh, investigated here and to understand better. But I have so far in my own personal projects made about 34 analyzers this way. and they've been guiding agents with higher degrees of accuracy in the work that I'm giving them. So, at this point, I'm not at a sort of a crossroads where I need to dive deeper, but I really want to, especially now that I have an excuse because I have analyzers in my codebase. I didn't before. So, I think that's all for this video. I think in the follow-up videos, we'll look at some code fix up. I think if you haven't seen that or you don't know what I'm referring to, that'll be pretty helpful. Maybe even this one will do it. Yeah. So you can see here this has its own fix up enable analyzer release tracking. So fix all occurrences in document project or solution. This is the kind of thing that we can go ask co-pilot to vibe code for us as well. So if you're interested in seeing that you can check it out in the next video. Thanks for watching. I'll see you next time. Take care.