How To Vibe Code Rules To Keep Your Coding Agents On Track

That's weird. It wants me to reload, but I can't reload. [laughter] In this video, we're going to be looking at vibe coding some Roslin analyzers. And the scenario is going to be a very simple, maybe trivial one, but hopefully the idea gets across that you can take this approach and build your own Roslin analyzers to do more fancy things. The reason I'm making this video is because spending more and more time building things with AI, I'm running into all sorts of fun issues where the AI agents are doing things that I didn't really want despite all of my attempts to prompt and Roslin analyzers are helping me move things in the right direction. So, in this video, we're going to look at doing a Roslin analyzer for arrange act assert comments. Now, it's pretty standard that when we're writing tests, they follow a pattern that is like you arrange or set up things for your test. You act so you exercise the system under test and then you assert. So, you're looking at the results and then trying to make sure that you're asserting the behavior that you expected. That's all fine and well except I am personally offended. Maybe not offended, but I don't like when I have a range act assert comments across every single test. I know that it's following a range act assert. I don't need comments for it. It drives me nuts. Is this the best use of time to make an analyzer? No. But I think it should be pretty straightforward and we can try it out. One of the biggest challenges that we run into with working with agents is despite all of our attempts to have, you know, all these instructions and things in our prompts. It's just not going to be bulletproof. So, what I'm going to do is walk through a bit of code that I have, I've taken the liberty of putting a couple of things together. So, in Visual Studio, I'm going to jump over to this console app that I have. It has a really silly class. It's fun with numbers. I have a bunch of boolean based methods on here. They don't really matter what the implementations are, but the reason I made them boolean based is because in a follow-up video, we're going to build another analyzer. We're going to vibe code it, and hopefully we can see something more advanced. And the whole idea with this series is that we can just keep putting more and more analyzers together to do more helpful things for us. So we have a bunch of these methods here. And then if I jump over to these tests, I actually had C-Pilot. If you look in this chat that I have here, I have these tests that were put together by Copilot. We vibe coded them before recording here. So I had some compilation issues and stuff like that. So basically just had it go fix things up for me. But it did write these tests and it did put arrange act assert in here. In this case, I didn't have a co-pilot instructions file and I didn't say in the prompt don't put the comments. But I can tell you with a very high degree of confidence, I have literally seen AI agents where I say don't put a range act assert comments in the prompt in the co-pilot instructions and my documentation and it still puts a range act comments. Drives me nuts. We have this as an example. The tests all run and pass. So, we're in a good spot. So, what I'd like to do is have it vibe code the Roslin analyzer. Hopefully, it gets it right the first time. If not, that's cool. We'll iterate on it a little bit. And then from there, I am going to make sure that we can have co-pilot rewrite all these tests. And hopefully, if we follow the same idea for a prompt like follow a range act assert, right? If I go up to the top here based on the test that we have like in a new test project. Oh, I maybe said that wrong in my original prompt, but anyway, write test it following the arrange act assert format. I'm not going to tell it anything about the comments. I'm hoping that it will try to build it and it will run into the analyzer issues and then correct itself. So, let's go ahead and vibe code this. I'm just going to speak into my mic the prompt here. I'm going to put it to agent mode before I miss that. I would like you to add a new Roslin analyzer project that will create an analyzer rule that looks for arrange act assert comments in tests. The idea with this analyzer is that it will look for these comments and mark them as error so that developers including humans and LLMs are able to get flagged when they leave such comments and that way they're forced to remove them in order to compile. Please add a new Roslin analyzer test project and a Roslin analyzer to support this. We do not need a code fixup implementation. Strictly just the analyzer. I'm going to leave that as a prompt. I added that last little bit in compared to previous attempts at doing this because when you're dealing with analyzers, you've probably seen in Visual Studio when you hover over something, it's like it'll give you the shortcut to fix it. You can absolutely vibe code and create those things as well. It's just other levels of complication that I don't want to do in this video necessarily. So, we're going to stick to the analyzer, see if we can get this working. Fingers crossed. So, we'll let this finish up. The tricky thing with making LLM based videos is like it might not do anything that we want. Then, you're going to watch me fumble through this to try and get it back on track because truly analyzers are not really a strong suit of mine. This is a very new thing for me. That's weird. It wants me to reload, but I can't reload. [laughter] I think what's happening is that it's building it, but it's not part of the solution. So, this dialogue, if I press reload, it's going to kill off um co-pilot. So, let me reload. Keep and continue. There we go. Nice. And I didn't tell it to do this. And compared to the last time I tried this, it added tests. I wanted to talk about tests for this after to show you that if you're not getting the expected behavior, you can either keep vibe coding it and like highlight some code just as an example. So you could say rightclick co-pilot actions add to chat addit your chat and say hey the analyzer is picking this up or it's not please add a test scenario for this and make sure that it passes or you know whatever you want but if you can read these tests right these are pretty simple tests I mean it's kind of ugly all the code on one line but you can see there's arrange act assert comments some other comment here which is fine although this is kind of weird I don't I don't understand how that would pass [laughter] also like we didn't see maybe the comments are if we use a uh different comment blocks maybe that you know is different so we can check all that out but that's the idea behind having tests if I go back over to the analyzer itself we do have an analyzer here like I said I am not the most you know proficient at writing analyzers that's why I like vibe coding them we do have some compilation errors though so let me just Rebuild. Cool. I just wanted to make sure that it wasn't stale. So, I'm going to tell it. Right. So, there was a bit of a challenge there because if we tried to reload the solution, it was going to kill off copilot, but we needed to reload it or else it wasn't going to work properly. So, let's try the analyzer projects that you've created are not compiling properly. Please fix the compilation errors. I'll go do that. Yes, I did say please to an LLM. I think that was just natural. Um, I'm trying to be pleasant when recording a video. Usually I'm very frustrated in talking to LLM. So you get to see the the bright side. So hopefully I can fix this up. What do we got going on here? So the other thing is like we have to wait for this to pass. I think what I'll do in this video too is I will try to make sure that the analyzer tests are passing first. Once the analyzer tests are passing, then we'll go check to make sure that our analyzer is picking things up properly in our console test. Okay, cool. So that says it's passing. I'm going to try rebuilding for succeeded. So interesting. The reason I'm saying interesting is when I rebuilt everything technically what should happen. [laughter] Technically these should fail because the analyzer should be getting picked up. I just wanted to check if our project it's not obvious cuz I haven't explained it but this here just put it on multiple lines. Uh we need to make sure that the analyzer is referenced in the project where we want it to run. So I do have that here. So, let's go run these tests. That's these ones. Oh, is it not going to work? Oi. Okay. So, again, things are vibe coded. So, from here, I want to put this to net 10. I'm just going to take over the stuff that I see from my other test one because I was having a lot of fun with this earlier. I feel like a lot of the time with LLMs, especially if you're going from the older XUnit to the newer one, does a lot of weird stuff. It seems like it's following patterns that used to work and now they don't. Oh, we have to do this as well. Output type is not library or string. I want it to be exe. We'll get that going. So, again, like I didn't tell it xunit v3, but it picked a different xunit than I already had in the solution. Why? I don't know. Clearly, it didn't try to run the tests because they all fail. That was a hiccup. Nice. Cool. Let's get that closed off. Let's get that closed off. all of your tests in and then I'm going to put this here are failing when they're run. Please fix all of your tests because we cannot confirm that the analyzer is working properly if all of the tests are failing. So this is sort of the you know it's a good example of when you're trying to like vibe code things if you don't actually understand what's happening you in my opinion waste a lot of time. If I was really familiar with this analyzer and I was familiar with the test for it and it's not working, I could probably more quickly go figure out or I could give it more context. It's almost like I'm paying as little attention as possible to keep going. So, is it going to write good code? Like, I don't know. It's saying it's passing, right? Actually, it didn't say that it ran it. It just says I fixed the analyzer test. Are you running the analyzer tests? because when I run the analyzer test, every single one of them fails. Let's try that now. Um, is that helpful? Not really. I should be giving it this text here, but I want to see it run the tests. And I still don't think that it's running them. And to be fair, I didn't tell it to, right? I asked if it I said, "Are you running the tests?" I am an AI programming assistant. I can't run your test runner. Okay, let's ask it. Why can you not run the test from the command line? using Microsoft andnet tools. Amazing. So, this is very bizarre. [laughter] This is what I mean like uh recording videos like this. This is it's ridiculous that it's saying that it doesn't have tools to invoke a testr runner. For full transparency, I do most if not all of my coding [laughter] sitting right here working in Visual Studio and having Copilot run tests for me. So, this is pretty silly unfortunately. And I think what I might need to do is pause, copy and paste over what is mostly a working analyzer at least and then we can see on the test. But let's go put this in here. This is what I mean like with when you're vibe coding if you don't actually know and I genuinely do not have enough experience with analyzers to be able to go explain them in detail. So, the actual test setup is not so bad because it's like you give it example code and then tell it to go run the analyzer on it. Okay, I still don't think that this is right. Passes now. Like you have to go do this for all of them. This is kind of annoying, but not what I wanted to do in a YouTube video, but hey, this is what vibe coding looks like if you actually don't know what you're doing. It's interesting. It can build it on the command line, but it won't run the tests. It just doesn't make sense to me. Okay, one more. The issue will still be if the actual test project the console app tests where I want to see things come up if that's not actually triggering the failures. So let's find out get these tests to work. Okay, green light across all of them. That's lovely. But if we go rebuild this now still succeeds, right? So the point is that the analyzer is not being applied to this because arrange act and assert these should all have errors. We don't even have warnings for them. There's warnings for some other things. I will tell it that the analyzer tests are passing now. But the fun with numbers tests should be failing to compile because they have arrange act and assert comments. The analyzer does not seem to be picking up the arrange act assert comments and causing an error for the Visual Studio compiler. So let's go run this again. Right? This is again one more example of when I look at this like why don't I know why it's not being picked up in my mind like the test pass uh are we missing an editor config do we have something else we needed to toggle if you see on my screen as I'm blabbing away here it already figured out what the issue is I don't know what the issue is so I'm very curious ah this was left over so I didn't even see this when I was talking to all of you earlier, right? I reformatted this line to show you, but this is left over from a previous attempt at recording this before I made this video. I had the wrong project name. Now we have the right one. All that it did, well, I've accepted the change, but all that it did was it put the right name for the analyzer. And now you can see that arrange, act, and assert all have red squiggies, right? So if I try to build this now, we get 24 errors. So all of these comments are now being picked up by a Roslin analyzer. Okay, here's what I would like to do. Now we have our analyzer working. We have our tests here. I'm going to delete all of these tests. Okay, I'm going to go back to that's not the right one. I want this. This is a previous conversation. So write tests following. I'm just taking this part here. I just want to keep it kind of true to like how simple it was before. So I'm putting this in here. Write test following the arrange act assert format for the different public methods that we have on fun with numbers and go. So let's see what it does. I'm curious to see if we'll watch it put arrange act assert in the comments then try to build and then realize that it has to change it. This will be very embarrassing if what it decides to do is either unload or suppress [laughter] the messages, but let's find out. Okay, arrange act assert. It put all of them in, right? We don't want that. Is it going to try to build? I hope it didn't just delete all of the Ah, no, it got the comments. It didn't delete the tests. Good. I added the tests using a range assert structure without the band comments. This is perfect. Um, this is we got there, right? Uh, this is exactly what I want to see because we didn't stop it from going down that path, right? I think originally if you're working with your co-pilot instructions or having better prompts, you can kind of like you can and should try to put this kind of thing into your prompt, your instructions, your custom agent, wherever to give it, you know, more context, the right direction to go down. that might have prevented it from even trying to write a range act assert. The whole meta point with all of this in this series that I'm trying to do is basically illustrate no matter how many guard rails you try to put in place up front, agents can and still will go off the rails. They will do that. So if they are going to go off the rails, how do you put the if you've been bowling before, right? You put the bumpers in right in the in the gutters, right? We want to put we want to put the the bumpers in the gutter so the ball doesn't go into the gutter. It's the same thing with these agents. We want to keep them from going off into the gutter. Putting something like a Roslin analyzer like this means that even when it screws up because inevitably it will deviate from what you want. Can we bring it back on track? So that was the idea behind this one. A pretty simple analyzer completely vibecoded. I didn't even show you the analyzer code because it doesn't really matter. Right. The I mean I want to learn and I will learn more about analyzers so I can make videos on this. But we have these tests that assert the behavior. If you're content with that then great. Like if it's missing scenarios like we could briefly go look at one. What if I did this? Is that allowed? It's not right. That's great. But I don't think we have that in our analyzer tests. And I think the reason we don't have that I mean we don't seem to need it because in the actual analyzer itself it is purely looking for I mean you it is looking for block comments right there. One of the previous implementations of this where I was scanning through it. It was using a slightly different approach not just this and not looking for this either. It was doing something different but you know I can say fun with numbers test right let me I'm going to just show you add to chat. I notice that we don't have analyzer tests specifically for using a forward slashstar comment. This is correctly picked up by the analyzer right now. So we do have the expected behavior, but we do not have a dedicated test for these block style comments for a range act assert. Please add test coverage to our analyzer for block comments for a range act assert. do not change the implementation of the analyzer itself because it is working properly. So, we're just going to fire this one off to kind of round out the video. And just want to remind you that as I'm doing this, the follow-up video that I'm going to do on this is um and this made it kind of awkward to go do. So, it made all of these use assert equal. I was hoping that we would get some examples where we had assert true and assert false. So, I'm going to have to change these, but I want to show you that we can have assert true and assert false. And I want to enforce that when you use either of those, you must have a message in place for your assertion. The reason I like that is because if I'm looking at test failures in a test output somewhere, I like to understand why without having to go read code if possible. So, usually you still have to read the code, but it's helpful. So, it fixed this. Great. That's not what I asked for, but I wanted just examples where we have block comments and it added a whole bunch. I'm assuming that it ran them, but it told us earlier I can't run tests. So, let's try. And of course, [laughter] they all fail. I'm going to say all of the block comment tests fail. Here is example failure. So again, this is I'm not some I'm not recommending that you code this way. [laughter] I do think that you should spend the time understanding why that happens. I didn't even read what the failure is. I think it's because it's getting some other type of diagnost Oh, it's actually the the offsets are wrong. My bad. I thought it was going to be something else. But let's see. Earlier it was saying that it had something else unexpected in the message, but yeah. So, I think it has some extra information or sorry, we're expecting that things are a little off, right? 61 64. It's kind of interesting. So, I'm going to give it to two more. And then I'm going to put one here. And then I'm going to copy and paste the other one. And what's nice is as this is going to do it, I've already accepted it, but I wanted to show you that we can go look at the changes it's making. But I'm pretty sure it's just going, "Oh crap, I have the wrong offsets. which is fine, right? It's changing 60 to 63 and something to 64. Let's go run that. Great. So, there we go. This is literally us vibe coding an analyzer to look for special comments that we don't want. We have tests for the analyzer. We were able to say we have test cases missing from the analyzer and have those go get added. And we ran into a scenario where C-Pilot, for some reason, couldn't even run the tests. Ultimately, that created a lot more work for us to go navigate here. Kind of a pain in the butt to have to go copy and paste test output. But this is just an example of like we've really vibe coded the crap out of this without really understanding what we made. Um, I do not recommend building large amounts of software like this. Truly, the reason for that is like I now have this analyzer where if this stop working how I expected, right? I write more code and it's like highlighting weird stuff. I'm going to have to keep vibe coding my way out of it, maybe I can get lucky, but I think it would ultimately serve me much better to understand like what is this code actually doing? I don't have enough familiarity to know like what is all of this doing here? I can, you know, quickly tell these are the blocked phrases. Okay, simple. Do I understand these diagnostic descriptors enough? I'm not sure. You know, can I rationalize what this code is doing at a high level? Perhaps, but I don't I don't know this code well. And so, just a word of caution. It's not that you can't do things this way, but I wouldn't go off the rails too far yourself either, uh, so you don't lose sight of, uh, what's in your codebase. So, that's a wrap for this one. In the next video, we will look at vibe coding, assert true and assert false, having forced messages. Thanks for watching. A vibe code away. I'll see you next time.