Multi-Agent Architecture in .NET: Microsoft Agent Framework Tutorial

In this video, we're going to look at Needler and Microsoft agent framework, which is the extension or the successor to semantic kernel. So, in this video, we're going to see how we can use reflection and source generation patterns in needler to be able to set up agents as well as orchestration. And this will be a very introductory video. I just want to show you some of the things that I've been playing with when it comes to building things with AI, working with source generators, dependency injection, Microsoft agent framework, all this fun stuff. And I'll put more videos together just like this one to try and link these concepts. So, let's jump over to Visual Studio and get started. The top of my screen, I just have a bunch of code and it's kind of boring, but this is to get your configuration set up. And then I'm going to be using the Azure Open AI client. So that means that I have my models and stuff deployed in for use uh in Azure. And I think I'm just going to be using one of the GPT 40 mini models or something just small and cheap to be able to run this. But this is really configuration setup and then setting up the chat client. And both of these things ideally I will end up making some type of helper methods. I already have one for configuration, but I'll do another one for the chat client because honestly I really don't love looking at this kind of stuff. And as I've been building out Neler, which if you're not familiar with that's my dependency injection sort of scanning utility library that I've been putting together, I end up building something that will wrap this right so that it looks like a oneliner or we get it automatically. Something nice so that we don't have all this ugly code at our entry point. Now we need the configuration to pull in our key and our deployment information and you can see that is happening in here. So if we look at getting the Azure OpenAI section from the configuration all that I'm doing is pulling in which endpoint it is my API key and the deployment name. So if you just wanted a shortcut when you're playing around with this stuff sure just drop it right in here. But that's not something you're going to want to commit and that's not a practice that you want to get used to. So just just a heads up. Mine is coming from configuration, but that's all that this code does up here. Now, if we go a little bit lower here, we're going to start to see some of the needler specific things. And again, needler is what I have built to do dependency scanning for myself. So, it ends up registering things onto the dependency container. And that way, I can resolve all the dependencies I want for my application without manually registering them. That's one of the patterns that I like. It's definitely opinionated. It's not for everyone, but that's what we're looking at here. So, the first pattern that I'm showing you right here is using reflection. And Needler does support reflection and source generation to call out what is going on here. This syringe is sort of like a a fluent builder pattern. And I'm saying that I want a syringe that uses reflection because it's either going to be source generation or reflection because when you get into source generation and you want things that are purely source generated for things like AOT and you don't want reflection because of things like trimming, I wanted to make sure that they were very separate. So we're using source or using reflection in this case. And then I have this extra little bit here which you might be wondering why I need a funny method like this. And that's because I have purposefully set this up to be plug-in based, which is a little extra layer on top of all this. I'm kind of mixing a bunch of different things here just to show you things I've been playing around with. You'll notice that this is matching assemblies that only contain this part. And if we look over in my solution explorer on the left hand side here, you can see that I have my simple agent framework app. And then I have the simple agent framework app.agents project. So, I have all of these other agents, like the actual things that we're going to be working with pulled out into another uh assembly. And that way, just to kind of prove it to you, if you look at the lines right here where I start resolving things, I didn't register them anywhere explicitly, right? So, needler is the one that will go look for those things and automatically register them. Now I want to explain a little bit about what I was building in Needler specifically for Microsoft agent framework and of course what you can do with Microsoft agent framework. If you're familiar with semantic kernel and maybe if I take one step back maybe you're not even familiar with semantic kernel that's cool too. If I start with semantic kernel semantic kernel was a solution where you could start to have a little bit more layering on top of this really sort of straightforward chat client. Okay. So with this chat client, of course, you could send messages to the LLM and it would respond. But with semantic kernel, we could build this layer on top of that. And that layer allowed us to register things like tools. And that way when you're having this conversation with your LLM, it can have tools that are written in C that it starts to be able to call as part of the conversation. So now you have this power where you can have C code that is callable by your agent. And that's pretty awesome. and then Microsoft agent framework there's a lot more to it but the sort of initial step that I see right on top of semantic kernel like as the next extension is that we can start to think about orchestration of agents so it's no longer this idea of talking with an agent and even calling tools it's all of that plus more agents and how they work together with that said when we start getting into this part of the code or we're starting to resolve our uh dependencies off of this container. Like what is the workflow factory? What is the agent factory? And these are things that I have from Neler specifically. But let's go see like what one of these agents is and how it works and all of that fun stuff. If I go a little bit lower, I'm just going to uncomment some code here. But you can see that we have this ability to go take this agent factory and create an agent. But again, what is an agent and what is this factory? Let's go see. So, I mentioned that my agents are defined in this other project. So, I'm going to jump over to this triage agent cuz that's what we were looking at. And in Needler, what I was able to come up with is that I can start defining agents really just by source generating the whole thing. And it's pretty cool in my opinion because if you look at how Microsoft agent framework works, you can start creating these agents. you can start passing in or defining I suppose the class with all the methods and you have to annotate everything. Instead of doing that I just said why don't I basically annotate once kind of like what I have up here and then using source generation I can just go make the rest of the class. So that was the idea that I had when trying to mix needler with Microsoft agent framework. So for a triage agent, instead of me defining what this needs to do, right? And you could you could still go write more code inside of here. There's nothing stopping you from doing that, but at a high level, if you just want an agent defined to be able to work with Microsoft agent framework using Needler, you just have to start giving it the parameters that you want. And really, it's mostly just prompts, right? So, I have a description up here and then the instructions are really the prompt that you wanted to be able to work with. And then we have a few more things that are slightly more advanced, but the function types. And then if we're talking about orchestration, right, if we're chaining these things together and we have agents that can work together, this is where we have this agents hands off to and then another agent. So that way in a group conversation, if you think about these agents working together and talking together, this basically defines this triage agent that is able to hand off to these other agents. And here are the conditions. If the triage agent is having a conversation and it needs to be able to ask a question regarding this, then it will go say, "Oh, I need to go send that off to the geography agent." So it kind of gives guidance to the agents for how they should be using and interacting with the others. And of course you may have more complicated situations where perhaps a you know straightforward constant at compile time kind of little prompt like this is just not sufficient for you and that's totally cool. But I figured for needler I wanted to do some source generation for some just really simple scenario so that the boiler plate gets reduced to like really just defining your promps and that's essentially what I've been able to come up with here. So this triage agent as you can see like it has these attributes defined on it and what I'm going to do is go back to this program.cs file. You'll see that I have this agent factory. Right? So, I took this agent factory off of our dependency container. And you'll notice that I have this create triage agent. That's kind of weird, right? Because I just showed you back here that I have a triage agent. There is literally no code inside of it yet because it is source generated. But if I go back to the program.cs, where is this coming from? Right? How did I get create triage agent on this agent factory? Right? I only showed you that I have the type defined. And again, this is because of source generation. So I'm able to essentially go scan for things that have uh I what was the attribute called? Uh needler AI agent. So if you have this attribute for your agent, I'm able to scan for them at source generation time. So at compile time and then I can go essentially tack on these other methods. So these extension methods on agent factory. And to prove it, if I jump into here, you can see that this, well, it might be hard to see, but in the very top right here, it says that this is a generated file. So, all of this code inside of here is generated. None of this existed from me writing it out. And all that we're doing is giving a nice little extension to be able to get a named version of this. It's really just like some syntactic sugar so that you don't have to have the type pulled out. And I did this again just to try and make things a little bit more readable. Um, you can see that it's also source generating the doc comments and things like that. But this pattern again like in some cases it doesn't really save you much. It's just a different way to write it. But this pattern is just me playing with source generation. And if we go down a little bit lower, because we don't actually need this to run this program, which we'll run in just a moment, I promise. But if we go down a little bit lower, I have this sort of demo program to do here. And you'll notice that I am going to now use the workflow factory. So what I just showed you was creating one agent, but we're not really interested in one agent. That was kind of the semantic kernel way with Microsoft agent framework. We want a workflow. And so I'm going to get this triage handoff workflow. And again, this is source generated. So if I jump into it, you can see that I have this source generated method. It has doc comments on it. A little bit uglier to read it this way, but if we go to the tool tip, you can see that I source generated it to have this doc comment so that it tells you exactly what was registered on the agent with that handsoff attribute annotation. again to prove it. If you read on the screen, it says handoff targets declared uh via that attribute and then it explains the conditions for it. But those conditions again, they came right from here. Really just some syntactic sugar to be able to do this. And one of the reasons for doing that is that when I have this, let's go double check again what the name space is. It is in this generated name space. means I don't end up having to have the type parameter everywhere because that's a little bit annoying. The other thing that it's able to do if I go find down here, this run async is also an extension. So in the underlying code, you end up having to set uh this kind of stuff up. And so instead of doing that, I just have a couple of different helper methods that we're able to use uh when we start working with some of the the needler source generated agents. Again, all of this is not stuff that you could not do with Microsoft Agent Framework out of the box. This is just some syntactic sugar to clean up how you're able to do it. And so, does this have a cancellation token? It does. So, um I should pass a cancellation token as well. But in this demo, you can see that I'm going to use our new source generated extension method. And then we print some stuff to the console. We have a couple questions to ask in the workflow. And all that we're going to do is ask the questions one after the other. Okay, so nothing too crazy, but I think the cool part about this is when we go to run it, it is going to use Azure OpenAI, it will go run this workflow. And because we configured this workflow to start with the triage agent, right? Because this is the triage handoff workflow based on this triage agent. This is the starting point. And then that way it knows how to delegate to other agents. And this is a pretty simple boring example because it's not like it's going back and forth and everything's taking turns and stuff like that. It's really just like a sequential handoff. Okay. So, if I save that, have another double check here just to see ask the questions and then it's going to print out the responses. And that's it. I'm going to go ahead and run this and we should be able to see it work and ask the two questions. You can see that when we start this off, it says that the topology was declared uh through this. Okay. And just to prove it, let me Right. It's just printing out this console right line stuff and then it asks the two questions and gets the answers. In my opinion, the really interesting part about this is that when we look at what was output, we did call this triage workflow, right? And you'll notice that the two responses that we have have nothing to do with triage. You don't see the word triage in either of these two parts here. And that's because the triage agent started in the conversation and then it was able to go delegate to the other agents because of this workflow that we set up. So this is on purpose to be a really simple trivial example. But you can see that this geography agent was able to give an answer. And then we can see in the other one this lifestyle agent was also to give another answer. But where are these agents? So I didn't answer that. So let's go back and we could see in this triage agent I had two of these. You can see when I went to press F12 might be a little hard to see but there are two um places we can go. One is the source generated one and the other one. So this is the source generated one. And if I go back this is the geography agent. I'm just going to jump to the actual air quotes implementation here because this implementation is again just one that has the attributes on top of it. I'm going to show you one more little trick here because it's not totally obvious and if you're paying attention and you know anything about me, you might say, "Well, Nick, I don't think that you lived in those places." Right? So, let's go back here. So, the geography agent runs and it says Nick has lived in Canada. This this is true. I am Canadian. That's where I'm from. It says the United Kingdom and Japan. Neither of those are true. So, is this a a hallucination from the LLM? Is it just guessing? And it happened to guess right that I uh lived in Canada as the first guess. Well, no. It's a little bit more than that. That's going to be the very last part of this little demo here, which is that I mentioned function groups very briefly earlier, right? So, when I was showing you the triage agent, function groups didn't have anything. It's empty. Okay, so this is function types. My apologies. And then function groups on this one is geography. And we also have this lifestyle agent. I'm just going to jump over to it quickly. And you can see that the function groups it has are lifestyle. So we have these two agents that the triage agent can delegate to. And these two agents have these different function groups. So what are function groups? Well, if I go over to the geography functions, we can basically define a class, right? And this is if you're familiar with what semantic kernel was able to do. Very similar type of thing here. But we're able to have a class and we can use things like dependency injection to be able to pass in all sorts of dependencies that we might need. So this gets registered up. Then it gets resolved at runtime for us. But what you'll notice is that it has a function group on it. And that means that the agents that match this function group, they are able to use these functions, which is awesome because that means we can start to expose again more C functionality to these agents to use while they're having a conversation with us. So if you're following along, this is two different functions, right? Returns a list of the uh Nick's favorite cities and returns a list of countries where Nick has lived. Ah, okay. So if the LLM can see this tool, right, this function, I guess we're calling them functions in a Microsoft agent framework. If it can see this function and when it's going to answer the question that it was given, it will say, well, wait, I have a function that can that can do this for me, right? So it will go invoke this and that means that we'll jump into data provider get countries and there we go, the secret. This is truly the LLM invoking the uh triage agent with a question. That triage agent then knows to go hand it off depending on which question it is to the geography one or the lifestyle one. And then both of those agents have different tools or in this case they're called functions that are exposed and it's all registered through dependency injection and it's able to call those functions and they're just in C. So you don't have to hardcode an array. This could be a web API call. This could be reading something from a database or doing whatever you want from C. And that's the really cool part about all this is that you can start exposing anything you want via C to these agents all thanks to these little tools, right? These function classes and they get wired up just like this. Now again, you don't need needler for anything that I just showed you here except for some of the syntactic sugar. Otherwise, defining some of the uh the agents and stuff is, in my opinion, a little bit more verbose, a little bit more sort of like the thing I try to avoid with Neler, which is jamming more and more code up here. I don't like having to go do this all like in one big spot. I like it being more declarative and that way I can go pull it out and have it declared somewhere and then not all piled up in here. And that's why coming back to the very beginning, this kind of stuff don't love it here and I'll eventually migrate it out. That's going to be it for this video. In the next set of videos, I will show you a couple of other more uh sort of different pipelines that we can do. You can see a little bit of a spoiler down here, sequential pipeline workflow. And there's also like a group chat kind of thing going on as well. So, thanks for watching. If you're interested in checking out Needler, you can go check that out on GitHub. It's just on my GitHub, and it's called Need Dlr and it's an opinionated source generation dependency scanning framework. Uh, I use it for all of my plug-in based applications. You don't need to use it for Microsoft Agent Framework. And otherwise, I highly recommend you check out Microsoft Agent Framework. Microsoft, of course, has a lot of good documentation on it, and I will be trying to put out more content on it. So, if that sounds cool, you can check out the next video right up here when it's ready. Take care.