C# Semantic Kernel Plugins: Get YouTube Video Info!

what if your llm wasn't just constrained to having chat conversations with you but instead you could have it customized to go interact with services online hi my name is Nick centino and I'm a principal software engineering manager at Microsoft in this video we're going to be looking at leveraging semantic kernel in a searp application and working with plugins that interact with YouTube If you haven't seen the previous video that I put out on this you can check it out up here to get started with semantic kernel in C but this video will be building on top of that with slightly more complex plin if that sounds interesting just a reminder to subscribe to the channel and check out that pin comment for my courses on D train but with that said let's jump over to visual studio check out semantic kernel and these YouTube plugins on my screen I have this sample application and if you have gone ahead and watch the previous videos on this or you're familiar with setting up semantic kernel what we have on the screen is getting Azure open AI set up right up here so we have the information we need from the Azure portal and then on line 25 we're configuring the kernel to work with Azure open AI from there we end up going ahead and building it and we get the chat completion service but what we need to do from there is add in plugins so that semantic kernel can leverage them when it's having interactions with the AI in this particular case I have two different plugins that I've created one is called YouTube captions plugin and the other one is YouTube videos plugin what I'm doing here is I'm adding these plugins onto the kernel and I'm doing that by saying kernel plugin f Factory create from type and then I can just give it these two plugins the plugins that we're working with do not have to implement any interface and they don't have to inherit from some abstract class or any class for that matter so there's no dependency sort of in the hierarchy of these classes we just make them and then we add in the functions that we want to have the kernel leverage and we annotate them so I'm going to jump over them in just a moment but I did want to mention that in order to make these examples work today we're going to be working with a new get package called YouTube explode so if I jump over here to the different packages that we're going to be leveraging you can see that I'm using YouTube explode version 6.5 if you want to take the concepts that we're covering in this video and apply it to literally anything else it doesn't have to be YouTube explode here there's even other YouTube nougat packages you could use or if you just like the idea and you want to tailor this to working with some other data source or you want to have it into interact with your database you can take literally these same Concepts we're about to walk through it's just that I have this one tailored to be for YouTube with that said if I jump back over to program.cs we're going to go ahead and look at these two plugins so if I press F12 on the first one this is the YouTube captions plugin the first thing that we need to do is on the function that we've declared we need to make sure that we've marked it with this attribute kernel function and I just have it called get YouTube video captions right here right this one here on line 119 is the actual name of the method these two things do not have to be the same this is just the kernel function name and this is of course just the function name that we have in C you'll also notice that there's a whole bunch of description attributes added on here and this is just so that we can annotate the different parameters the return type and the method we're working with to give more context to the llm about what it's able to do with these methods that's because when we're talking to the llm we give it a message to work with if it's trying to fulfill our request for what we're asking if it realizes that there's a function that it can call one of the kernel functions it can go ahead and call that but it's going to be difficult for it to know how to call that if we don't give them good names and if we don't leverage things like these descriptions to give it more context about what's going on so the first plugin that we're looking at is for getting captions you'll see that I'm making a new YouTube client here this is not necessarily the best way to Leverage The YouTube explode nugate package so yes I am going to be making a new one every function call there's probably a much better way to do this without just newing up a new instance but I haven't gone and checked if this is thread safe I just wanted to make sure that I could play around with it for this tutorial the next thing that we're going to be doing is based on the video URL which is important here we need the video URL in order to work with this YouTube explode API we can ask for the Manifest of this video URL so we get the track manifest from there we ask for the track info based on the language so this is going to be English and then we say go get me the closed captions and from there what I'm doing is I'm iterating over those captions into a string Builder and then I'm returning the results of building up that string now you'll also notice that right here I have this kind of silly check here I'm just checking if the length is getting over 2,000 if so we'll break out of here and that's just because the current deployment I'm using is an older model and the contact size is very small so we're dealing with things like getting captions and stuff like that I'm going to overload the token limit that we have so just to make sure that I could work through this tutorial and keep it simple I'm just kind of cheating here to break out early but this is something that you're going to want to think about and I wanted to make sure I left this in here to force myself to talk about it when you're dealing with these different models there are going to be token limits you can configure that when you're working with semantic kernel but you'll also want to think about what your apis are providing back so if you're dumping tons and tons of content and the llm is basically getting that and it's unable to do anything with it you might basically always be overriding your token limit if you're doing ridiculous things so you do want to keep that in mind not try to return tons and tons of data but the other thing is that you need to consider what model you're using a lot of the newer ones have a much larger context so you can leverage more tokens but I have a perfect example here where I have an older one it's still totally functional it's just just that it's an older model and the context window is smaller so this is the first plugin that we have to work with again the key points here are kernel function and these description attributes what's inside of here is just specific for working with this YouTube Library this is just a quick Interruption from this video sponsor which is packed publishing now P has sent me over this book from Mark J Price which is C13 net9 modern crossplatform development fundamentals and I have the previous edition of this book sitting on my bookshelf as well and now I have this one which is super exciting because this book is packed with tons of awesome examples that guide you through the different functionalities of the language now I think that this is an awesome reference guide I think that you'll have tons of things that you can walk through from the very beginning to learn about C but one of the best parts about this is that it's not just limited to looking at the language you'll get to see things like asp.net core Entity framework as well as Blazer all in action with practical examples in the book I personally cannot recommend Mark J Price's books enough like I said he's got tons of awesome examples and he has other books as well from pack that I do highly recommend you check out so if you're interested in this you can check out the link in the description and I'll have a link up here as well that you can check out thanks and now back to the video the next one we're going to look at is very similar so it's annotated in a very similar way but this one is to get the videos you might be asking Nick why did you not put these into the same plugin I totally could there's no reason I didn't except to show you that we can have multiple plugins working together but you are totally right to say that if I wanted to I could take this function I could go add it into the other class that we just looked at and have two functions in the same plugin totally acceptable would work the exact same way this plugin in particular what it's going to do is take in the YouTube channel handle and I'm also going to trim off the at sign if it's provided in here and that's because this API here online6 63 does not allow for the leading at symbol so we need to make sure that that's removed if it does exist but from there a couple more things that we'll see is I'm going to ask for the uploads from that channel specifically I'm going to ignore any short form videos why just to make this example nice and straightforward also I'm taking 10 instances back not the whole set at the time of recording on my own channel I have 370 videos I don't just want to return an array of 3 70 videos I'm just going to take 10 uh again this is one of those examples where you're going to want to think about in your use case how you want to properly restrict this sort of thing right if I had to dump back all 370 every single time that might just be completely overkill for working with the llm it might constantly overload the token limit there's sort of different things that you need to be thinking about because this is going to be used by the llm and you need to be considering that when you're passing data back and forth but finally from there I'm just going to take the results of that which are in this uh the data transfer objects from the YouTube explode API I'm just packaging them back up into this record type that I've defined right here just to have the title and URL keep it really simple that's really all we're doing with these two plugins again I'm working with YouTube in this particular example but you're welcome to go tailor this to go work with some other service that's external you might have a database you might have some other service that you manage and you want to interact with so you're able to do all sorts of things like that with plugins now what I want to do is go run this and what we're going to see is that if I scroll back up here I have this little chat Loop that I've had from the previous videos in the series what I'm going to do from here is I'm just going to have a conversation with the llm and I'm going to ask it information about my YouTube channels and I'm going to ask for the captions so let's see what it's able to do for us so I'm going to ask the AI for the most recent videos from Dev leader okay so let's start with that one you'll notice that compared to the other videos if you've watched them the response time is a little bit longer here and that's because this is actually going and making a web request to go fulfill this but if I go expand this to make it a little bit more readable you can see that I have uh these different videos on my my channel so this shut up and take the money is a live stream uh this one is also a live stream um this one is a podcast that I have posted up um live stream live stream um this one is not so this one is actually a tutorial so these are a bunch of the recent videos from my YouTube channel this actually did go ahead and use the plugin so the next question that I can ask this is about another YouTube channel I have called code commute so I can say what are the most recent video titles specifically let's see if it understands titles specifically from code commute and let's go ahead and see if this understands just the titles so in this case no it didn't which is interesting right um it went and printed out the uh the full thing again um maybe I can say um try again but just the titles there we go so again this is really just like talking to an AI right we need to make sure that we're being specific uh with what we're prompting it with but this is my other YouTube channel called code commute this is essentially a vlog that I do primarily from my car when I'm commuting to and from work sometimes I do it directly in this office right here but if you're interested in more stream of Consciousness topics on software engineering most of these are submitted by viewers so if you're interested in having your software engineering questions answered you can check out code commute on YouTube and that way I can answer questions for you when I'm driving to and from work it is not live so it's not like I'm reading a chat and having conversations but I read them ahead of time and come up with my responses as I'm driving so check that out if you're interested in those types of videos so we've seen one plugin in action so far let's see if we can go ask for the captions of one of these videos so I'm going to go ahead and pick one from this list um let's go ahead with this one here so ship your product let's build an installer in visual studio so please get the captions from the video titled so what I'm interested in seeing is if it's going to use the chat history to go look for this title because it's already printed it out and we know that the plugin the function call needs the URL so I'm giving it the title it needs to go find the url I'm also not telling it which channel it's from so I'm kind of testing the AI a little bit here to see how is it going to reason about which method to go call let's go ahead and try this and see if it works ah okay so this says the captions were cut off and I realize I have a typo here I said the video titles not titled but the captions were cut off but from the available text it appears to be a tutorial on building installers for visual Studio Projects the video is about setting up for making an installer configurate for a product and downloading the necessary project templates from Microsoft's website interesting but like that's not what I asked it so please print out the captions for the video I apologize but I'm unable to rri the full captions for the video so let's go ahead and put a breakpoint in here and see if it's actually getting called so one sec this one was the captions so if I put a breakpoint here and we go ask it again so this was interesting I just asked it again for the captions right and it's saying here are the captions for the video but I put a breakpoint into the code and I put a breakpoint um inside of the method get captions right and my breakpoint never got hit so I feel like this might have been hit earlier on and this is actually in the conversation history now which is kind of interesting kind of peculiar this is truly the captions from that video but what I want to do is restart this experiment because I want to see if it's actually calling this plugin and now I'm not so sure I don't know how else it would have got these captions I also don't think that that in the captions that I'm outputting I'm putting any timestamps so kind of um peculiar but let's go ahead and stop this I'm going to go ahead and start the conversation off again so uh what are the most recent videos from Dev leader and I'm leaving this breakpoint on line 123 to see if it will get the captions please print or let me just say please provide the captions for the video titled I'm going to spell it properly this time so again this ship your product let's build an installer in Visual Studio that's number six year ship your product let's build an installer in Visual Studio it has the URL but I feel like it must have gotten this in the last time but let's go ahead and try so it does in fact hit this break point if I hover over this video URL I just want to pick out a few characters from here if I look at the end it ends with and list equal so I want to go before that and look at the actual video so ccore I think it's EJ or yeah ejw so let's go see in here um we want the installer one so ship your product let's build an installer in Visual Studio you can see that it has the ccore ljw so it did go ahead and figure out based on this title that it had the URL so that's pretty cool right that means that it is leveraging the history from it the conversation and feeding it into this one let's go ahead and step through um I'm assuming we're not going to hit an exception but maybe we will so let's see if it gets to the end and it does so that's great news but let's see if it actually goes ahead and prints this right so let's go ahead and add this as a watch I'm going to build the string so I'm cheating a little bit here I want to see what the results are like okay so this is pretty gnarly the way that it's printed this out across so many lines but um so this is truly the captions that we have from that video um if I scroll through there's a whole bunch but it is um cut off right because I did have this uh break out of the loop here to trim it if necessary so I know that from the video and because I filmed it not long ago that this is only partway into the beginning of it so um pretty cool it did work but let's go see if it prints that to the console properly now and it's interesting because it's telling us unfortunately the captions of the video titled ship your product are not available right so um I find that kind of baffling because it literally just uh went and got them so perhaps we've labeled some things wrong um but I think overall this is a pretty cool experiment to go ahead and get started with in my use case if I want to go ahead and make something that can be YouTube captions I need to now figure out uh if I've annotated something wrong so this is the English captions for the YouTube video perhaps this is not uh specific enough perhaps if I turn on logging so if I go back to the top of the program.cs file here maybe I need to put on tracing and I can see what's going on more specifically but the important part is that it is calling the plugins I think that was the meta point that I wanted to show you in this video and of course we got to see that the one plugin it is printing out the return value in this particular plugin for the captions I guess it's unable to figure out um that those are the captions for the video so interesting that it calls it and also very peculiar that once it calls it it doesn't think that it has the captions but anyway that's a pretty quick overview of how you can start leveraging plugins to call things like YouTube and again I want you to think about the fact that you can go call other services now right it doesn't have to be YouTube it doesn't have to be social media at all you might be able to interact with your own data stores or other services that you're interested in so thank you so much for watching I hope you thought this was helpful and I'll see you next time