Introduction To Cache Stampede Protection In C#

caching is an important but complicated Topic in software engineering and one of the big challenges that we can face with caches is something called a cash Stampede hi my name is Nick centino and I'm a principal software engineering manager at Microsoft in this video I wanted to walk through three different cache types that are popular in ASP donet core and that includes the IM memory cache the hybrid cache as well as Fusion cach which is a third party library and what we're going to do is look at inmemory implementations of all of them and see how they handle cash stampeding or not if that's sounds interesting just a reminder to subscribe to the channel and check out that pin comment for my courses on dome train with that said let's jump over to visual studio and start looking at some simple web apis all right everyone is familiar with the weather forecast sample that we have available to us in asp.net core and what I've done is enhanced it just a little bit by putting a cache in place so to explain what's going on we're going to be looking at the memory cache the hybrid cache which at the time of recording is in preview and as a result they don't want you to go use it without explicitly making sure that you acknowledge that it is still just for sort of U preview purposes and then we have Fusion cache up here which is from Ziggy creatures if I'm not mistaken and that is a thirdparty library what's great about all three of these is that the usage patterns like the apis and how you set them up are all very similar but I did want to call out that the memory cache is of course in memory but hybrid cache and fusion cach do support distributed caches we're not going to be looking at that here we're just going to be talking about the API usage and this idea of a cash Stampede before we go any further and start to explain what's going on here I figured I'd take this moment to sort of explain what a cash stamped is and this is just going to be a very high level so that you understand the impact of this the idea when we're cashing things is that when we have something like a request coming in what we're able to do is trade a little bit of memory to make sure that we can serve a request much faster so that means the first time we go ahead and do a request what we might do is take that response cash it and that way the next time a request comes in we have information to be able to give back immediately that means we can reduce things like latency it also means that we can reduce stress on Downstream Services say like accessing our database or maybe another service that has to go compute some value for us we can basically say hey we don't got to go interfere with you at all we already know the value thanks this seems pretty simple on the surface and if you're just dealing with a simple web application one instance of it really it's not so bad but things get more complicated in distributed systems especially when we have many requests in parallel so what is a cash Stampede well the idea is that if we have a bunch of requests coming in and they all in parallel go to check the cash what can happen is the cash goes yep I don't have that value cached go ahead to the downstream service whether that's the database or some other service go calculate that value and then I will cash it for you but if there's many requests coming in at the same time they're all sort of going to that Downstream service calculating something and then end up cashing it the challenge in particular is especially when you have something like a cash value that's expiring and you have a lot of traffic hitting a service all of a sudden instead of it returning very fast immediately for all of these things you now have this stampede of requests that go through to the downstream service and cause a ton of load on that other thing the other side effect of that is of course you're going to have increased latency and obviously increased load just because of all of the extra processing Downstream the idea with Stampede protection is that we can sort of throttle how many things are getting through just to give you a quick idea behind this if you have one single process you can generally have Stampede protection for that process and that means if you had something like a 100 requests coming in at the same time you would be able to restrict one request at a time from getting past your cash because the value is not there going Downstream getting the value cashing it and then the other request sort of get the benefit of that one single request cashing that result they might all have to wait for that result to come back but at least they're not going to go all affect that Downstream service now in a distributed system where if you have multiple instances of that service let's use the same same example you have 100 requests coming in but you have five instances of the service if you could uniformly distribute those requests that means you would have 20 requests going to each of these services at the same time each one of those will have Stampede protection and that means that what you're able to do is have one request from each of them get past the cash because the value is not cash there go to the downstream service that will come back and then each one of those will have the new cash value and then the other requests can take the benefit of that the idea is that you didn't have all 100 go through but because you had five Services it meant that five requests could get through so that's significantly better it's not guaranteeing one single request when you have a distributed system but it's significantly better than all of them that's a very high level of the idea behind cash stampeding and some protection that we can have in place so let's go back to the code and talk about these three different things that we have access to and of of course there's many other thirdparty libraries but these are the ones that I like using because they're built in or because Fusion cache itself is quite popular already the first one that we're going to look at is imemory cache and I'm going to go ahead and expand this here you can see that the API for memory cache is get or create a sync this is a pretty common pattern generally when we're talking about things that have stamp protection the reason that this is a popular pattern is that you're not doing separately a get call and then a set call the idea is that the API itself has both put together but what we're about to try and see is if the IM memory cache actually has Stampede protection so let's go ahead and see exactly what's going on here we have the cach key this is going to remain constant for all the requests coming in and essentially if the cache key is not present we will call this Factory method and if I scroll right down here we're just going to see that the create forecast Factory method is just going to have a delay of 1 second this is just to kind of simulate some observable behavior and then we're going to print out to the console that this forecast has been created so if I scroll back up the idea here is that hopefully if we call this method and we call it multiple times by hitting the API we should essentially cach the result and then everything else after that will be good to go what I've also done is I've created this traffic spammer it's essentially just a loop that's going to have a bunch of tasks fire off and hit our API not to complicate things but I'm going to go ahead and run this and then I'm going to use the traffic spammer to go send a whole pile of requests and we're going to see essentially how many times we actually call create forecast async if we have Stampede protection our single process should only allow one of those requests to get through and that's because it's waiting it's blocking the other calls saying hey someone's trying to access this cash already just make sure that you're waiting meeting when we get the result back all of you can use it before we move on this is just a reminder that I do have courses available on D train if you want to level up in your C programming if you head over to dome train you can see that I have a course bundle that has my getting started and deep dive courses on C between the two of these that's 11 hours of programming in the C language taking you from absolutely no programming experience to being able to build basic applications you'll learn everything about variables Loops a bit of async programming and object oriented programming as well make sure to check it out okay using our traffic spammer I'm going to say weather forecast is the endpoint we want to hit I want to send a th000 requests and let's go ahead and run that we're going to pull this up at the same time and you can see already that we've called the create forecast method a ton of times right forecast created keeps getting logged to the console here by the time this traffic spammer has finished doing a th000 requests basically as fast as it can and parallel so it looks like the imemory cache does not have good cash Stampede protection it allowed a bunch of these I didn't count them out I'm assuming that's either a thousand or a whole bunch that is way more than one obviously so this is not protecting us from Downstream Services being accessed and if you're wondering well what Downstream services that whole idea of putting the thread sleep here or the task delay in this case is sort of simul ating some behavior that might be expensive like calling a database or going out to another micros service to get a result unfortunately IM memory cache despite having this API that's getter create sync does not seem to support protecting multiple calls from going through so let's go ahead and stop this I'm going to use next hybrid cache and again we already have it added into the uh dependency container so I'll comment this out and I'll pull both apis up at the same time so you can see the one and the hybrid one are almost identical it's just that we have different cash entry options so same idea we have dependency injection we're able to access it on the minimal API and then we have very much the exact same type of syntax here with the factory method and just some options we can use but let's go repeat this experiment so I'll go ahead and run the API all right now when we go run our traffic simulator we'll go send a th requests we see forecast created one single time and then we had this finish off so all 1,000 requests went through but we only made one call to the you know Downstream service so we went and created that forecast we waited 1 second but then everything else was able to take the benefit from that so it does look like the hybrid cache does support Stampede protection which is awesome so let's go ahead and we'll stop this now let's go try Fusion cache so again Fusion cas cach is from Ziggy creatures if I go over to the package references here you can see Ziggy creatures Fusion cache it looks like I'm using a a preview of the 2.0 version here let's go back just for reference in case you're curious maybe when you try this the API looks different or something like that or the behavior is different right so we'll comment that out we are going to use I Fusion cache if I can spell it properly I Fusion cache there we go and then I will take this and what I wanted to do again is have all the apis on the screen for you now we can see that this one is almost the same it's get or create down here this one's get or set if we look there is no get or create um there's no add there is get so it's just get or Creator get or default so we're going to get or set into the cache but everything else is basically the same right we have the same cache key we have a factory method and then of course it has options as well the us cach options are a little bit different we're not going into the details of the cash options in this video we're just going to be looking at the Stampede protection so if we go ahead and run this let's go repeat the exact same experiment okay with our traffic spammer ready to go I'm going to send another thousand there we go we'll see on the right side forecast created and it's done immediately so the forecast created only got called once it does suggest to us again that Fusion cache just like the hybrid cache does have Stampede protection just to summarize in this video we looked at three different caching options yes there are more available including the built-in one that's just I distributed cache which we did not look at because that one does not have stamped protection it quite literally has two calls to either get or set the cache values but imemory cache does look like it might have stamped protection with that single API call to get her set or get her create but unfortunately it looks like it was still calling that method the factory method multiple times when we had a lot of requests coming through now with the hybrid cache we did get Stampede protection and the fusion cache option also did provide Stampede protection as well that's just a quick overview of what stamp PE protection is some popular options and I hope that gives you a little bit better of an idea some things that you can do to work around Stampede protection thanks for watching and I'll see you next time