Scalable Systems For Swifties and Beyond - Michael Dodsworth of Fanfare

a customer of this stuff, too. I want it to be fixed for myself. There are 500,000 people ahead of you online. Like, what does that mean? >> Yeah. >> Is this good? Am I going to get through? Is this bad? There has to be a better way than this. >> Today's guest on the Dev Leader podcast is Michael Dodsworth, who is the founder and CEO of Fanfare. In our conversation, we were chatting about the significance of being close to customers when it comes to building software that's really going to make a difference. And one of the things that I found super interesting is that when Michael was talking about solving really complex engineering problems, it wasn't just about tackling the engineering problem directly head-on. It was really about thinking outside of the box. And if there's a way to solve this problem, especially in a way that you can please the customer, then that might be an even better way to approach it. Of course, with services like fanfare, we have to think about scale for these distributed systems, and that's something that they have woven into everything they're doing. So, I think that that mentality and thinking about building systems that way is extremely valuable. So, for me, tons of awesome information and different perspectives from this conversation. So, I really think that you're going to enjoy it. So, with that said, sit back, enjoy, and I'll see you next time. Michael, if you'd want to kick us off and give us a little bit of background for uh how you got started and where you're at now. And of course, you can take that and go as deep as you want or as shallow as you want. Totally up to you. >> Uh yeah, well, it's great to be here. Thanks for having me. Uh, so my background, I have an engineering background. Uh, I first got into this. I have, uh, two older brothers and one of them brought home a Commodore 64 one day and that was enough. Like, I was hooked. It had, I don't know if people are familiar, the tape deck where you had to change the you had to adjust the tracking and all of those things. But for the first time, like I could go in there and I could code. I could build things. And it came with this kind of flimsy instruction manual where you could make a balloon go across the screen. And for me it was new. It was exciting but it meant I could just from nothing build something that you know did something right. >> Um and that continued on. So my brother kept bringing these machines home. So Amigga Atari STS got more and more sophisticated and that's kind of that set me on this path. university. Uh I kind of did my little tour of the UK, as you could probably hear from the from the accent. I'm not from around these parts. Uh and then found myself at a a very early stage startup called Coral uh just west of London. And we were building this content management service and we got the attention of Salesforce uh intentionally like we got ourselves on their app store pretty early and they realized that you know their docs tab that they had was struggling. It didn't do the things that they wanted to. This is web 2.0 and we had a tag cloud and we were all sort of shiny and fancy and they ended up scooping up the entire team around seven people and moving us all out to San Francisco. I worked at Salesforce for quite a few years. My focus was mostly uh search and search indexing and the infrastructure for that which is like it was a great place to learn some interesting architectural solutions to these gnarly distributed systems problems and issues of scale. Uh I think we were processing I think it was 4 billion records at the time. >> Oh yeah that's nothing right. >> So it was kind of crazy. I joined in 2007 and we could fit all of engineering in a very large room. >> Uh I was there for eight years and by the time I left the search team could barely fit in one giant room. But yeah, really amazing people, really interesting issues of scale. A lot of that is kind of foundational to what we're building here at FanFair. Uh from there went to Optimizely just down the road. Very much like I did my Goldilocks tour of company sizes. So very small, very large, and now medium-sized. Um, felt like a company going through almost like adolescence, >> right? >> Product market fit, growing quickly, trying to figure out what's next >> and trying to find like what products next and and trying to like move from an early stage company to, you know, a company that has more seasoned uh senior engineers and so on, >> right? And that's when someone from my old uh old days came up to San Franc uh San Francisco to talk me through something he was doing in the live event space. Uh this was a company called Rival. Uh we were just a seuite and a pitch deck at that point but we had some uh great early investors. We had the Crom Keys who owned the Rams and a bunch of other teams and it felt like one of those problem areas where everyone feels pain. Everyone's tried to buy tickets for whatever it is and they've had to go to ticketmaster.com and they've gone through a bad experience, >> right? >> And it felt like there had to be an alternative. There has to be a better way of doing these things. Uh so that was Rival. Uh we were definitely on the nose with a name and fast forward to the start of the pandemic uh we were actually scooped up by ticket master in the first months. Yeah. Absolutely. So strange uh how these things kind of work their way around, but ended up at ticket master, spent a few months there at the start of the pandemic and then just kept feeling this kind of rage around these processes that people were going through. >> Just a reminder that if you're enjoying this conversation to give it a like and subscribe so you can see more interviews on the Dev Leader podcast. >> And it wasn't just in ticketing, it was in retail, it was in with collectibles. Like I have a bunch of things that I'm into. Everyone has the things that they're into. And every time you go through a process for one of those things, it's painful. And it really doesn't need to be this way. We've kind of It's kind of like a tax that everyone pays when they feel like they want to go to see Taylor Swift or see Oasis. I'm going to go through some pain to get there. And having spent some time trying to address those problems at Rival, realizing like there is a better way of doing this, >> right? And it people >> people assume like I I we we just are accustomed like if I want to go do this thing, I know I'm going to have to like go through hell to >> to have to make it happen. And like we've just assumed that that is the way that we have to go live through that kind of stuff, >> right? I'm going to be staring at a spinner for several hours. I'm going to have multiple tabs open. >> Yeah. I'm going to see those tickets on the secondary market immediately for like 10x the price. All of that pain, like we see that with the Switch 2 launch >> a few months ago. Like it's it's expected. >> I'm going to I'm going to be there at midnight. >> Best Buy is going to fall down. I'm going to go to Walmart. Walmart's going to fall down. Like all of those things should not happen, right? And that's um that's the goal of fanfare is to make this something that you know we can stand up a process that's not going to fall down under the pressure. That's a difficult problem to get your arms around, >> right? >> Strip out all of the bad actors and bots and then try and provide a fair process to everyone showing up and reward their time even if they're not successful. So that's the that's the goal here. And it came from just a thousand cuts on my side like every time buying one of these things and feeling pain. >> Well, it's very interesting because it seems like, you know, some of the best problems to solve are the ones that give us the most like either I'm like anger or frustration, right? It's like there's got to be something there if it triggers that much of a response and consistently like, okay, nothing's actually improving in this space. So, >> yeah. And it's been a long time. Like how how long has ticket master had that same process? >> Yeah. >> Decades at this point. >> I don't know the last time. I think at this point for tickets my wife will go do the ticket stuff and uh I think she's happy to do that or okay to do it but I certainly feel like for myself if she was like I need you to go buy the tickets for something. I think I would immediately feel like >> like I don't I don't think I want to do that. Like I don't know how badly I want to go to whatever because I know that I'm gonna have to be unless it's something where I'm like no one actually knows about this this artist or this event. Sure. And it's far enough out whatever. But otherwise, no. Like I for example, we're my sister actually in this case got a sleep token tickets for in the fall. And I think that if she would have said to me, do you want to go to this concert and if so, would you mind getting the tickets? I would be like, I don't I don't think I want to go. Yeah, it really shouldn't be that way, right? Like I want to go to these things. I want to own whatever it is. I don't want to have to go through this painful process >> to get there. And like it shouldn't be that way, >> right? And I I wanted to call out too, it sounds very coincidental, I don't know, but you seem to have a a track record where you were starting something or early at something and it got scooped up. are is this uh would you kind of say like right place right time or were was there like other things that kind of went into that and I don't know if you know the answer to that so I don't mean to put you on the spot for it but it's pretty cool. Yeah, I mean I think there's absolutely luck thrown in. Like a healthy dollop of luck goes into that. That the company I joined right out of college was like we had this incredible products lead who saw where the market was going. like it was a fairly niche product when I joined and he saw that this was something where you know cloud tech was going to allow people to be able to store this thing you know all of their files elsewhere and like he was definitely an amazing person that's gone on to amazing things um and I think with Rival the same thing was true >> like I I think you definitely you have to build up your chops and be ready to take advantage of opportunities when they come your way but >> right >> like There's always a healthy dollop of luck with with things like this. >> No, super cool. It's uh when you said it the one time and then you had another follow-up story like that, I was like, that's really awesome. and um going through differentiz companies um did did that kind of change for you over time where the um maybe not sorry I don't mean to to kind of pick on the the product itself or the service itself but the feeling of being at a smaller company versus a larger one versus midsize for yourself did was there like a more of a natural kind of like at the smaller one you're like actually this is really nice or the bigger size companies you're like this is actually a nicer fit for me. Did you have a sense for that as you were going through it? >> Yeah, I definitely like early stage smaller companies. I think, you know, I always want to be kind of going up and to the right. Like I always want to look back on what I did a few years ago and be upset, embarrassed, you know, like I' I feel like I've made progress, right? And I feel like you know a large company you can narrowly focus on a thing and you can definitely you know build chops in a particular area surrounded by amazing people >> but I think when you're at a small company you just have to pick up so many different things >> and you learn a lot in like a very very short space of time right >> like not just techreated but business related just the market you're in you'll learn a ton through that process so that's where I felt more more comfortable Um, I really enjoyed my time at Salesforce. I there were great people there solving interesting problems. I think you but you you're so far away from the customer that you don't really you're just kind of a cog in a machine, >> right? >> And it felt like if you if you were to take a two week vacation, like you could come back and everything was was just fine. like if I take a two week vacation now things kind of grind to a to a halt to some extent especially in the very very early stages. So it's a completely different feeling and there are pros and cons and I I just felt like I was definitely an early stage person. >> Yeah, that's super interesting. I think like for myself I was at a small company that kind of grew from like seven people to like the 250ish range >> after I left they IPOed. I think they're probably over half a thousand people or so now. going from that to to Microsoft on a platform team like the it's very interesting right it's like solving you know huge scale challenges um so it's not like the work isn't interesting or it's not impactful but your comment around um be like feeling removed from customers is for me is incredibly real because I went from like helping law enforcement with like helping save children to like something that is absol Absolutely not. That and it's not that there is no impact or that the work is not valuable, but the the disconnect from like feeling like you're talking to a direct customer is very real. And I I wanted to kind of call that out because I think for people listening depending on where you're at in your career and what you're doing, I think some people get into software and they're like, okay, like this is what I like to do, but they also don't necessarily love the domain they're in. And we're very fortunate as software engineers that we can basically go into all sorts of different domains, but people may not find that thing that they they love yet. And it can be really hard to be motivated by a particular domain. Maybe it's time to look for a different one where you are passionate about what you are building. So just wanted to mention that. >> Yeah. No, I think if if customers don't like what we're doing, I hear about it immediately. like they will be on the phone shouting about a thing like that's how close you are to customers at an early stage company. Uh which obviously comes with with pros and cons like that can be incredibly stressful. Um but yeah, I I do feel like, you know, I think it's worth doing that Goldilocks tour because I think especially at large companies, going from a large company to a very small company, you often will reach for something that you just expect to be there and it's not >> right. >> And that could be a process. It could be just something that you just assume will be there because people have been through it time and time again. There's going to be a wellestablished process for doing this thing. when you're at an early stage company, you have to define that process often. Like, it's just not there. There's no person that's going to do that for you. You have to kind of pick things up and do do it yourself, >> which is I find that to be to be great. Like, I I've seen enough of this that I have a good I I feel like I have a good feeling for for what's going to work and what's not. So, you get to kind of build those things up from the ground, which is uh exciting. >> Yeah. And the inverse of that being like, okay, you're at a company, they do have a process and you're like, oh, like I have seen something like this before hasn't really worked well. Like now you're in this position where you're like, hey, we don't have that. It's a different pressure or stress to be able to say, oh crap, we don't have that. But but now you can draw on your experiences and say like, okay, well, I've seen this not work, this not work, seen this be somewhat successful. Maybe we can borrow a piece of that and a piece of this and and come up with something to start with. >> Very cool. >> Yeah. And yeah, I think um it's definitely something I see people making that shift is is just it's kind of jarring to to go either direction. Um just to realize like there's an established process for this at Salesforce. I cannot do things outside the bounds of what's been set up for me. Um whether it's like I feel like it's in the right direction or not, like this is the way that it's done. >> Um so yeah, it's it's definitely a completely different experience. >> Very cool though. Uh, and I I agree with your your comment around like doing the Goldilocks tour like um, of course easier said than done. I realize that for people especially right now like some people might be saying I can't even get my first job but um, you know I I do expect that these types of things in terms of market correction will will change and adjust. Um and then at least my thought on that is that given the opportunity over time like you know I think it's worth exploring because unless you land in something where you're like oh my god I love this like I you know love waking up and going to work and working on stuff like if you find that in the beginning great. Um, if you're kind of feeling like I don't really enjoy going to work, but I feel like I have to do it, then I would personally strongly encourage people to go explore things until they they find something that they they're starting to align with. And I'm assuming for you at this point in this problem space, like as you were talking through it, it sounds like this is something you are passionate about, right? Like this is a problem that you want to solve. So when people complain to you about the software, it's not working or we want this, like you're like you're very motivated to be like, "Okay, like I hear you. Like I want to go make this better." >> Absolutely. Like I'm a a customer of this stuff too, >> right? >> So I want it to be fixed for myself. Like I don't want to ever be sat for three hours waiting for this. You know, you there are 500,000 people ahead of you in line. Like what does that mean? >> Yeah. >> Is this good? Am I going to get through? Is this bad? Like there has to be a better way than this. Um, yeah, it's kind of a cliche that an English person is obsessed with queuing, but uh I do feel like it kind of it's the ven diagram of things that really works for me. Like the surge in traffic means, you know, two million people showing up all at once, right? >> Is a really gnarly technical problem. like that dealing with those peaks and dealing with the troughs too like making sure that it's costefficient to run this thing like those are really really difficult problems to solve and the the consumer impact I think if you can solve for this problem I think you take a lot of pain away from people running this kind of thing. Yeah, definitely interesting like the two completely different fronts of that that are absolutely intertwined. One being the scale issue where of course if you're having like an incredible surge in traffic um you need infrastructure, you need a system design that's going to support that. But at the same time like what is it that you're actually trying to support? It's this enduser experience, right? So if you are putting all of these things in place to optimize your system, but it's genuinely for the wrong, you know, things that the the consumer is interested in, it's all for nothing, right? It's like it's not actually solving the end user needs. So it's correct me if I'm wrong, but it sounds like you need to be really channeling what that enduser experience should feel like to optimize for that and then design systems that support that entirely. Yeah, I think it's making sure that the base is stable, like it scales, it can deal with that surge in traffic, it can remove all of those bad actors and bots, and we can identify people uh as they come through, but then layering stuff on top of that. And I mean, for me, this is everything scale has to be woven through everything we do, >> right? like it has to be that we think about this in terms of like millions of people showing up all at once the first time we run this and like that I think means that we can keep ahead of people like a ticket master because like it's it's basically just a Frankenstein's monster at this point. You know, it's going to be difficult for them to to deal with all of the issues that they have all the way through all of their flows. Um, but we have the opportunity to try and do that from the very beginning. So, yeah. And I think there's all kinds of interesting stuff that we've been talking about layering on top. >> You have people who are maybe they love the brand or they're particularly focused on that particular product. What would you ask those people in that moment? Like what would you want to find out from them in that moment? What would they be interested in learning from you? What kind of other things can you do in that experience? Could you maybe live stream someone doing a how-to of a cosmetics product? Like that sounds like a really interesting and useful thing that you could do through the process. Ticket master, for example. Like, would I like to tell them which dates I'm available and how much I'm willing to spend and how many people before I even get into the process? Yeah, that makes total sense that you would want to give them that information because then they can try and figure out, you know, where you are and bucket you in and all of those things, but that's not done. So, there's definitely stable base and then a ton of stuff that we can do on top of that that's going to be interesting. That's a that's a really good point because I wasn't thinking about that as we were chatting through this so far because in my mind was like okay like I see this I see this ridiculously challenging scale issue and I say that because like if it wasn't ridiculously challenging probably wouldn't have a lot of these problems that we're all facing um when we go through these processes but >> it's like I think a lot of the time we think about these things like edge cases where it's like oh like this huge spike in traffic but like that's not the edge case in these scenarios. So my brain was completely thinking about that. But you're also, as you're saying, like layering on top of of this of this platform, thinking about different ways that people go through this experience and or sorry, these scenarios and like changing the experience fundamentally so that it's not okay, I'm I click the button, I'm sitting in line, and I watch the counter go and say like, do I sit here and refresh? Do I, you know, cancel all my meetings for the day and just watch this tab? Like, how do we change what that looks like so the experience doesn't totally suck? >> Yeah. And I think I mean, to strip it all the way back, I think there's the stuff that we need >> and for that, convenience is king. Like, I want I need to go buy detergent. I want it to be as simple and as easy and as painless as possible. In fact, I may not even want to do that myself. Maybe I want to offload that to something else. But then there's the stuff I want, right? The stuff I really care about. And right now it's the same process that you go through for both. >> Like I'm hitting just a product page. It's flat. It's dull. And like maybe there's a queue thrown in. Maybe there's a weight list thrown in. But for the for the most part, it's the same experience. And I don't think it needs to be that way. I think brands want to create some experience like some moment that people go through so they feel some attachment to that brand, right? >> Um, and I think back to the purchasing experience that I remember uh all the way back to I think it was a limited edition Super Metroid that I bought in the '9s. Like I remember the unboxing experience. like I remember all of that stuff viscerally because it was a limited edition and I went through an experience to to go and get that thing. >> Um, so I think brands can create more of those experiences. It's just there's no infrastructure for doing that well. So they end up dealing with all of the negative ramifications of a failed launch. Right? I see so many people they pour themselves into creating this amazing product and then it falls down at the final hurdle because the ecom platform decided to just fall over or they decided that you know bots were easily let through that caused all kinds of problems for them. So I think there's um it's a great model. I think more brands should feel like they should reach for it and there should be something there to make that easy for them. >> It's a really good point though on having like the concept of a different experience for the things that like you just I don't know like you just kind of need them. You don't care but like it has to get done. I need toilet paper. I need soap. I need toothpaste. Like uh even like my groceries I'm not I'm pretty basic with my food. Like I eat the same things over and over. If I could press a button and not think about that and the same food would show up every week for me to to go cook, great. Like I'm fine with that. But there's >> I think at this point I'm trying to think um as you were talking I'm like I know as a kid like I loved Pokemon cards. Um I'm fortunate that as an adult I do not collect Pokemon cards because I think being an adult and collecting Pokemon cards I would probably be having an issue with like buying Pokemon cards. So, I I'm hap I'm actually happy that I I don't do that now. But I was thinking through that if I'm was like, okay, if I still wanted to collect Pokemon cards, you were talking about like the brand and the experience and I was thinking like I actually would see a ton of value if like if I were lined up waiting for a launch of like a new box set to release. As a kid, I didn't really have to do this cuz number one, I didn't really understand it. Number two, it was like, "Mom, please please go get me the Pokémon cards." But as an adult, if I wanted that exper or wanted to go through that, there would be a lot more to that experience or I think it would be interesting if there was a lot more to it and perhaps there just isn't. >> Yeah, absolutely. Yeah, we're doing some things with um in real life experiences towards the end of the year. Um it kind of comes full circle for me back to live events, but the idea of we have the Olympics coming up in LA, right? What if the whole stadium can participate in this, you know, drop for 70,000 people, like 70,000 people all at once can find out if they've won something if they haven't. And maybe that's, you know, signed merch, maybe it's some meet and greet with the with the athlete. Like, it doesn't have to be a thing. It can be an experience, >> right? >> But it creates something that's lasting, right? It's a moment that people remember. I just have to make sure it's not a negative moment that they remember, right? Like you have to make sure that it's it's solid. >> And I would imagine that for most of the time, like, you know, for when you're queuing up to buy whether it's tickets or a product, it's like probably if you're going to have a moment to remember, it's probably a bad one at this point because you're like, I missed it or the site was glitchy or like I don't know. The good experience is like, oh my god, it actually worked. And that's probably brief. >> That's right. It's such a low bar. >> It worked. It didn't fall down. Is like the is like an amazing experience for people. >> Yeah. Yeah. That's such a funny point. Like Yeah. The bar is so low for that. Um very cool. I think that's a I I don't think that I had really kind of piece a lot of that together. So, thanks for walking through that. um when it comes to scale and going through these types of scenarios um I kind of said like you know I think a lot of the time we treat these types of things in systems as an edge case and it's very much not an edge case for for these scenarios it's probably the common case and so when we think about building software systems like this we have to have scale in mind right like as you were saying for what you're building it's it's just woven through things it needs part of the fabric of everything that you're building. Could we maybe walk through and it doesn't have to be an actual, you know, specific uh literal point in time where this happened, but could we walk through maybe a scenario? Uh I have in my notes like Taylor Swift or the Switch launch um something like this where we can think about like what should engineers be thinking about or like how do we how put our our scale hats on and think through like what needs to happen. Yeah, I think the Taylor Swift example is a really good one just because there is there's so much in there. Like it was a huge event. Like it was a pre-sale too. It wasn't general on sale. It was a pre-sale. So access codes had gone out. These are Swifties. These are people who are legitimate fans coming in to buy tickets early. And the issue with something like a a TM is it's it's decades old. There are all kinds of crusty, you know, connection points between different systems. It's like multiple systems all kind of grafted together. And I think it's important to think about the legitimate flows that people go go through. Like people are going to have to sign up, right? They're going to have to sign in. They're going to go through this process. They're going to click around on the site for a little while. It's going to throw like a couple of thousand requests our way. We have to think about that, too. but also the activity that's not legitimate >> or the the traffic that maybe isn't just straight down the line someone's just going to go buy a ticket because it was a pre-sale for people who'd been given an access code but everyone found out about it and everyone showed up so not just the Swifty showed up everyone showed up and everyone just tried to sign up at the same time so if you like maybe the flow to buy tickets is solid >> if your authentication mechanism, if your orflow isn't set up to deal with three million people signing up all at once, bad things are going to happen. And I think just thinking about you know bulkheading to make sure if one part of the system is struggling it doesn't impact the others thinking about making sure that there is like graceful failure in these areas because like TM when they fall down like they're down like they're down for some chunk of time and you know on the other side of it there are fires happening there are people trying to like stand things back up there are people trying to diagnose like where exactly ly is this painoint kicking in? Like you have to you you have to have a graceful failure so things don't just collapse like that. And then just simulating this stuff like if you know you have a Taylor Swift on sale coming you have to prepare like just you hope for the best but prepare for the worst. Like you have to be expecting a bunch of bots to come in, right? like maybe 10x the traffic that you're even anticipating to get. And you have to like try and flush through those flows to make sure that they like you understand where the the bottlenecks are and where the pain points are because it's usually not where you expect like just things can seem like they're fine until they're really not fine and then everything grinds to a halt. So, you know, I think you really have to simulate your traffic or have some kind of simulation to to kind of weed those things out. So I I have two questions on this. One is on the bot side of things and the other is around the simulations, but maybe because you were just finishing off with simulations there. As you were talking through it, I was thinking I feel like we like if you're in a situation like this, you need to have tests that would do this, but this is not going to be a test that's like, hey, we have unit tests or we have some functional tests that, you know, uh that integrate some pieces and we can see them working together. Yeah, you probably want stuff like that for your codebase. positive things. But >> these types of scenarios are like for to exercise. You already mentioned earlier that when you want to be able to have this kind of scale, you know, when there's low traffic, you don't want to be scaled up, you know, to the max and paying out the nose for it. So when you want to go exercise a scenario like this and prove that it works, like what what does that look like from your perspective to say we've built or we've added features, we built part of the platform, we want to test to make sure that we can handle things like this. What what goes into that to to go run a simulation? Yeah, I mean I we've been building the harness to simulate this thing so that we can configure different user behavior, all of those good things to make sure that we can have a reasonable >> um simulation of the process and it has to be end to end. It has to run externally so that you're you know you're not testing things right sat next to the thing that you're testing is important. um calling the vendors that if you can because these are expensive and especially as an early stage company like you don't want to find yourself on the hook for this giant uh bill >> because you ran a performance test and often the vendor will be open to that like interesting >> it's it's good for them like if you you know if you are running through Cloudflare and you you have some particular use case in mind and you're willing to do some co-arketing with them or like it's an interesting problem that you're solving on their platform, they'd be open to waving the charge or like bumping the limits to make sure that you can run those tests at a scale that is actually going to verify the thing that you want to test. So that's I think in two companies we've done that and every single time the platform's been open to waving those costs. So definitely do that first. Don't just kick things off uh and then you'll get a call from some of those vendors like why are you doing why are you doing this? definitely be a deterrent from like wanting to test things. You're like, "We'd love to test this, but that's going to cost us $10,000 like just this morning if we go run this." >> Yeah. And often that's it's a good way of getting on their radar, too. Like if if we're expecting an enormous traffic spike, like having people that you've already spoken to and they understand what you're doing, so if they need to bump a limit temporarily, if they need to resolve some issue on their side or they need to wave something through because it seems like it's a DOS attack, but actually it's a legit process, right? >> Like that's important, too. So I definitely go through that early just so the vendors know exactly what you're doing. >> Yeah. interesting point that like it is worth exploring that right it's not like because on the surface like anyone who is using these platforms is like well that's a constraint right like that's we have these limits that's what we're paying for but it's it is worth kind of going to the vendor and being like okay >> yeah cuz sometimes those limits they're not well documented right >> and like sometimes you can just run into a thing and you'll be throttled or worse and you know you'll get that phone call and you you really should talk it through before >> just to make sure that you're aware, they're aware and maybe they will they'd be happy to jump on a call whilst your event is going on so that if something comes up >> in that moment they can resolve it on their side. >> Yeah, very cool. Um and that I think that's a great tip. Um because I think a lot of people just wouldn't think about that. Maybe in their heart they're like, "Oh, I wish." But it's like, yeah, if you don't ask, you're not going to notice that or there's even an opportunity. So, um, okay. And then you mentioned like we talked uh, well, I mentioned that I wanted to talk about bots. You just mentioned DOS attacks as well. For me, this is super interesting because like the space that I work in at Microsoft is literally for helping with uh, DOS prevention uh, for Office 365. So um I I like I find this part pretty fascinating because earlier you were saying we're talking about scale issues, right? Like one thing that you can do is just you can throw money in compute and try to scale and try to like cool like and I'm not saying it's trivial to do that necessarily, but one of the directions you go is that way to automatically scale up. But um one of the other things you mentioned is like if you can figure out which traffic is not legitimate like you may not have to scale as dramatically as a result of that because you can cut out like a lot of the noise. >> What does it from your perspective like what needs to go into like identifi like sifting through the traffic to figure out what's legitimate, what's not and like what does that space look like? >> Yeah, I mean it's a it's a layered approach that we're taking. So multiple providers kicking in at different times. >> Yeah. >> Um just starting off with the least invasive. So trying identifying people based on you know what is installed in their browser issuing a challenge and going through that process and then gradually ratcheting up how difficult these things may be. So if we identify that you may be a bot we've never seen you before, right? you've you've just signed up, you're going through a process, you're you're you know, this is a particular brand that is prone to this kind of thing, then maybe we throw something in your way where you have to interact with it. Those things are getting easier to deal with uh in an automated fashion. So, that's going to get harder over time. >> Yeah. And like my feeling is if we can make these interactive experiences where you're engaging with the the page in some way in a way that's interesting for you. Like going back to my Commodore 64 days, >> there often used to be a game whilst the actual game was loading. >> Uh like they they realized you were there for 30 minutes. We're going to give something to you so you can take the pain away. Like some way of interacting allows us to then identify you as a human just because you are interacting in a way that's like beneficial to you. You're enjoying it and also beneficial to us in that we can try and identify people who are not legitimate and remove them from that process. >> A capture that you don't hate but you're actually happy to do the capture because it's interesting to you. >> Exactly. Like I would if if I'm on the hook for 30 minutes like I would for sure appreciate something been thrown my way so that I can just enjoy myself for 30 minutes. Maybe it's something that's communal too. >> So I'm playing against people who are also weighing. Maybe I can even incentivize people who have high scores to be moved forward in line. Right? So now you've created like a competitive element to it. Like there's all kinds of things I think you can do that are not painful, >> right? that are enjoyable even that allows us to identify real people >> and that com like it's again very cool to take this space that before we started talking I was like this is you know this is how it is done it a million times right like hate doing it know we got to do it and looking for creative ways like you're in that last example solving like a technology problem right like how do we how do we find bots like instead of it being like, okay, well, what's the, you know, the perfect signature that we have to match on these things? It's like literally just humans would be engaging with this kind of stuff. So, like, put that kind of stuff in front of them. They're going to enjoy it. or you try to create experiences where they do >> and then it's not actually like because the problem with captas is like I'm getting >> you know nine uh like a 3x3 like so nine cells of like pick the bicycles and like it's getting to the point where I'm like I don't actually know which pictures have bicycles but I'm training AI to to go find bicycles and like it's not enjoyable. I feel like I'm being used and at some points I'm like it's just like some of these captures are getting hard enough where I'm like I don't even know. >> So >> yeah, like where has a tiny tiny part of a bus in one of the corners? Like technically >> yeah, but I'm not sure >> two pixels of the tire in this square, but like so now I'm sitting here being like, wait, am I like >> how much of this is like abusing my time? Um, so everything that you just described there is very interesting because it's again solving like a technology problem not by throwing more technology at it by by changing you're changing the game, right? Like it's let's take a different scenario and it it fundamentally alters what you have to go solve for. >> Yeah. And I think also like building up that profile over time. Like if we see you come in and you're a legit person, you've queued up multiple times. Like maybe you failed to get your product five times in a row, right? >> Like that's valuable information and it's valuable information that most brands fail to capture. They tend to only find about the the successful people, the people who actually transact. >> There's enormous amounts of information in the people who just are unlucky. Like these are potential VIPs. These are people who maybe they're about to leave your brand just out of bad luck, right? Maybe you want to bring them in to an exclusive experience. Maybe you want to move them forward in line, >> right? >> Either way, like incentivizing people who are legitimate fans of the brand to keep coming back to these things and then rewarding them for doing that, I think is important to this too. That's a really interesting point that you know you might have some individuals and and I don't have stats or data on this so I'm kind of thinking through this for the first time but you might have people that let's pick Taylor Swift right they've been fortunate enough that when the tickets go up they're like they get in they happen to get them they're on their lucky streak and you have other people that it's not like they're not as big of fans but they just cannot get through. It could be that every time they have the crappiest internet or something. I don't know. But they're huge fans and they're getting to the point, maybe this is a bad example cuz it's Taylor Swift, but pick a different product or something where people are frustrated enough where they're like, I I just can't can't have the experiences or I can't have the product that I want no matter how hard I try. Why am I even bothering anymore? It's almost like I'm spending more time and energy >> doing something I don't like and being disappointed versus enjoying the product, the service or the brand. So to your point around like I it's very interesting that there's this other opportunity where it's like hey there's this group of people that their satisfaction with your brand not necessarily your fault but the satisfaction with your brand is like it's going down and down and down because their experience is going down and down and down. Yeah, I mean these are like I say these are potential VIPs. They just they have bad luck. So yeah, I think rewarding their time and effort and like making them feel like even though they failed, the process is fair, like I didn't see all of those things just vanish and suddenly appear on StockX or eBay for like 10x the price and feeling like the next time I come in I have more of a chance. So my time has been rewarded in some way. Interesting. Yeah, I think that's there's a lot of interesting stuff we've talked about so far that I haven't like on the surface I just wasn't thinking about things this way. So, so far in the conversation, it's like completely changing my perspective on like >> I don't know. I just thought we'd be talking about like, oh, here's exactly how you go scale up a system or whatever and like here's the engineering approach, but there's these other pieces to it where if you change the perspective of how you're looking at the problem, >> you start to you solve like the challenge, but it's actually um like an entirely different uh direction to go do it. And I think that that's pretty fascinating. I hadn't really thought of a lot of these things. So >> yeah, I think uh I mean one of the things that we had uh written on rivals walls was respect the problem, solve it simply and I think that's really important like it's it's it's very easy like I love this stuff >> from a technical perspective. I love taking these things on but sometimes it's the wrong problem to be solving. Like sometimes a simpler solution uh is like a much smarter approach to dealing with it. Um, so I I it definitely thinks like thinking about this as a whole rather than just as the focus technical problem is is really interesting. >> And that was I I wanted to ask you this and again I realize some of this is like I don't want to put you on the spot for it, but it's a it's that's actually fascinating to me. So the like do you have sort of like I don't know like a mental model or system that you that you leverage? I say system, I don't mean like a technical system, but like a a thought process or whatever, like a brainstorming technique where what you just said around like we have like a scale issue or we have some technical problem we have to solve, but how do you pull yourself back out to say like like instead of just focusing on the exact technical thing that's right in front of you to be able to zoom back out and say like maybe we can look at this problem space a little bit differently. And it's not that that problem goes away, but like now we can navigate solving it in a completely different manner. I don't know if that question makes sense, but I hope I hope it does. >> Makes sense. I mean, I think it it gets back to the connection to the customer >> because it's easy to to focus on the technical aspects of it. Um cuz that's what like every day I'm showing up and and building these things. But when you talk to customers like you find out things that you may be dedicating a ton of time on that to them is >> unimportant >> or to them like they have some other way that they're dealing with it which is really interesting. Like maybe they found a workaround or they're they're thinking about a workaround that's like sidesteps the issue completely. Um so I think that's really interesting for me as well. just stepping away. Uh, everyone has their thing that allows them to kind of switch off, you know, going for a walk around the neighborhood. For me, even though I'm in LA, like hopping in my car and going for a drive takes me out of what I'm doing and often >> allows me to to think through that stuff. Um, I'm a strong advocate for whiteboarding. I have whiteboards all over my uh apartment now right now. So like those kind of things of stepping back, talking to customers and just working through what their problems are or like what consumer problems are >> is really helpful. Uh but yeah, I I really think just stepping away for a little while, just let the subconscious kind of churn away on things and then come back to it really helps. That's uh that's very interesting because that that technique in general I think like programmers have if you've been programming for long enough like a lot of people have have seen that kind of thing where it's like I'm in the shower and all of a sudden it's like oh my god like I solve the problem or you wake up in the middle of the night cuz you had a dream and you're like you wake up and you're like I actually know how to go solve that bug and then you're like do I get out of bed to go code it because if I go back to sleep like it's >> absolutely yeah people who have notebooks by their bed, right? Because they just wake up and they're like, "Oh, I've solved that problem that I've been chipping away at for three days." Yeah, absolutely. So, yeah. So, the, you know, that's like a it's a pretty simple technique, but I don't think a lot of people actually lean into that. Like, it's almost like an accident when it happens. So, you know, probably a good point to raise that, you know, if you actively try to practice that when you're getting stuck, take a break. Like, it's not pausing for a little bit. you know, 5, 10, 20 minutes, 30 minutes, whatever, going to do something. It's not like your day and your productivity is destroyed now. In fact, it might be quite the opposite where you have some time away to to clear your mind on the on the engaging with users. What's um and I don't know if this is a like a simple answer, but what what do you find is effective in terms of getting that feedback or getting exposed to the the customers or the users so that you can have that direct channel versus the sort of the reaching out and like maybe looking for information like you might be able to go on to Twitter and see people complaining about experiences >> ver so that versus how do you get people to to provide that to you directly? any strategies there? >> Yeah, I mean there's there's some great material on this. There's a a great book called The Mom Test >> that talks about how easy it is to even if you're talking to customers and you're doing the right thing to to latch on to the wrong piece of feedback >> or to move things in a direction that's validating to what you already believe is true and not paying attention to maybe some of the other things that they're saying. And just you know people are people are more likely especially if you've invested a ton of time and it's obvious you've invested a lot of yourself in a thing they're more likely to you know try and be positive about the thing even if they don't believe in it right so I think it's really important to try and and and like read these material for sure but like try and like elicit this kind of detailed uh honest feedback And like sometimes that's, you know, asking the right questions, not asking the the the wrong questions. Like if you frame things in a particular way where you feel like there's an investment on your side, you're probably not going to get an honest response back. So I think that's important. uh and I definitely like people should should go and read that because even if you go through this process and this is the process that you know all VCs and and the the advice that you'll see on LinkedIn is like make sure that you're talking to your customers on a daily basis and that's true but if you're asking the wrong things like you're still not going to get the right feedback so I think it's important to understand the kinds of things that you should be trying to tease out and not like try and untease your ego and excitement about the product you're building and try and really hear what their pain points are, >> right? Because you could go into a conversation with someone and being like, "Hey, so like we're building this feature and it's going to do this thing and it's going to be so exciting and like that sounds good to you, right?" And they'd be like, "Sure, yeah." >> But like is that actually helpful feedback to you versus understanding where their pain points are? So, it's I I feel like you almost have to like shut up more and like listen to people and get them to kind of express where their challenges are and like as much as possible not not guide them down to like, hey, we have a solution. Does this feel like the right thing? >> Yeah. And it's difficult because like that's you spent your entire day doing that thing and you have to kind of distance yourself and hear where they're coming from. Yeah. Absolutely. Like it's it's very easy to to announce the thing that you're doing for them to say that's amazing and then you say how much are you willing to pay for it and the answer is nothing. Like they don't want to pay for it. Like they they're they're more than happy to say it's a great idea but then when they actually have to pay for it like it's a no. Like that means like it's not really valuable to them. And that's that's the thing you need to tease out. >> Yeah. And it's funny you say that too because the for the software that I'm building on the side, one of the things that we always talk about is like, you know, it's it's great if you you know have something that's going to solve someone's problem, but a lot of the time it's not even for for us in this moment. It's not like we need your dollars because if we don't have your dollars, we don't survive. Like that's not at some point that would be nice to be able to say great, we have, you know, sustainable income from this, but it's actually more like >> would you spend dollars on this? Because if you wouldn't spend dollars on this, like we're not building the thing that you need. It's like that test of would you depart, you know, would you hand over money for this thing or this feature or this, you know, this service. If the answer is no, then like we're probably not doing the right thing. >> Yeah. I think it's a real it's a real qualifying test, right? like are you willing to to spend money on this is very different from like are you just happy to to nod and say this is the best thing I've ever seen. Um yeah, I absolutely >> very interesting. Well, Michael, I think this is for me this was super enlightening. So, I really appreciate the time. Um wanted to and I can get links and stuff from you after, but uh for folks that are listening and watching like uh what's the the best way to read more or learn more about FanFair and anything else that you're doing that you want to share? I mean, we are posting fairly often to our blog and pushing case studies just so people can find out more about how people solve these problems. Uh, it's been less on the technical side, so that's probably a reminder for me that it's really interesting technical aspects that people would be interested about hearing, but head over to fanfare.io. Um, I'm always on LinkedIn for my sins. So, you can find me there and I'm always happy to talk through this stuff if people are interested. Like, we are going to be growing towards the end of this year. We have some interesting stuff coming up at the end of this year and through to the next. So, please reach out. Always happy to talk through this stuff. >> Awesome stuff. Okay. Well, thanks again. Like I said, super enlightening for me. Uh, and a lot of different ways to to think about problems. I think like I want to challenge myself a little bit more to think outside of the box versus just like put my head down and like you know what's the way to to optimize the code or like to think about just the system itself. Um I think that's super helpful. >> Absolutely. Yeah, it was it was great to meet you. >> Cool. Thanks. >> Thank you.