Effective Software Teams: Islands and Autonomy - Principal Software Engineering Manager AMA

all right we're getting going here I'm pretty agitated because restream is not working on like any platform so like uh LinkedIn is not going I don't even know if anything's working to be honest so let's see um I'm very flustered because I got home on time to do this and nothing's actually working I see people are joining on Twitch which is cool thank you but like you can yeah like uh it's crazy um yeah LinkedIn will not join so I have to go it keeps trying to do it to the last event which doesn't make any sense give me one more SEC here sorry folks I'm just trying to get my myself situated but yeah it doesn't even crazy okay well sorry LinkedIn uh not going to happen uh everywhere else I mean Instagram's not working either YouTube is working hi folks on YouTube finally the YouTube audience is going to beat out the the other audiences but usually usually Twitter's pretty good so um that's really unfortunate I can't believe that anyway whatever I'll give up um okay so topic for today is going to be about software engineering teams and uh different things to consider for for structuring them in efficient ways and this is just based on my experience so um if you're new to these streams then uh hopefully the way I kind of explain things and walk through them you get the feeling that I'm not telling you like this is the only way it can be because uh that's not how I think about things uh instead it's more like um it's more like here's my experience here's like lessons I've learned and like I would much rather you hear the lesson and then you can have your own interpretation of that so if you have a different experience um I mean like always your mileage may vary right so if you have a different experience that's totally cool like I hope someday you can share your lessons with other people and stuff that's all I'm doing here so to start things off I'm going to put a link to the full article if my computer will work give me one moment and that way if you want you can follow along uh if not that's cool too but this is where if you're ever curious like where my the topics for my live streams then it's on my newsletter website so if you want you can check it out this is the uh today's topic um just checking where it's going no LinkedIn is not live anyway it's not going there uh if you're on YouTube you should see it in the chat Facebook and twitch uh if you're on Tik Tok sorry um but it's weekly. deev L.C um you don't have to subscribe to the newsletter if you're not into that kind of thing that's cool but if you're ever curious like you like these live streams you want to see the topic that's where it's going to be uh and of course these are amas so if you're familiar with joining these you hopefully Realize by now I try to watch the chat as much as possible and kind of chat through any of the questions that you have I would rather prioritize that than the topic that's being discussed but with that said um let's jump into this so this kind of stemmed from a conversation on um my my Vlog YouTube channel that's called code commute and the the topic that was originally proposed was around uh structuring like way teams bran good to see you uh thanks for joining I'm glad it's working on Facebook and you can join uh so the original conversation on code K was like okay like do you structure software development teams with dedicated uh QA do you have a QA team do you have QA at all um and I've kind of lived through all of the above and different flavors so I wanted to share what my experience has been but that was the code commute conversation and when I wanted to write about this in the news newsletter article and have the conversation for this AMA I didn't just want to focus on like like just QA and how QA is set up and um I think one of the reasons why is I think that's a that can be a pretty heated conversation so I wanted to share some insight about that but I think there's this like this meta point about team structure that I wanted to get across and so I called the newsletter article and this live stream islands and autonomy and that's because the these are two different things that come up when I see Team structure that I think influence the effectiveness so hopefully that's a okay starting point that makes sense but um to dive into things I think at a very high level to introduce one side of this one thing that I have seen work not very well is creating Islands so Islands in autonomy the one thing that doesn't work very well is islands and I've seen this come up in a hand of different ways um so I felt like I would walk through some of these examples and again as I go through this it's not my intention to say like this can never work or this will not work but I want to share with you like some challenges I've seen with this so that if you're and I had even someone on code commute commented and said by the way like thank you this is so helpful because literally the teams that I'm on their organization they're literally considering some of these options so they were like now I have you know it's not a rule but I have some some data points to kind of go by from someone else's experience so when I worked at Magnet forensics uh before Microsoft this was very much like a like when I started there was like seven or eight people and um the idea of testing in general was like did not exist which sounds kind of funny um but I mean that is like a role right so we didn't have any like unit tests or coded tests in the code we didn't have QA or testers hey Ryan good to see you um hopefully it's a hopefully it's a helpful topic so thanks for joining um but yeah we didn't have like there weren't testers there weren't QA um and then no coded test so like we literally kind of started From very basics of like we got some code how do we like truly turn this into like a software engineering organization and so early on it was like okay we're going to hire 10 ters and so we bring on testers and one of the other things I should mention too is we're going through this is like we were really trying to figure out kind of like I was hinting like how do we be a software engineering organization like what's involved with that we like have I worked in like a software engineering organization yes I had at that point I had six internships so I had like two full years of you know working in software engineering teams but this is a brand new company and like I should mention it's not founded by by software Engineers right like the people that founded this company weren't like hey I'm a software engineer I know exactly how this is done so we're like we got to make this work and figure it out um so what's interesting is we said okay well we know that we value quality we should test the product we should bring on testers so we started doing that and then when we were scaling up the teams we're like okay well we should hire developers and testers but we're kind of scaling up and we're kind of figuring out agile and stuff and we don't really know what's going on but we got to try things and so we were set up to do Sprints so if you're not familiar with like some I don't know some uh pretty simple agile Concepts like a Sprint would be a period and for us it was two weeks but these are sort of arbitrary you can set them to be whatever you think is appropriate and then at the end of a twoe Sprint we would come up with new commitments for work for the next Sprint we would do a retrospective and talk about things we kind of planned for for a Sprint so in our case two weeks at a time and what we did was we actually had the team of QA as a dedicated team and then we had like the developers uh that were on one team to start with and kind of split into a couple to work on different areas but the point is that we just had QA as like a unit as a whole team and developers separate and this was okay so spoiler alert this did not work well but um it like we tried this for what I think is a very good reason and we were looking at sort of in isolation how do we speed up developers and in isolation how do we streamline the work that the testers are doing so what we did was we had these two like these separated teams and then we also staggered the Sprints right so they weren't Sprints starting at the same time they were literally staggered and the idea behind this and again spoiler alert did not work very well was that we could have developers kind of take their their Sprint commitments and they can say I'm going to go build these features or fix these bugs and I'm going to you know crank through them so they would land a feature they would land a bug fix they would integrate that into the mainline branch and then what that would do was feed into be work for the next Sprint for the testers and that meant that the developers could keep going and then the testers would have this solid pipeline of work that they could chew through and you know they could go test this they could dedicate their time to it so um if I mean if you have some experience you might say well I know exactly what's going to happen here and you're you're probably right but what would happen was as soon as the the next Sprint would start and a tester would go to pick up their very first work item the very F like within minutes they would be like this doesn't work this is broken or sure this works but like this is totally unusable like not the user is not going to understand this or I have questions like this doesn't make sense you didn't do balance checking like it was whatever you could dream up within minutes sometimes even less testers would be like no like this is just not okay so what had to happen well either because it's in the product already we would have to have the developers context switch because it was very urgent that they introd a bug and we had to go fix it or we would say ah it's not pressing enough but we'll we'll kind of we'll schedule it back in for our next Sprint we have two problems that occur with this at least um one of the first problems is that you have this ridiculous context switching so oh you introduced a bug like yep now you got to go fix it so get back to that or you have this huge DeLay So you you've landed a feature now like two weeks later you have to go back into it and like go upates to it so we we set this up because we thought it would be efficient for either sort of Ro and it was terrible absolutely terrible and I think that came down to some of the mentality around this stuff was we were trying to optimize things but really what we were doing was being like not obviously it wasn't as explicit as this but kind of what our actions were saying was I'm a developer I don't care about the quality that's your job right that's what was happening in practice I'm just going to code all of these things not my problem now it's your job so that's one example where we had things set up in Islands so when I talk about Islands in these examples the concept here is that we have individuals of a particular role Focus area that are truly operating on an island together separate from say a core team another example I've seen with this is is with uh documentation and you know no like if we deem that documentation in terms of like technical writing or uh help guides or whatever else it needs to be if we're saying part of our product releasing is that that documentation has to be up to date at one point we had sort of like an island again use use the concept here an island of documentation writers now what that meant was the same type of thing would happen if except maybe not as obvious because I think the relation between developers writing code and things being tested is a lot more tangible where people can relate to this but with the documentation writing it was similar right so someone would go build something and they're done but now it's like I don't know like someone's got to write the documentation so when's that getting done oh the documentation can't be done on time do you pull the feature out because you can't write the documentation well someone's going to say no that's ridiculous like the feature is already in there it's already doing the work like no way so now are you saying the documentation is not important well no the documentation is important okay so is the release date not important oh well no we need the release date well one of those things at least has to change and more often than not it was that documentation would we would say like we can ship this without the documentation we'll follow up with it but the point of this is truly not to say like hey documentation isn't important the point of this is to say we created an island with a role that we said this is going to be a necessity for shipping this and then we created a bottleneck because of that island so again the island became this thing where if we have the island we think it will optimize stuff in practice absolutely not I see some questions coming in so uh and hey Gabriel from Brazil awesome that's sweet thanks for being here I'm going to jump over to the questions in just a sec uh but I just wanted to finish up this thought with one more example and that's with um even with product owners okay so I've seen this I've seen this become interesting and I try to share examples where it's like you can see maybe both sides um so I've had it with product owners and to kind of paint the picture here again take an island you take all the product managers in this case you put them onto to an island you have the product management team if they're not working alongside the core team that they're product managing for you now create this separation of different priorities now if I put my manager hat on I can totally understand that you might want to have a product managers report to a product manager manager by the way is my stream super choppy someone wants to let me know that would be great I can see like some of the playback from restream and like it was frozen for like 15 seconds um but let me know um but the whole point of the product manager example is like I can see from a reporting structure uh and I can see that you want to keep people aligned to talk about things that are related to customers but again with this example when you create the island people end up having different priorities and if they're not integrated together working on things together the more separate you are and the more that you're isolating each other on these islands you truly create silos so it's less clear for communication and your priorities become more uh more opportunity for misalignment so being able to work together and ship value becomes more challenging there's inevitably more friction because of impaired communication and difference in priorities if you can reduce the impaired communication and you can uh get your priorities alive you can work around this but having the islands actually makes this more challenging okay jumping over to some of the chat um so Ryan saying I have to admit I'm personally one of those in one of those or I am personally one of those bottlenecks okay definitely curious how to fix this for myself along with building up the team I'm on um so Ryan saying I work in healthcare one of the bottlenecks that I have run into is have these system integrators that handle the central software crucial for hl7 messaging I have no idea what that is but it sounds fancy sounds very health car I've seen some leadership believe that if anything with the hl7 messaging needs to be changed then they need to be the ones to do so since it's their system bread and butter I personally believe that you should always perform the work in the place that makes the most sense how do you suggest to approach these leaders and help them better understand that doing the work where it makes sense instead of just the same place all the time so this kind of like the sort of the next phase of this conversation is like I'm giving you all the examples where we have these islands and again I think that on on the surface and I I'm saying this because I literally lived through this multiple times of saying you know we put people on an island because we think it's going to be awesome and then we're like it's not that awesome um one more quick example is with Architects I've seen software Architects pulled out from teams and you might have heard what do they call it like the ivory Tower architect they design things they're going ah this will be perfect uh this is my design teams go off and do it and then the teams are like what the hell are you talking about that's not going to fit in here this is ridiculous and there's a huge misalignment because the architech just built what they thought was perfect and the teams are like that's not going to fly like it's going to be so much work we're so far away from that do you want us to ship value or do you want us to ship architecture as an example so for Ryan in your case I think what needs to happen here is the next part of the conversation which is explaining when the bottlenecks come up this is probably the first part when the bottlenecks come up like what is the impact of that like one thing is that it's a huge risk and I think that that's worth calling out so single point of failure huge huge red flag right so if you have a single point of failure where that needs to be done in one spot um say if it's truly a single point of failure like one person big no no that needs to be addressed how do you get the skills the expertise how do you spread that knowledge that's one part if it's a team and that's a single point of failure right what happens now is and I think what you can do with people in leadership is if you have examples of hey remember that time you had to you Dro this onto the team and you said you got to get this done what did that delay and if if the answer is well that delayed nothing and no one was overwork and everything was fine maybe there is no issue but was there actually a delay like did people have to reprioritize things and something had to fall off if that didn't happen were people working after hours to try and make this stuff happen um like talk maybe kind of talk about the impact that it's actually had I think that's beneficial to understand because I think what happens is that sometimes people see this as like a you know they're on the outside of it and they go well this seems like the right thing to do and it's not a malicious thing they get comfortable doing it they've seen it they've seen it work do it again right so it's not like a it's hard to blame people but if you can present like information and say like hey we should talk about the effectiveness of this I think that's important I think another thing is like what are some solutions what can we do differently and which is kind of the next part of this conversation which is the opposite of islands and that's autonomy so instead of having an spoiler but I'll talk more about this but instead of having islands with roles you think about teams as an autonomous unit that can tackle things and that means that they have less opportunity to get blocked on other teams with different priorities and you can kind of take the roles that are needed to make something happen and put them within the team if that means I'm not familiar with hl7 messaging but it's like if you need an hl7 messaging sort of some expertise like that across different teams so you don't have one single point of failure maybe that's what has to happen and maybe you have 10 teams and you don't have 10 experts cool do you have three of them can you sprinkle those people out can you start to teach the other people and sprinkle that expertise around so in this case it's less about a particular role and more about like a skill set but it's truly enabling teams to be able to have some of that autonomy so uh Ryan let me know if that helps to start with and if if you have any thoughts or feedback on on those suggestions so far please just you know great job for providing all the details so far thank you and if you want to add more or uh take a different angle on that just let me know uh Romeo Sweeney what's your take on the H1B debate I don't want to really get into politics and stuff to be honest um it's uh I don't know I I have opinions about this kind of stuff I I feel like it's not it's not good spot for me to to like whine or complain and because I'm not doing anything about it so instead of me just adding like noise to it I just feel like it's not really worth me talking about um to be honest I'm I'm on a Visa working in the US I'm not on an H1B but I'm on a Visa working in the US so you can probably assume what most of my position is but yeah um so sorry Ry I'm just not going to I'm not going to go too deep on that um choose a reality uh I've also seen these islands at friction in the case of streams emerging where the islands are a bit more of an autonomous team but used as a way of coordinating work on new vers uh live things ah yes um that's an interesting one and actually I can speak to some of this uh because I led prototyping teams before and we often made the new exciting things and and have them integrated um so I'll I think that's a a good uh a good thing so I guess semi-autonomous island is more a hybrid thing yeah so let me let me come back to that one because I think that's an interesting interesting thing right so my my meta point when I talk about autonomous teams is about the role and function uh with their business value so for example if you want to set up a team and you're saying hey team you are going to be responsible for shipping the product like this is your thing you own it or this is your service like you need to make sure it's up and healthy and meeting all these metrics like you want to set teams up with a charter that's very clear and tangible for them to own right you are the you're the the metrics team you're going to support getting all the metrics and having infrastructure for the other teams to leverage like that's your Charter um I think clear Charters and I think um also finding like the space of product or service deliverables I think is really important but within that now especially from like from my perspective as an engineering manager like what goes into that okay well we're go like we're a software development company we're going to need developers okay now you can have this debate about uh testers and QA I can come back to this if there's time because in the um the code commute Vlog that I mentioned uh I talked about how we leveraged testers and then did not and how we leveraged hybrid roles that were not testers but more like a testing architect and had a lot of success with that so I can I can talk more about that after if people are interested but the the whole idea is like what goes into a team that you need to be able to deliver effectively so I mentioned documentation earlier okay if documentation is going to be a critical requirement of you being able to deliver your software if it is do you need a role for that within the team if there's enough documentation to write perhaps or is this a responsibility that has to be taken on by the team I'm losing my voice out of nowhere for some reason so you have to think about roles like that if you're going to have testers like same conversation right and this is more nuanced I know that people will have strong opinions about this but you could say like like is this something that you need a roll for is this something that's a responsibility of the individuals on the team uh I have the same opinion about um product managers I I have worked most most of my career without product managers embedded on the team I have seen it work and I've seen it work mostly because the product manager is either responsible for only one team so it's almost like they're directly embedded or they're responsible for like one major team and like a smaller team and they can they can split their time and it works because they're not they're not juggling that much but the point is that they're able to balance it so I like thinking about having these roles within the team as much as possible so that you don't have bottlenecks on other things that way when the team is planning for how they need to deliver stuff and get stuff done they can focus on their priorities and inevitably there's going to be dependencies with other teams because if you're building a suite of products or you have one product and multiple teams contributing into it uh or you have platform teams and you need like new features from the platform team this stuff's going to come up it comes up no matter what so it's not about saying oh look Nick's trying to propose a way to eliminate all of the friction all the dependencies no that's not my that's not my point here my point is to minimize the the overhead and the friction on a regular basis and personally from my experience when you start taking these islands right take the Architects from the Ivory Tower embed them in the team bring in that skill set right have people responsible for architecture within the team have people responsible for testing within the team for documentation writing whatever is important have it within the team now your team can move forward as a cohesive unit and go tackle together so that's that's sort of the meta Point around the autonomy part and um I'm just scrolling through the article to see if there's anything more cuz I usually find when I do the live streams I blab a little bit faster through these um oh um single points of failure I wanted to talk about this a little bit more too um because I talked about some single points of failure with the island kind of set up but I wanted to mention that single points of failure for when we're looking at like a healthy team and an effective team a single point of failure does not have to be like hey we have this island of people like the documentation writers and they're the single point of failure so let's do something about that a single point of failure can truly be like one person that could be on a different team or it could be with on within your own team so um with the architect example right if you have um you might even have okay so I use Architects already with like a an ivory Tower approach let's take a different example let's pretend you have um a team of Engineers and there's on the team there's someone that is you know they're the more the most senior engineer on the team um and depending on who's watching and your your experience and stuff uh there is a what's a good way to put this I don't know how much I'm allowed to like curse on YouTube so I never do but the the phrase is like intelligent and they're someone who is very smart you know very technically Savvy but they're also like they're like difficult to work with um they they basically steamroll a lot of conversations and a lot of the time it's not malicious or anything they don't even realize they're doing it but it's kind of their part of their personality or their Persona to to steamroll conversations they uh they're suggesting things their way is basically always the right way um they don't necessarily always back things up because like they don't they don't need to they're the authority and it makes it really difficult for other people to to get their work done now if you've worked on teams with someone like this not only a whole different conversation about about correcting that but like they become a single point of failure and that's because any poll request or code review that has to happen they need to review it they need to sign off on it uh and more often than not they want everything changed right it's not up to their standards it's not how they would have coded it so go change it but they're doing this on every poll request and code review so it's almost like they might as well have just coded everyone's code um they might as well have done everyone's design dock right because like it's not up to their standards for their design and people like this and one of the reasons one of the many reasons why as a manager this is something I need to to be careful about and try to um you know correct within a team is like it creates single point of failure and that's like it's just not acceptable there are other reasons for you know why this type of person is challenging but single point to failures one um the I kind of actually I'm looking at my article I mix two people together uh so you can also have this type of thing happen when it's not an intelligent it's just like a senior person on the team and that could happen if uh say on your team you have a pull request rules it's like the senior person or you know senior person has to sign off I saw uh choose a reality said in the chat the domain expert okay domain expert has to sign off on your poll request but there's only one right they are the domain expert that's one single point of failure so when I talk about single points of failure in this context um this is something again for engineering managers I think it's really important to think through to create healthy teams that are effective but I would also say like if you're an individual contributor on the team um this is something to have awareness about I think some people I hope this kind of perspective is changing but I think sometimes people have historically been like oh single point to failure that's called job security like they need to keep me um not really um but I think the other thing is like if you're observing something as a single point of failure I think this is something that you should try to vocalize within a team you should try to bring awareness to it so that um if your manager doesn't notice or whatever it happens to be like there can be some action that's taken right so it might not be immediate right like I'm not I'm not suggesting you say hey look like there's only one domain expert so therefore we snap our fingers and now the is gone no but if there's only one domain expert and say your manager wasn't fully aware it's like oh crap like that's the one single point of failure that's becoming the bottleneck for people's uh poll requests in this area okay like how do we start Skilling up other people in that area right like how do we how do we work towards removing this single point to failure this bottleneck so I think yeah my my recommendation is like if you're an individual contributor please try raising awareness about this stuff I think it's important um um hopefully as you know the manager on your team's able to have visibility into this kind of stuff but yeah uh architect oh it is live it's definitely live it's absolutely live just not on not on LinkedIn today because re stream didn't work for some reason I don't know why kind of dumb and I couldn't do do it to Facebook or sorry I'm doing it to Facebook I'm not doing it to Instagram so kind of kind of disappointed um but it's definitely live so thanks for being here I appreciate it um I wasn't lying when I said these are live streams and I wasn't lying when I said I read the chat too but um I think maybe the final part from this article I don't know it's not really covered in the the article but it is from code commute when I kind of thought of this topic in general and that's going to be about I want to spend a little bit of time on this CU I think it's an interesting I think it's interesting interesting walk through about um testers QA and um sort of like a journey that I went on structuring teams and navigating what that looks like so at the beginning of the stream depending on when you joined uh or if you're coming from code commute uh you might kind of have a little bit of the precursor here but when before Microsoft I was working at was what started as a startup and we kind of had to figure everything out in terms of like how do we become a software engineering organization and when we started off and had a dedicated testing team eventually when we moved away from this model because it was extremely ineffective to have like staggered Sprints with a dedicated testing team we started having testers so when I'm saying testers um originally and this is a bit of an omen clature thing we were just saying oh it's QA like these are QA uh QA roles and then we had people kind of pushing back over time what the heck LinkedIn who's on LinkedIn and how how did you get here oh is it streaming to the wrong event one sec this is important yeah it says LinkedIn you oh my chat's not showing oh my goodness I'm falling apart there's the chat um it says LinkedIn user and I don't know where it's streaming to so I don't know how you got here so I'm amazed sorry I needed to pause for this because I don't I still don't even know where I'm streaming to oh it's it's streaming to the old this is so dumb is streaming to the old event this is last week's live stream so thanks for join I think it's Lenny right yes thanks for joining Lenny um yeah Lenny saying with Team structure I don't see any magic formula that fits everywhere team should add team members that complement weaknesses encouraging knowledge sharing helps reduce the single points of failures absolutely and that's the thing like what I was trying of saying at the beginning of the stream is like I'm not here to say like this is the only way to do it but kind of like fundamentally what Lenny is saying is like it it boiled down to like where how can we reduce single points of failure how can we uh have like knowledge shared how can we make sure that basically the team itself can be effective without one person kind of carrying all that weight um and yeah it's not going to look the same so I think I'm hopefully going to illustrate this with this QA and testing example that this is the case so um I was just touching on the fact that QA and testers uh we basically made a I think our our testers made a push to say like we we want to be referred to as testers not just QA and that's because quality is owned by everyone now I would also argue that testing should be owned by everyone too which is where we're going to get to here again all of this is my experience um I'll Circle back on some of this stuff but what happened was we started to have our testers integrated within the development team so you'd have the programmers and the testers and that meant that there was no longer this like let me let me go code it and then like I'll just give it to you when I'm done uh in fact I mean this still took a little bit of time I think I think a bunch of teams that we had still operated this way kind of in the beginning and um one of the things that I would say that I'm I'm still you know to this day like I'm quite proud of is that uh we we focused a lot on continuous Improvement and that meant that like we kind of started off trying to do Agile things by the book as much as we thought uh whether or not that's reality or not different story but we tried and I think that ultimately when we we saw things that weren't working it's like we tried to make adjustments so I think over time people learned that when we had testers embedded on the team it's not sufficient to just say I'm going to go program this stuff and when I'm done I'll let you go test it there needed to be some more interactions that happened in between so not just waiting till the end because like we learned historically when you wait until the very end and say okay now you test it you're going to have a rude awakening within two seconds when someone's like there's a bug right away happens all the time so one of the things that was a little bit unique about the teams I manag was that we moved away from this and it kind of started by accident and that's because we just didn't have people so um I mentioned that I I ran like prototyping teams I also ran a uh a small team for uh mobile phone acquisition so that meant that in digital forensic we could take a phone and um you'd plug it in and we would go take all of the information off of that phone which is a fun problem to solve so that's what my team was responsible for and that team started off by myself and a product manager doing this kind of after hours and we had the first prototype working when we were off in an airplane I love telling that story so then we said okay like basically we had approval to go kind of kick off this this product and okay so we got to make this real now if it's going to be real don't we need people testing it too so we brought on someone who is what I would call uh the the phrase we use was a a test strategist and if if you're not familiar with like the idea kind of think about like a software architect but for testing but we brought this person on and really at the beginning it was kind of like you know we were so Scrappy that like I can remember going through this phase of like we're just trying to build whatever you know tossing things together and I can remember we reached a point where we were like should we should we like start using jira because like we literally reached a point where we're like we think we need something to track the work we're doing because we're losing sight of like all this stuff going on so like that's how Scrappy it was but we leveraged a test strategist for kind of like just any Hands-On like basically a user proxy so they were really just being like a tester right in the beginning we had to bring on a ux designer right but the ux designer she was kind of on an island so as a ux designer because okay this product it's going to be real we need ux for it okay so off on her Island goes and makes the ux then she showed us and we were like oh no we can't we can't do that we're so far away from that and this is a really cool thing that we did was we just basically started to actually work close together so even though she was on an island we made sure that the iterations of the user interface we were going back and forth and like staying you know on the same page so it was a really good working relationship after where she would say here's like the North Star for where I want the user interface to go I get that you're not there so let's take incremental steps and move in that direction and spoiler alert we never ever reached the north star ever and people were using the product and they were happy so like it didn't need to happen we just wanted to use that as The Guiding uh direction to head in so um things evolved though with the testing part and the reality was was that we had the one test strategist on our team and the other teams were having like more testers and they were they were responsible for like they had to go click through the user interface and you know they like Unfortunately they end up being like the quality guard right and we said like that's not really sustainable for our team and so with the test strategist what we were doing was saying look like we're not telling you that you don't get to click through the application and use it um like we need a user proxy that needs to happen but we have this special role was a really cool thing that magnet did they hired in people that were either forensic examiners or investigators they were literally real users and they were hired in and they were they had roles like other people in the company so there be people that were like they worked in sales and they were they were previously like examiners or investigators or they worked in training or they worked in or they worked at like uh some I think eventually in development or they worked alongside the developers at least so we had these individuals that were truly when I say user proxies I mean as close as you can get to a user without being a user so we had this expertise in house so we also didn't need our test strategist putting as much effort into being the user proxy let me go use that as a user but we used our test strategy just to help us think about the gaps in our testing so how this started to look was if you take the base scenario where if they're just operating as a tester what we would have been doing sort of like some of the other teams in the early days was hey I made the thing uh I've I've written some tests for it you know it works seems to work for me you go try it out and then they would go test it and be like okay these 10 things are busted go fix it and then we're okay this is this is super inefficient okay I'm going to go build this next feature and before I actually start coding let's sit down and talk together me and you me as the developer you as the test strategist let's talk about the different parts of this thing so we're on the same page and we could bring in the user proxy as well so the product not quite the product owner because we actually had a product owner as well as these these examiners in house let's get on the same page and talk about this okay cool we agree with what roughly this should look like now test strategist I'm I'm telling you now like based on what I think I want to build here here's some tests that I think I could write for this obviously when I get things coded up we'll have another conversation but here's what I'm thinking and then the test strategist might say okay like that makes sense but um we should think maybe here's an edge case it would be good if we could cover that okay good so we get on the same page conversation about it then I would go code it up and before and I know if you're a tdd person you might go a bit crazy here but before I wrote the test I would say Okay test strategist I have something that I think is working I want to go cover this with tests I'm going to start doing what we talked about and here is a a build of the software it's not pushed into the main production Branch but here's a build of it try it out while you're doing that I'm going to go write test like we talked vote and that way I'm building in the confidence that you you thought and I thought we agreed on that we need so I go start writing tests and what was cool was that that person because we had talked about a lot of these things they felt comfortable that they didn't need to go test all those edge cases they would test some of them and they go oh yeah like that's still broken and I go I know I caught that in my unit test I was writing and then they would go oh now that I'm trying this out and I have a real thing there's a couple of extra things we should consider like can you add test for those I'd say no problem and then at the end I would walk through I could just show them all of my tests and they go yep I feel good about that and what that meant is that the more we did this the more that the test strategist was a lot more comfortable being able to walk through test cases with me in code and not have to go click through the application to verify everything so they could spend more time thinking about where are gaps in the types of tests we're running like do we need different test infrastructure so that was a huge thing um do we are we missing some general types of scenarios and you know are our conversations about our unit test coverage our integration test coverage are we are we on the same page for these things or are we constantly having escapes so ultimately we use this to like to have one test strategist on the entire team enable the whole team to be productive this way instead of saying hey tester go test this thing now that I'm done us and to the point where going forward I basically we and not not because we like got rid of him or something like that but we ended up being able like once once he left the company we didn't bring testers back onto the team we basically enabled the entire team to think about test strategies they were going through all of this so it's not that we thought the role was obsolete but we we took on the responsibility and said no we're we are responsible for for testing we have user proxies to help us with you know gauge that uh in terms of like the user experience but like we are responsible for the testing and the quality and now that we've gone through so many iterations of this I'm not saying that we're the experts on all things testing and we can replace the test strategist but we could operate very effectively using the things we learned um so the the next team that I took on um in this same context was kind of interesting because um slight departure or slight change because I I kept I maintained this team but I ended up uh forming another team for a different uh different purpose and a unique thing about this team is that the work in terms of the what's a good way to explain the type of work I guess was very repeatable okay that doesn't mean that it was like it was boring or not valuable it's just that the type of work I don't have a better way to explain it the type of work was quite repeatable so there'd be there'd be new things we'd uncover but like for the majority of the time we're like the steps we have to go through are very similar and when I formed this team we had one tester so it was a similar thing it was like by the way you as the tester this is you going to be a bottleneck on this team if we do things like traditionally so how do we how do we make this effective and I think sort of the the framing was hey look in the beginning of this it's it's probably going to feel like you're the bottleneck and it's going to be a bit of a pain in the butt for you for the the developers trying to get stuff done it's going to feel kind of awkward kind of crappy but the goal of doing this is to transition away from being the manual Hands-On tester and to like to basically become the test strategist so what that meant was that we had a lot of work that was very similar like I said and the first few Sprints it was like yeah it's painful because your hands on testing stuff but we said you need to teach us every time you're going through and you're finding this sort of class of problems teach us about it how are you uncovering it like what's your thought process so that we as the developers can go build this into the development process right we can go build that into the beginning of it we can build tests that will catch it and let's see if over time we can reduce your workload and it absolutely worked did not mean that the tester we just replaced them and they were no longer useful but it meant that the tester we had could focus on the strategy they could focus on the gaps we could prove to them hey here's all the tests we have it's everything we've talked about do you really need to go click through all the exact same stuff that this automation's running no you don't and you can focus on things that are even more valuable now and you caught these other things great let us go learn about these and make sure we can try to integrate automation to help catch them as much as possible because we want to improve our confidence level as we're going forward so I think it was a really cool I want to call it an experiment um and it it worked from my perspective it worked very well um and yeah just a kind of a unique experience of like of how we leverage testers and I think ultimately it came down to like what's really important is not uh like I don't think it's a good use of anyone's time aside from like being a user proxy clicking through stuff if you want to go understand how if a user finds this stuff obvious totally fair I just think it's a very poor use of someone's time to go testing things on repeat by clicking through stuff computers are very good at that and I think it's a way better use of someone's intelligence to be able to help focus on the strategy and areas that need to be improved with testing so um the final thing that I want to say on this is like um up until I moved over to Microsoft that has been what has worked very successfully for me in terms of forming teams and around uh programmers and testers kind of like we don't need a dedicated tester but someone who's thinking about test strategy can be very valuable and if someone said to me today like if you had to go for a team like how many testers would you put on the team I would say I'm not putting testers on the team not because I don't think testing is important it's because I want every single person on the team to be thinking about how to test and we would need to make sure that we have a good representation as a user proxy so that's been my experience um if I try different things in the future and they seem better I'm not saying that what I've experience is the you know the perfect solution and that it can't be improved I'm absolutely open to like different things but that's been so far what's worked best okay over to the chat uh I don't know how to pronounce your name so I apologize uh but it's an architecture question when you have a moment broadly speaking do you prefer CA or vssa for small to mediumsized projects with a small Dev team um I'm biased because I just like uh vertical slice architecture um it just like I have to just have to admit like I just am very biased towards it I don't think that I've like professionally used clean architecture I think I have professionally used a lot of layered architecture um and I've seen a lot of layered architecture be done very poorly um and I I'll I hand up I'm the one who did it poorly so um oops uh but with layered architecture I've seen that done poorly where your it's a good way to explain this where dependencies kind of bleed through the layers so for example if you have like your your data entities where you're what your database looks like in terms of the records that you are able to see those in full resolution and use them in your UI layer just as an example I've seen that become ridiculously bad because now you want to go change your data structures in your database and all of a sudden someone's like you can't because you're breaking all of these UI components and you're like why the hell is my UI breaking because I'm changing something in the database and it's like oh because you didn't translate anything between these layers there was no separation so um I know your your questions about uh clean architecture I'm just making a point about layered architecture um and that's like I said I'm not blaming anyone I was responsible for a lot of that and then going oops how do I undo this um so that everyone's not yelling at me um but I like vertical slice architecture because I like developing features in vertical slices um so what does that mean because I I realize some people might hear that and they're like H what the hell are you talking about um I actually don't even primarily think about vertical slice architecture first um because I think that there are different architectures and I think that there are many different ways to accomplish you know similar things and I think one of the big problems with architectures I see is that people go this is the definition if we have a concrete definition of this architecture things must fit into this box which is great until you have something that doesn't fit into the box and you force it into the box and then the next thing you force into the box so uh I like things being a little bit more fluid in terms of architecture I think there's philosophies about different software architectures that are important that we should focus on that are going to be slightly different from Project to project of course but when I think about vertical slices is I think about that from a development perspective what does that mean well if you're going to go deliver a feature instead of saying okay we have a feature I'm just making something up off the top of my head uh we're going to take a a desktop based app application we need the users to be able to have this uh part in the user interface they can fly out and show them some I don't know some options that they can configure I'm just I'm trying to make this up on the spot sorry what I don't want is to go okay what are the different layers or the different parts of our architecture that we have to go touch okay well someone's going to have to go update the database schema for this so like let's go to the database team or or or the the single point to failure person that's responsible for the database hey you go land the database change for this let us know when it's done okay great that's there now who owns the data layer like who who owns that data access layer let's go land that feature we'll go do that and then when that's done let's go have a conversation okay and now who owns the business logic like let's go deliver that feature and the it's not maybe I'm doing a bad job of explaining this it's not even just Landing the part but when people have to think about like sort of let's go build out like a layer to go do this thing I think becomes a ton of overhead and what I would much rather see is like we're talking about building this feature go implement it top to bottom right if that means that you want to go land incremental parts of that that's not the end of the world but I think when you start having like dedicated people to go touch that I think that was more of my point like you have the you know the database team or the the data layer team or the the business logic team and then the UI team when all of those things are very separate responsibilities like good luck Landing stuff it's painful um however my example was kind of bad because I do think that if you said hey I want to go do a an ink like a small piece of this so you have your vertical slice and you can think about it I when I think about vertical slices I still go into layers just for my headset so you could go say I have this vertical slice of work that I need to go deliver for this feature but I want to go land the database change and the data access layer first okay go do that like I don't think there's anything wrong with that but it's more about addressing the feature from a delivery perspective not across teams because I think that starts to get very messy um and then sorry on Tik Tok there's a question uh from Good Citizen thank you by the way for uh joining in the chat on Tik Tok it's the least active chat so I appreciate it does a principal software engineering manager at Microsoft manage a single team of software Engineers six to 18 devs this will change from from Team to team um So currently I think I manage a team of 11 people I think that are direct reports to me um now it's a bit unique because it's one team of 11 people but I think there's four there's four and a half cuz one's kind of like virtualized uh feature Crews which are sub teams that work in completely different sort of functional areas they're still on the same team in the same service area but responsible for different components of that so that means that with 11 people I actually have some teams that are like two people or three people uh and it's kind of interesting because it changes the Dynamics up a lot it means for me there's a lot more overhead in terms of the different things I have to be responsible for it's part of part of the role um it also means that it can be really challenging when I have a bunch of different priorities how do I balance those because I don't want to like hey this other thing's really important over here like I'm just going to pull these other people from this other feature crew over like I have that ability but like it feels very bad um so yeah in my case it's about 11 people um in a single team but multiple feature Crews which are like sub teams the previous team I was on I don't know how many people I managed probably around the same number but I had two teams um and the I had dotted line reports so for at least where we are in in Office 365 uh the way that a lot of our teams are structured we have like people North America and then kind of like a follow the sun kind of approach with our team so there' be people in China as well so those individuals are not direct reports to me they have their own manager in China but they'd be more like dotted line reports to me so when it comes to um you know managing priorities and and what the team is responsible for then I'd have some dotted line reports in in China for example so that other team I had was I actually inherited a second team so I had two teams and and so yeah it's going to it's going to vary from situation to situation but um I I hope that makes sense good citizen uh and then good citizens I'm I'm an upline second line IBM with two managers an architecture and IBM 40 plus people love your Tik Tok thanks man I appreciate that um we are all we are all in on the geek way awesome no I do appreciate that thanks I think um it's kind of interesting my career trajectory is kind of um it's been cool to be able to go from like a startup to a you know to to Microsoft and like I started at Microsoft at the principal level which I you know I think that's cool I feel proud about that it's kind of interesting though if I didn't go to Microsoft I basically would have had it shortly after I was not there I would have had like a a director title um at this point arguably like a VP title um so kind of funny um if I look at my my title I'm like ah like kind of doesn't it's not as cool um but it's a it's different uh different responsibilities right so uh where I am now it's like you know I I'm not the manager over the whole routing space but it's it's a really cool you know huge huge responsibility um in what we're doing so yeah the the level of impact I think is it's really exciting so but yeah thanks good C and I appreciate it I you know thanks for joining on the chat um architect TJ 37 how to estimate the time needed for unit integrated whatever test implementation how long it usually take almost never have time to write a test at work okay I love this question um the answer that you're expecting is it depends um but my um here's my my approach to this kind of stuff um when I when I make videos and stuff on on my main Channel Dev leader I I like to talk about testing a lot because I've spent a lot of time trying to to find ways to like I don't know like optimize getting testing as part of development because I think it's critical I've seen an infinite number of examples at this point I know that doesn't make sense um but tons of examples at this point where having tests has saved our asses I I think that they're they're a necessary thing to have in your software development process however I don't think that it necessarily makes sense to try and say we need 100% code coverage everything needs a unit test everything needs a you know integration test I look at testing as a way to build confidence and uh even like outside of work in my own projects right I'm building something called brand ghost and I've talked about this on early streams on code commute where I'm sharing some like behind the scenes of like what is it look like to go building a service and try selling it to people in the very beginning we were basically saying we don't have test why don't we have test because that code that I just wrote I'm literally probably about to throw out and guess what we don't even have users right now it doesn't matter it doesn't matter if it's busted users aren't going to complain because they don't exist and I'm probably going to throw it out because I'm just trying something and then we're iterating and iterating and then we get some users okay but they're like they're kind of like uh you know good friends that are trying it cool yeah I'll try it out cool if it's busted they don't actually care they're just trying it out okay not a big deal do we need tests yet uh like not like maybe maybe not really and then all of a sudden we reach this point and this is a this is a point about confidence where I'm going and this is real like I need to make a change to the database I'm actually nervous now to make this change because I don't know if I'm going to break things maybe now we have like not not at that point I mean like maybe it's still not paid users but it's enough that I'm like I feel nervous that I'm going to break this and if I have to go tell a buddy like ah yeah brand ghost is going to be down like my content might stop posting on social media their content might stop posting and I'm like do I really need to be having this conversation with myself that I don't feel confident and to me that's a point where I'm like okay I need more tests in place or at this point I need some to be able to start having confidence so for me it's always a question of confidence so um architect to answer your question how much time the way that I would frame this or how I've tried to address this is that I try to write code such that it will be testable what that looks like is going to depend from person to person um but at this point in my life not professionally but in total I think I've been programming for 21 almost 22 years um certainly most of that was not writing tests but at this point changed how I program to make things testable that means I you C some people freak out about this but I like using interfaces on many things because when I go if I want to write a true unit test I know I can go mock out those dependencies and just isolate stuff I also over the past few years have been using things like test containers a lot that means that I'm thinking about okay when I'm writing code can I at least set this up to use test containers have a Docker contain container spun up and I can go easily write a like an integration or a functional test for this so it's hard to put like a number on on your question and like to give you an answer but my point is that I've structured how I develop things to make sure that testing is minimal effort I write code and I design software in a way that enables testing and that way if I go oh like I need to write a test on this what's not happening anymore for me is a I have to rewrite this I wrote it in a way that's totally not testable I'm screwed because now I want to go add tests but I have to rewrite it to be testable so I'm probably about to break it and then I want to go test it like it's bad spot to be in so I try as much as possible to write code that can be tested in terms of time um as a result it's minimized and that allows me to make decisions about do I want to test this now or do I feel pretty good about this and if I need to come back later and add tests because I'm doing a pass and trying to make things a little bit more robust that at that point I can say you know what I'm adding tests I might as well add a few more unit tests or something else that I have that that luxury without having to go rewrite stuff so uh I realize it's kind of a crappy answer uh for maybe what you're after but um in if you wanted like more more of a number I would say if you're just starting out and you feel like you're rewriting stuff to write tests double your estimate and then start carving it back from there maybe maybe use that as a forcing function to say okay if I can't allocate twice the time because that looks bad how do I start writing code that enables me to be able to write tests more effectively so drop your estim it's not double anymore that's a bit extreme I'm going to take an extra 50% to go WR test I going to go take an extra 40% 30% until you're at the point where your estimates are kind of factoring in you writing tests and that your test like you're writing code in a way that you're thinking about adding tests I think that's something that would happen over time that's my recommendation is to kind of don't say tomorrow starting tomorrow I'm going to get this perfect but instead go over the next couple months I'm going to try to refine this estimation and make sure that I can add uh tests in and feel more confident about my estimates I hope that helps I realize it's not a perfect answer for you though um choose a reality oh I already saw already saw a nasty word in here how do you feel about the TD argument that the tests themselves lead to a more solid design literally I'm also more pragmatic like you though yeah I think um for the tdd folks um I think for the tdd folks my challenge with um primarily with almost anyone talking about tdd I realize can't generalize like this but almost anyone I see talking about tdd says tdd is the only way that you can do things and I just say no and and that's that's an argument to anyone that's why I tried to start this stream off by saying I hope you don't hear what I'm saying as Nick says this is the only way tdd can be absolutely awesome I don't use tdd most of the time I have some scenarios where I'm like tdd works really well for me so I use it and then people will say well if you're finding scenarios where tdd doesn't work well for you you're doing it wrong so okay that's nice I'm still getting my stuff done and I'm happy and I'm confident in it so okay like I don't know um but I I like to write code that can be testable and I feel confident that I can write test for it and that way if I need to I will so that's that's my my approach to it so if you're a tdd lover I'm sorry but that's my take uh because I try to be pragmatic about everything um good on Tik Tok says why even estimate its ceremony focusing on as you say working code yeah I um I I didn't get into estimates uh much um but going back to some of the startup days that was one of the first things that I dropped on my teams was estimating things because that was the absolute biggest waste of time I have ever seen in my life um to be clear that doesn't mean that I think a conversation around talking about how long we think things will take is useless what I think is useless is arguing about if something's a five or a seven or a three in terms of story points when during that time you probably could have just made it that's the ceremony part I think the part that I actually liked about talking through estimates was like are we on the same page for what this thing is it's not it's not the estimate it's like a conversation about like what are we actually building here what's the goal those are really good conversations I think for people that are doing uh that need some a bit more predictability in work I think forecasting is a better way to do things so you just remove the entire let's estimate and pick story Point cards just use the history there you go there's your forecast keep going doesn't matter um it's going to be wrong anyway so um Regular communication keeping people up to date I think that's the best way to do it forecast if you want to use prior knowledge pardon me keep coughing I don't know why um but yeah I agree why estimate I don't um Mar Marcel sorry if I have your name wrong hi Nick from Barcelona awesome you were sleeping what about tdd thank you for sharing nothing about tdd we don't talk about that here uh no I was just uh answering question from choose a reality so yeah uh tdd I think time and place for it for me it's on all the time some of the time uh to give you an example where I find tdd works really well for me is uh so I gave you an well depending on when you joined uh when I was working in digital forensics when we had to write things to parse um unstructured data so give you an example you want to recover a picture from a hard drive it's not as easy as going to like C drive go to this path get the picture because what happens if someone uh has quick format of their drive or you're trying to recover stuff from a page file or it's like basically a part of memory right um how do you get that picture what if it's not the full picture so when we look for um when we look for unstructured data um it makes it trickier to go basically create like a parser to restructure that data and I have found historically that I like tdd for that so I have my test data to start with I will come up with like a you know here's the input and here's what I want to get back out make my test cases and go design things to go work for that um but I don't like um I'm not a huge fan of of having test structure everything for me it's to me it's not the way my brain works and I would say for people that like to do it very good for you excellent I'm happy for you but it's like anything else I will never tell people this is the only way um why did the engineer estimate because the producers and stakeholders said we had to yeah that's um basically it why estimate because we don't have budget I like tdd for validation and exception testing as well it's like uh you know I hope that this is actually one of the main takeaways that I hope people have when they watch my content is like um is that hey that was that was an interesting perspective or insight and like thank you I will consider that that's all right I don't want you to what I don't want people to do is to leave a a live streamer to watch a video of mine or read an article and go oh some some random dude on the internet said this is the way it's done so I guess it's the way it has to be like please don't um please critically think about the things I'm saying or what other people say um it's just just information and experiences uh I am sure you're all very smart people um hopefully smarter than me and you can take information and you know make good decisions with it that are your own I just want to give you some of the data points from my experience and then you can do better with it than I did so I think with that said I'm going to wrap up this stream though so thank you so much for being here uh this is uh where I do the kind of uh pushy thing where I advertise stuff to you so um if you join the stream if you miss kind of the beginning um and choose reality says I get worried when anyone starts to get dogmatic about any Silver Bullet honestly yeah that's the that's when you know when someone's like this is the only way it's like okay can't trust anything you're saying this is the link to the um to the article where this conversation was based on so again depending on when you joined if you're interested in architect is asking the question I'm about to answer uh next live stream will be next Monday 7:00 p.m. Pacific so exact same time uh I don't have a good spot for my coding live streams right now uh I want to revive those I have basically a project at work that is taking up one of my mornings and not complaining about that but it means my my kind of free time to do live streams is uh is going to interfere with things like CrossFit and stuff so like I'm just kind of making a decision that like I might have to pause those for now um I uh I do not stream on code commute so I'll share that in just a second uh architect has all the good questions but yeah the the link I put in the chat is my newsletter if you're interested in these live streams and you want to know what I'm going to be talking about it's just weekly. deev L.C if you think newsletters are dumb and you don't want to subscribe to a newsletter that's free that's totally cool I'm not offended I understand I get a lot of email I think I have like over 990,000 emails in my inbox at work get a lot of email um so I get it but if you just want to see the article and know what the topic is you can just go to the website weekly. deev leader.com so this is my second YouTube channel um this is the one that architect is asking about uh aside from these top three that you'll see here uh basically they're almost all done except another couple there these are all done when I'm driving to and from work or going to and from Crossfit uh so I I make Vlogs I talk about software engineering Concepts very much like this live stream it's just that it's not streamed so to Architects question um I don't stream on code commute this is the channel That I stream on and I make um like tutorial videos on and that's my Dev leader YouTube channel so let me put code commute in the chat as well for folks um most people I assume probably come from um this Dev leader Channel that's how they probably recognize me put that in the chat as well but Dev leader is you know I have my my podcasts on here where I interview people that uh have gone through software engineering and have unique stories to tell uh and then tutorials if it's a programming tutorial it will be in C because that's my language of choice um and then I have a bunch of other stuff too um I'm trying to see if I scroll through you can see a bunch of stuff on testing uh but other conversations are kind of like General software engineering kind of like this live stream um and in fact I have a couple of videos coming out between this week and next that will be uh both tutorials and some of them will be like software engineering Concepts so check that out if you're interested and finally the last kind of salesy thing I have is that I do have courses available um so couple things to point out if you are looking for more like Career Development courses um I have courses on dome train let me put them in the chat these ones that you're seeing at the top are uh for General software engineering career sorry they're kind of mixed in so soft skills getting promoted and nailing the behavioral interview uh and then the other ones are in CP and.net so if you just want to get started programming in c there's actually a bundle where both of these are available for a discount um this is Dome train this is not my site um but this is where my courses are posted so there are tons of awesome courses on this site uh if you're just looking for learning uh a lot of net development everything you could possibly want is here so I'm I'm proud and happy to have you know the getting started in C stuff and the Deep dive um I'm proud about that because like that's my I feel like that's a really cool opportunity for me to give back to people that are just getting started so um yeah architect says I came from code commit somehow you pop in my YouTube recommendation ah there we go perfect and I I appreciate it so thanks for being here but yeah courses are available um and of course if you guys you know you're like hey I don't want to buy your stuff that's cool um I appreciate you being here um I post tons of stuff online uh whether that's like on LinkedIn for for text posts memes like on but I'm assuming if you're here you probably have seen my content and stuff online so uh I do appreciate you you you supporting um doesn't have to be monetary with courses or anything else just thanks for giving me the opportunity to talk through this stuff with you so with that said I'm going to wrap this up it's the last live stream of the year so thank you so much for an awesome year um we got a full year next year so I hope to see you more but yeah I hope you all have an awesome New Year's and uh wishing you have all have happy celebrations so we'll see you next Monday for the first live stream of 2025