GitHub Senior Engineer: How The Best Software Engineers Lock In System Design

When you work professionally as a software engineer, this is not practicing a hobby. You need to have numbers, not just fluffy words. What was the actual landed impact on the business? Sometimes, solving business problems means building software that is good enough. Good enough for today. I just want things to be simple. Simple is complicated enough, especially at scale. System design. A skill that separates good engineers from great ones. So if you want to level up as software engineer, this conversation is for you. Joining me today is Basem Dekaii. Senior software engineer over at GitHub. And especially at GitHub, they operate at scale. He shares how they design their software and solve specific problems in a way I didn't expect. So enjoy it. From an outside service, I feel like I've shared this many times on the podcast. But I feel like the tech community is quite unique. People create software, they create products and they put them out in the open for free. And GitHub is the backbone that enables a lot of these things. I feel like I don't see in other industry. And it's typical for me to say that because I'm in this industry. But I don't see any similar other industry where this is possible. Precisely. And not only that, the teaching aspect that we have, how openly we share, what we learn, and we have this status associated with people who actually teach in our industry, which I don't often see elsewhere, a lot of industries. And I worked in other industries. I've worked in container terminals, for example, for a while. We were building the whole infrastructure and whatnot. It's very rare to find information about how a container terminal operates, unless you've went to a specific program that teaches that, or taken specialized courses. You're never going to get that information anywhere. There are no blockposts that describe how things work. You just got to learn it either from people who have been doing or practicing this industry for a while, or you go do a specialized program. So I definitely relate to what you're saying. And I think it's pretty unique. Which is awesome. There are some topics that I have in mind, especially that I think are going to be a bigger focal point going towards the future. Something like system design, where there's a lot of theory out there, and then actually putting it in practice. You can understand the theory, but then putting it to practice is still very challenging, especially in what type of organization you are, how far in the future are you going to design? I feel like it's challenging bridging the gap between theory and practice for that aspect. How to present? And I was thinking about this topic earlier today, right? I was just reading a post on X that went viral, where a startup was talking about the migration to Kubernetes, and then it almost cost them their startup. Oh, well. The startup could have almost failed because of that particular migration. And the owner was listening, the reasons why they went for the migrations, and none of them were technical. The reasons were their VCs were pushing for cloud native. Okay. Their engineers wanted the Kubernetes experience, and they thought that they can deploy faster, bigger, scale better. But what they ended up with is just a FATTER AWS bill, and then a much more complication, features shipment, crawl to a halt. They had to dedicate certain engineers for this. And I think this stems from the fact that a lot of people want the status associated with fancy architectures, and they want the experience of working with fancy architectures, and not necessarily solving problems with fancy architectures. Or growing organically a certain architecture to solve a particular problem. And the problem with system design is, again, a lot of people want the fancy stuff, but the boundaries for when you need to scale are very blurry. There are no concrete empirical numbers that tell you, like, when you hit this particular scale, you need to do this. And when you hit this particular number, you need to do that. So people assume that as soon as they hit on their service or API, or whatever, like a thousand requests per second, that's a lot. So now they need to build for the next evolution of the system. And often that's not really the case. At GitHub, for example, I've built services that handle millions of requests per second that are just simply running on five or six containers, and a very tiny Kubernetes cluster somewhere. That's not what I would expect at all. Precisely. There's a lot you can do with very little, and you can even run entire services on very basic VMs and scale some stuff around it. And the way we approach problems of scale, we'd never design or over-engineer. So when I design certain things, and I recently did a lot of that, we were rewriting some parts of actions, GitHub actions, that's what I work on. So we're going to talk more about that in engineering, block posts, and what not. This is like, in my opinion, a case study for decades to come, because this doesn't often happen. That's awesome. Yeah, so when we were designing this, we were designing for scale, but we know already we have established scale. So we know that we've hit the limit of what the previous architecture could do, and now is the point where we need to squeeze more out of a different tech stack to get to that next level. And we know that we're going to get to that next level, because we see the demand, we see how it's growing, we see the curve, we see where it's going to be in five years from. So we solved for that. We didn't prematurely scale. We just went all the way till the end with the existing architecture. Okay. And that's when we made the decision to rewrite some components and redesign certain aspects of it, and make trade-offs based on data we had, and based on projections that we're going to attain, forget to. Is this the new advice as well? So to stick with what you have, start with an initial design that is simple, and then stick with it until you actually reach certain limits, and you can kind of based on your usage forecast what the next iteration would be. Precisely. If I was a startup founder or a CTO of a startup, right, I would never go and start building for 100x scale. Even if I'm a firm believer in my idea and know that it's going to reach the masses, and it's going to be global, I would never design for that. I would design for getting 100 users, or maybe even a thousand. And I would just run everything on a single VM, to be honest, maybe two, make it a bit more available. I would not go and build massive database clusters. Yeah. Just start with one, maybe two nodes, tops, replication, that's it. And this will, like, carry you really, really far, to be honest. Once I attain, or I'm close to 80% of these limits of what my resources can do, and by the way, vertical scaling can take you a long way. Like, you have no idea. We have my SQL nodes that have hundreds of CPUs on them. So you don't really need to go and start sharding and doing all sorts of fancy stuff when vertical scaling can still carry you. Right now, with cloud native solutions, you can go pretty far. Like, you can get a 1vm with 120 CPU cores, right? You can get terabytes of memory. Yeah. And that can do a lot for you. You don't need to start doing a horizontal scaling before you get. You really start hitting these limits. Interesting. See, I feel like I'm not great at system design. It's just not something that I specifically put my thoughts to, not something I prepared from an interview style. And that I actually did a system design interview. And my architecture was, it was very simple. And I wanted it to be simple, because I don't like complexity when we don't need it. And it was also for the use case. I was like, this is the simple aspect. And then the question was, okay, but what if we hit X amount of scale, and then we need indeed, we need high availability, and we need sharding, and we need consistency. And then I was like, the argument that I had is, in this case, it actually doesn't matter. For me, the part of the benefit that we have a lot of information out there, if people can self-educate, is not also turning into this pitfall where complex system design is almost, I think, in most industries, where we don't actually have high availability across the globe, is not needed. Yeah, people see that, and they look up to big organizations like GitHub, where they say, okay, this is what GitHub has. So we're going to do the same. We're going to build for the future. 100%. And this is, in my opinion, again, going back to the original thought that people's chase status. So, and this was honestly also very badly perpetuated by big tech companies, right? And a lot of people followed. Like, in an interview, for a place like GitHub, for example, it's understandable that they would want you, or that they want to explore problems of scale with you, just because from day one, you're going to be exposed to these problems. So what we are looking for typically is people who have that experience, who are going to come in and actually use it on day one. Yeah. Not that we might use it in a few years' time. We're just hired for what we need today. Because let's say you're shipping features to a service, and a lot of our services have hundreds of millions of requests. Short periods of time. So you cannot really do a lot of the, you need to think about a lot of things, right? You need to think about caching from day zero. You need to think about your queries from day. You need to think about your data access patterns. You need to think about non-relational databases and non-relational solutions for whatever you're building. You need to think about how you're going to gradually deploy this. You need to think about future flags, how they're going to play in this game. You're going to think about authorization, authentication, all of the security problems that might happen. When you have introduced security problem at what's some other place like GitHub, it's going to be on the news in a week or less, right? So you cannot make these mistakes. It's very tricky. So we look for experience in these domains so that people, they still have this room to learn, but they need to deliver from very early on, right? For smaller companies, they interview for the same, Bye. They don't need it. And that's where the problem stands. If I was in a startup and I'm doing system design, I would want to hear from a person who has exactly what we need for today. If we get to 100 million tomorrow, we start hiring for tomorrow when we get to it, to be honest. And with the churn rate in our industry being two or three years, like people don't stay in the same job for more than that, why would you hire for the job? For the next 10 years, a person. Yeah. I feel like people really underestimate how much they should or overestimate, how much they should design for the future. Like I mean, a consultancy right now I go from assignment to assignment. And some assignments that I've been in, the system is way too complex for what we had. We used to have 30 engineers, now we have eight. And the system we architected was maintained by 30 people. And now we have a problem because we only have eight. Because budget is running out. We're behind on KPIs. Like we worked on a big rocket ship while we actually needed a car. We needed to drive and actually make meters. And I talk about this stuff in my system design course because in consulting and when you sell a solution to a company, the company wants to pay for that solution once. They don't understand that software is evolved. It's not built. And the term here is evolving it. So software needs to continuously be evolved and it needs to continuously fight entropy in the sense that you continuously need to maintain it. There is a fixed or running cost for maintenance of software that you don't necessarily find in other industries. When you buy a car, there is a cost for maintaining it, but it's not very regular or as regular as software. In our industry, the maintenance cost is pretty high. So companies want to pay once for a solution and they want it to scale to the next 10x, 100x. And that's not very realistic in our industry at all. We need to build it literally for the next order of magnitude. So if we are at zero today, which is built for 10x or 100x, if we are at 10x or 100x, we build for the other order, right? 1,000, 1 million, so on and so forth. What is then your perspective on building systems that can evolve? Because some people, you can say, okay, this is the system design that we have. And we're going to continue on this until we reach a certain level of scale. And then we're going to revamp and say, this would be the next iteration. But some people also say, okay, we have a system now and this is evolveable to whatever scale we need. And then they put in the extra effort to make sure that that is there from the beginning. In our industry, very few technologies last for like a decade plus and the rest is changing often. And that's why I personally don't invest or I wouldn't design a solution that will last like 10, 20, 30 years, unlike what we did in the 80s or 90s. That's how that was the design mentality, but things were way less complicated than they are today. And the scale was different. So I would always come with the mindset that I'm just going to design for the next order of magnitude. And then I'm going to evolve it from there. And that's going to require another round of investment. There is going to be continual investments in this. And that business people are not going to love this because that makes accounting much harder. That makes strategic finance much harder because they cannot predict or anticipate the costs that might come in the future. They cannot extrapolate from today. You know, they don't like this fuzziness of software. It's all risk. Yes. And also they cannot calculate their bottom line or projections for the future, which are things that are very realistic. Like public to trade companies want to be able to do these projections and they want to be able to say that, hey, we're not going to be spending more here on evolving our software, but we're going to make much more revenue and increase our bottom line and margin. That's what businesses look for. So it's understandable. But from a software perspective, there needs to be regular investments. And I think the only approachable way, which is something we do at GitHub, is to continuously revisit these investments on a quarterly, bi-corturally, half a year or a year basis. We can make projections for the next three, four, five years, but the confidence in these projections is less and less and less, the more time grows. And I think a lot more companies need to start doing the same. When you say we will then design for the next level, the order of magnitude for the next level, how far in the horizon do you look at? With the GitHub actions example that you had, you said, okay, five years, but is that like an arbitrary thing or how do you depend on the curve, right? It depends on the curve. GitHub actions has been growing massively over, since we brought it up at the beginning. We're introducing more and more features that are increasing the adoption. So the curve, we can see how it looks like, but it might change at any point in time, right? It can be disrupted, it goes to zero, or it can continue growing exponentially. So often the hockey stick that a lot of tech startups talk about, like exponential growth, it can appear and you can start seeing it. And when you start seeing it, you know if you have a solution like actions, companies that migrate to it are not going to flip flop. No. And very short periods of time. You can have some confidence that, okay, this trend is going to last for a year or two or three or four, right? Nothing lasts forever, obviously. But you can start thinking in that direction. And then based on that, you can make investment decisions. Gotcha. I've had many discussions also in the past when we're talking about our software architecture in something that can be very specific and also purposeful built versus setting it up in a generic way, potentially towards the future because we have ambitions and hopes that we have X amount of solutions that are similar so we make a generic right now. What is your perspective on making something specific versus generic in the first place? I like solving specific problems. I don't like solving generic problems because how can you make trade-offs then? Right? Well, in software there's always trade-offs. So how do you choose these trade-offs? You can build something super generic, sure. That basically is what building a framework that can be a toolbox and give you all sorts of solutions to all sorts of hypothetical problems that you might or might not have. That's not the North Star I follow when I design software at GitHub. With my team, we are very specific about not over-engineering things and we solve for the problems we have today. So for example, we'd never introduce caching until we really needed. And we never over optimize until we start seeing problems sometimes. Because we know that we're going to hit this problem a few months and we're not going to get the investment so we solve it right now because it makes sense. But we don't over-engineer solutions. We don't start jumping into things. For example, we don't necessarily jump to no SQL at the beginning. We just start regular relational databases. We build regular tables. If we need actually relational content. Sometimes relational databases can only scale so far. So we start needing to three-think our solutions here. Then we need to start going towards non-relational. Even if we need relational consistency or the consistency of relational databases. We need to start thinking about how can we move that to the application layer and start using other solutions that persist data that can help us scale. There are some efforts, for example, where we start using cosmos or Cosmos DB and things like that. But we never start exploring these until we hit really massive problems with our database clusters. Our primaries cannot keep up. I always pre-high. Right operations are substantially big. We cannot scale these anymore. Sharding is not an option. Application layer sharding is not an option as well when it becomes not an option. Sometimes it is. So we start thinking about how can we redesign this and how can we migrate the data. And migrations as much as they are painful, they are part of the job. Yeah. Interesting. And we do go through them. Considering when we're talking about system design and then specifically practical experience, what your perspective is on someone to really get a better understanding of the theory, but also be able to put it in practice. You already mentioned that, for example, like GitHub, you need to have a certain level of experience under your belt. And then you can work at GitHub where we have orders of magnitude with regards to scale. This is the vicious cycle, right? So how do you build experience that is required by this type of companies? But there's no other type of company that's going to give you that experience. Unfortunately, that's the reason why there is a proliferation of a lot of system design courses and things online. And quite honestly, passing the interview threshold for companies like GitHub is achievable by learning theoretically these concepts. Okay. And not necessarily having hands-on experience with them. The bar feels high, but right now it's been sort of exploited to the point where a lot of the tricks and tips and whatnot are already available very much online. Like hello interview guys. For example, they have this YouTube channel. They have pretty much covered a lot of the topics that you might get asked in a system design interview. And they go deep like they explain what Kafka is, what event streaming is. They explain right as they internals how it works. When you use it, when not to use it, they explain the differences between relational and non-relational databases. They explain caching and how to implement caching in specific scenarios. And they give practical hands-on problems where you can actually learn how to effectively utilize all of these different components to come up with a relatively scalable solution. And they give you tips also on how to approach the discussion of scale within a system design interview. And I think they do a pretty good job at it. And I think that's enough because a lot of people who go to Big Tech will go to Google and want to they don't necessarily have that hands-on experience beforehand. Even though the interview is looking for that hands-on experience, but because we hacked the people hacked the game, so the signal is given to the interviewer. And then people, when they have this foundation, they can actually do a relatively okay job on when they join. And they're going to obviously aren't going to be alone, right? So you're never going to join a team and start building Greenfield stuff from day zero. It rarely happens. You're going to join teams that are already established. So there's going to be people with experience more senior than you are going to give you guidance and review your solutions, your designs, and so forth. And that's why people make it, right? If you are interviewed for experience and you join a Greenfield project, you are going to own your mistakes. People do make mistakes often. Yeah, when it comes to these solutions. Not everything big tech ships is a great feature or it operates well in the world. And often we discover that, oh, this was a mistake and it was actually very costly and some companies fail because of these reasons. Any failures or mistakes that are top of mind for you that you can share for other people to learn? No, I want to keep my job. That's fair. For me, looking at what we do nowadays, I feel with AI tools and we're more productive in actually writing code. And I feel like when you amplify the amount of code that you write, you might also write yourself into a bottleneck, basically. So I feel like software architecture and specifically system design from the start, having a better understanding, especially I feel like like I do now, I need to grow there. And I feel like it's going to be a more important skill to indeed figure out what are we designing for, what makes sense, what would be the next order of magnitude after and actually make sure to convince people and have a certain level of buy-in. We're always working in teams, also from a business standpoint. How would I, for example, say, we're going to design for this until we reach scale and then we're going to need to revamp and this is what that's going to entail for people that don't understand how to build software. Understand the business constraints. So you're never going to be able to convince the business folks of something technically. There's just, it's too abstract. Even if it feels too intuitive for us, for them, it's magic. And even if they try to understand how things work, they're never going to comprehend the magnitude of problems and they're never going to have this intuition about how software is and how not hard, you know, how, how, how available it is. So it's our job to get closer to their side. Some people say, no, we want the business to understand they're not gonna. It's trickier. It's easier for you to acquire or assimilate some of the, not everything, some of the business constraints and discuss them from that perspective. And that's why a lot of engineers who grow are very good at understanding the business constraints and then discussing the technical solutions within those constraints. So for example, I used to work in the container terminal industry for a while. We built a huge container terminal. We furnished everything from the fiber optics layer all the way to the three-tier data center with all of the enterprise applications running on it and the operating system for the container. And it's one of the most massive operations you can ever imagine. One of the most hazardous as well. So the folks there, they're as far away from tech as possible. You're not never gonna find the terminal operator who's gonna understand what it means that your database is running hot and you cannot support their queries. All they care about is I have a ship here who's docked. There's like 10,000 containers on it. I need to empty them into my yard. I have a very fixed window of time. And if I don't do it in that window of time, this is gonna cost me money. We were operating at 12% capacity. That terminal was generating about $50,000 per hour. So 12%. So you can imagine if we get delayed, every minute of delay is actually money that's disappearing. So this is how you need to discuss problems with investments and solutions. You need to understand what the impact is, what the impact is on the operators. And sometimes it's life-threatening impact, right? If your software fails when a STS, when the crane is offloading a container, this can be a very life-threatening situation. You cannot really have your software malfunction at these points. So snapping of the software, sub millisecond latencies, all of these things matter, right? So you need to approach the discussions with the stakeholders by understanding them first. So what I've done, for example, in the industry is I've literally sat on a truck and went there. And I sat with every single individual who works in that terminal to really understand what they go through on a daily basis. So a truck driver who's driving a truck for six, seven, eight hours, and it was in Saudi Arabia in scorching heat is not going to understand why the button that they clicked is just is so. It doesn't matter. It doesn't, he doesn't care, right? He just wants it to just work. So you got to understand all of these perspectives. And then when you do, you can start approaching discussions from that vantage point. And this applies to banks, this applies to hospitals to any other industry. So understand the industry, understand the business constraints. Yeah. And then any technical solution you want to discuss and investment, you can approach it from that vantage point and you're going to have much more success in these conversations. Yeah, interesting. I feel like the opposite then also holds true. So if you do understand the business context and you don't have a case for why you need to work on tech, that way you need to scale, why you need to fix some of the issues, then there might not be an actual case. And right now is not the right time to do so. Correct. I do feel like it's, it is a lot of prep work that you have to do. It's way easier to talk about the technical things, right? Because that's what you see, that's what you perceive, latency, numbers, storage, scalability, that's what you look at. And then reframing it and saying, okay, what is the impact for the business? What is the cost per feature and what if we have a slower delivery rate compared to a faster in a new future? Talking that language is very different from what you're used to. Absolutely. And the trickiest part in my opinion is when the stakeholders are not transparent about their numbers. So they're never going to tell you like, oh, we're making this $10 million investment today in this piece of software because the investor, one of the biggest investors we have is willing to put that money in right now. But nobody has the guts to go tell him that, no, we need another 10 million in 10 years. Right. And nobody's going to come in and tell you that. But you need to also think about it, try to extract that information, try to understand who's funding this. What is funding this? What is critical? Is the business in jeopardy? What is driving this particular investment right now? Is it more strategic? So I worked in building banks, for example, in the Gulf. I worked as a consultant, that's right. There were banks that wanted to make a one-shot investment into what they thought it was their future. And they had a lump sum of money that they wanted to put down once and build software that will last them for 10 years. They're not going to, they're not going to, nobody has the guts to come in and tell somebody an executive who has no time whatsoever. Like, hey, well, this investment is going to last you maybe like two, three, four years, but you need to put in also another million every year to maintain it. And maybe even another 10 million and five, six years because you need to keep up with the new trends. Nobody has the guts to come in and say that at the beginning because then the decision maker might say, oh, well, this is not worth it for me anymore. Yeah. We might just scratch it completely. Precisely. What I've also seen is that some organizations operate with a number of engineers and then from an outside standpoint, I have no clue what those engineers are doing. Like if I look at a specific e-commerce website, then I have 2,000 engineers who are kind of a similar one with a similar level of skill might have a few hundred. And then I feel like the few hundred engineers, they need to have a bigger landscape of ownership, meaning what they do in the context on the day to day on a business side is easier for them than being part of a group of 2,000 engineers that kind of does the same in scale. If you're like, if you're a cog, it's very hard to argue. What kind of optimizing your little cog is going to do for the greater good? One thing I'll, one thing a lot of people hate, but I think it's very functional and a lot of the companies that operate based on the San Francisco Silicon Valley model is the ruthlessness in terms of optimizing how engineers work and layoffs. So in Europe, layoffs are not very famous or very well received. And I don't think they are very, very received in the US as well. But they are an effective mechanism, even though it's dehumanizing, but it is an effective mechanism to recalibrate how people or where people put their attention and what they are focusing on. So in a lot of situations when layoffs happen, there's huge product features that are cut entirely. Like, we don't need this anymore. And it's more expensive to repurpose the skills of those same engineers and other areas. So let's just cut it off immediately, hire new engineers that are more experienced in that particular domain, get them to focus on this right away. So this rapid flexibility in moving and rotating resources or people basically allows these companies to be much more nimble and effectively or more effectively allocate engineers to particular problems that matter today. The second thing I also think is a good that separates a lot of the big tech companies from other companies is your work or your reward, bonuses, growth, whatever, is associated with your business impact, not your software engineering impact. So you can come in and say, oh, I built the biggest Kubernetes cluster known to the planet. It's beautiful. It's amazing. You're not going to get any of that or any recognition for it unless you justify why that was important. So what did it contribute to? in terms of our revenue. Did it help us grow? Did it help us attract more customers? Did it solve a very hard problem for us? Did it affect developer experience? Are people more. And you need to have numbers, not just fluffy words. So I think that's also another very effective way to reallocate attention of software engineers. And it's very easy for us also something that needs to drift into the comfort zones. So I'm building something I'm comfortable in it. Yeah. Nobody's questioning me on it. Nobody understands what I'm doing. I'll just keep doing the same thing every day. Yeah. I feel like I've also shared this previously on the podcast. Some engineers that I've met they love the technical side of things. And when you let them go wild when there's no business context or direction, they will build you a rocket ship and they will really enjoy doing it. There is a craft. the craftsmanship aspect of software engineering. It's still there. You know, there is an aesthetic to it. It's fun, it's enjoyable. But I always, in my even my social media stuff, like I always strive to communicate that. And I think a lot of this craftsmanship thinking comes from the fact that a lot of people went into software engineering because programming was a hobby for them. Myself included. I started writing code around 12. I thought that this is amazing if I can make my job perfect. But the job is very different than practicing code for the sake of writing code and the aesthetics and the fun of it. Sometimes you are lucky to be able to bring that into your work. But most often, you're not really solving for the aesthetic. You're solving a business problem. And you need to keep that in mind always and often. It's not about the beauty of yourself. And if you can build a beautiful solution and solve a business problem effectively, fantastic. That's the cherry on top. Inwind. But the priority is for the business problem. And sometimes, solving business problems means building software that is good enough for today. Right? It doesn't have to be good enough for tomorrow. Maybe it's horrible for tomorrow, but it just keeps on functioning. And this investment we did in the good enough is going to give us a certain runway and we should be okay with that. I think though we should be okay with that. That's what I've seen where it's challenging for people. Where they say, "Okay, I'm a little bit of a perfectionist. I value quality in what I do on a day to day. My job is basically part of who I am. And if I don't deliver the highest quality that I can in this aspect, like I'm doing myself with this justice." And it's very, like if I laid out like that, it might sound selfish, but those people have no clue that what they're saying is selfish. They genuinely think it's best for the organization. There's just some misinformation and misunderstand. Because there's a dogmatic aspect of software engineering. And it was perpetuated in the previous era until 2000s, 2000, when I ventured in working professionally as software engineer in 2007, '89. There was a lot of dogma. Like there was a lot of, "This is how you do things, and this is how you should do things, and if you do things any other way, you're wrong." Yeah. And part of that is elegance of code, code, cleanliness, code quality. You were always on these concepts, right? And they're important. And everybody always complained about bad code and horrible code because it was difficult to maintain. Nobody enjoys that. You're going to make it very difficult for the future engineers to solve these problems. But sometimes, you don't have future engineers to solve these problems. Sometimes nobody's going to touch this code for decades to come. It's okay if it just works. And then we ended up on the complete other end of the spectrum where we tried to clean up everything that we started adopting all sorts of design, overcomplicated abstractions that are just horrible to read, horrible to reason about. It takes you decades to become proficient in just reading that type of code. And then when you write it, it gets even more complicated and it's very difficult to maintain. And Big Tech, and I always tell my colleagues and the juniors that report to me, simple is complicated enough, especially at scale. So just write it in the dumbest, most simple way possible. This is why I also love Go, for example, as a language. We do. That's amazing. It's dumb. You don't have to think too much. It's just all in your face. Whenever you want to understand how something is built, just trace it around. It's there. It's there. Fantastic. I love that. And in the situations where we're building at scale, I don't want to reason about abstractions. I don't want to think about edge cases. I don't want to think about complicated memory allocations and complicated memory structures. Memory leaks happen often and they are very catastrophic at scale. I don't want to think about these things. I don't want to think about how my garbage collector is going to shut down the whole world and start causing multi-second latency on my API requests, which at scale again become a gigantic problem for me. I just want things to be simple, easy to reason about. And that's it. Yeah. Interesting. I also feel like we are now in a trend where there is this conversation happening with people that are, first of all, vibe coding, loving it, being effective in what they do, solving a problem through software and they don't care how it looks. And then you have the older part, let's say, existing software engineers that look at this and say, both from a craft perspective and from an execution perspective, how does this work? This is kind of really coming into my territory. I feel uncomfortable. Do I jump on this? Do I actually like it? Or is this going to take away what I love dearly? What I used to do day to day? I think in essence, things are evolving. For me, that's a fact. The question is, where is it going? And I'm curious to see your perspective on that. It's funny. It happens in every community, to be honest. So I'm starting my motorcycle riding classes. Okay. And in motorcycle riding, a lot of the more advanced motorcycle motorcycles, they love manual, the clutch, they love changing gears, they love the controlling the motorcycle in every single aspect, especially when they attain that level of proficiency. So a lot of them are now resisting a lot of the motorcycle companies introducing e-clutches or clutches that just operate automatically. They hate that. I don't want this. I want to control every single aspect of my bike. And I feel this is pretty much the same thing that what's happening in software. A lot of the folks who have attained a high level of proficiency and they enjoy the craft mentorship aspect of software development, they love the coding part. They love reasoning about code structures, syntax. They get a kick out of coming up with a smarter solution where they utilize a syntactical feature in a programming language to do something that was not maybe designed for or do something smart with it. So if you take that away from them, you're basically introducing e-clutches to software development and they're not going to react positively to it. Even though e-clutches help. Like when, for example, when you're stopping at a light signal, you don't have to worry about stalling and you don't have to worry about these things. It just makes your life much easier, fun to ride the motorcycle. It's just easier to control it. And I think it's accessible. Yeah, more accessible. So, vibe coding is pretty much the same thing. I hate the term vibe coding because right now at GitHub, for example, I literally, and it was part of my presentation at universe, 90% of my code is written by agents. So I use the VS Code agent mode all the time. Yeah. And I've been shipping features to production with it all the time. But that's not software engineering. It's not just writing code, right? There's a lot of other stuff that come to it. And I now shift my focus towards the operational problems. I shift my focus towards making sure that we never go down. I shift my focus towards making sure that I don't introduce bugs that are unnecessary and cause people to have pain because GitHub has become part of the infrastructure of the internet now. So any problem that I cause has great triple effects. So that's where I put my focus on right now. Coding, fine, let the agent write it. And I will instruct it to write it in a certain way. And if there's a syntactical approach that I prefer better, I still push it to do that. And if something looks funky to me, and I also prompted to change the direction, right? But at the same time, one of the problems I was solving was recently was a caching problem, right? And I had different ways to implement the cache key. And each way forced a certain data access pattern because of how the key is structured. And it had a different impact on redis based on how the key is structured. So what I've done with VS Code agent is I prompted it to write different benchmarks. Yeah. And I told it, hey, this pattern, this pattern, this pattern, go write three distinct benchmarks, run them, give me the results, analyze them and give me the output. That's work that could have been done in what, two or three days maybe. That was done in 20 minutes. Why not? Fantastic. It's amazing. Yeah. So you've been programming. You said hands on coding when you were 12. Have you always had this mindset to kind of let go of more of the craft things and look at what it is from an execution standpoint? Because I hear you from now. And it feels like you're very much on board and making things more effective, understanding the business context. It's not just the craft because no, I was never like that. So when I was younger, I was all about the craft. And in my early mentors, I fought with them a lot of them. And I was lucky to have mentors, right? I was lucky to have great mentors at the beginning of my career. But they never were able to explain to me why their perspective of the business matters. For me, business was too abstract. Like, I don't understand the numbers. I don't understand a counting. I was not accountable for the business outcomes. I was just asked to solve a problem. And I thought I had infinite business resources. But when I I realized that my mentor sometimes took an extra mortgage on their house just to pay the payroll for their employees. Oh, well. That makes a different problem, right? That gives perspective to the solutions that you're going to come up with. These are real people. Company owners are not just all millionaires and all people who are not investing. Some people are investing their own money in building businesses, right? So that matters. So when you understand this and when you become your business owner yourself, you start thinking about these things more. And now I approach these problems from that vantage point as well. That doesn't mean that, again, people might, I always get this argument when I post about this stuff online, people think, "Oh, you write dirty code?" No, I don't. But there's a certain level of the 20% that is required to reach that highest level of aesthetics is not worth it. So I drop that. Gotcha. As a final thought, there's many people listening that learn about software engineering and software design in the first place that either are early in career, mid-career, or really just looking to level up within their career. What is your advice for people that really want to become great software engineers? What do they need to do? Increase your breadth. There's a lot of focus on the depth early on in your career, which is important. And there's an advice that perpetuates which says, "Focus." True. Focus. But at the same time, don't dismiss everything else that's happening. Personally, I optimized one thing that really got me far, which is having the ability to learn effectively in very short periods of time. And that's why whenever I tackle a problem, I don't have a problem digging really deep into anything new I encounter. From quantum mechanics all the way to any technical problem that I face on a day-to-day basis. I'm a, you know, I'm a university dropout. I never finished my computer science education, but that did not deter me from picking up that education myself and really digging deep into all sorts of theoretical topics, but also practical areas and topics. And I'm a perpetual learner. So in my opinion, the next phase where we're heading is going to require people to be able to learn really fast, get out of their comfort zone really fast, dig deep really fast, and attain high levels of functional proficiency in any new area. And then attaining mastery is not always required in everything. So just pick a few topics where you attain mastery in, but also increase your breadth. Gotcha. I think this is going to be very important for the next phase. I love that. How do you get comfortable with even saying I've gotten really good at picking up new things and learning from quantum mechanics to anything technical, or job related, whatever? I have a lot of interests and I see how fast I can get to a working proficiency with these interests. I play music. I'm learning to write a more cycle soon. Yeah. Right. So I never let, if I'm curious about something, I go and pursue it and I explore it. And I never let how complicated learning it is, determine from pursuing it. And I've become very comfortable with being uncomfortable learning something new. And actually that gives me a joy and kick to the point where it became sort of an addiction. But I keep it under control sometimes. But I do enjoy the process of and exploring something entirely different and new. And in my opinion that has given me so much breadth and the ability to talk about all sorts of things. But what I love the most about it is how I can cross different ideas together from different disciplines and have a much more diversified view of the world. And that gives me the ability to be able to empathize also with all sorts of people and different people, right? And understand their vantage point. So when someone's arguing with me, I take a pause. I try to reason about where they're coming from. What's their interest? How they got to that idea, but I draw from my own experiences also in learning about all sorts of stuff, including business and finance and investments and all sorts of all of these things, right? So I think that's richness. And yeah, I would definitely love for people to have that. Me too. I think curiosity is a beautiful thing. When people say I really don't have a drive to learn something or to explore something, I think it's a shame. And I honestly don't know how to solve that for people to make sure they have that kind of love for something or curiosity for something because I feel like that fuels drive and that makes you move. Absolutely. I don't know if we really should make people do anything. I would love for people to have it. If they don't, you can live a pretty interesting life still. Some people have other commitments as well, families, kids, less time, complicated relationships. That life happens, understandable. That doesn't mean that I'm also 100% always on the burner, learning every single minute of the day. I have my life as well. I have the complications of my life too. So balance is important. But I feel also curiosity is something that you need to develop. It's like a muscle you need to grow as well. And approaching the world from a curious vantage point is way more interesting and more effective than being dogmatic or stubborn or hardheaded about certain things. You can have strong opinions, but not before you actually did your homework. Yeah, absolutely. Thanks so much for coming on and sharing, man. This was a blast. Absolutely. My pleasure. We can be moving for hours. We too. We're going to round it off here. If you're still with us, leave a comment in the comment section. Let us know what you thought of this episode and we'll see you on the next one. [MUSIC]