The Rewrite Trap

Hello everyone, my name is Phil, I'm a software engineer at Brighter. I've been a tech lead before and also a CTO, even though I got fired from that job, more on that later, and today I'd like to talk to you about a thing I call the rewrite trap. And first, before we start, let me set the stage. So what kind of rewrite are we talking about? It's definitely not the one where you know the project for months or years, you're super familiar with all the ins and outs, the tech, the architectural decisions, all of that, and you're absolutely sure what you need to do in order to get this off the ground. Maybe even there is a new project and you actively make the decision to not stick with the current stack or the legacy platform or whatever, but start from scratch. This is not what this talk is going to be about, because I think there probably the rewrite or the starting from scratch might be the best option. I'm talking about the situations where you might be the new tech lead in town or the new CTO in town. You join a company that has an existing product, there is a stack, there is software there, and you kind of have that feeling that after looking at it for one or two days, you just think that you're better off throwing away everything and starting over. This is what this talk is going to be about. So let's start with where does that urge come from? I think there are three main reasons. The first one is you don't need to learn new concepts. So let's for instance say it's a client application, it's written in React, but you like Vue or you like Svelte or whatever better. Then getting into React you immediately have that repulsive feeling that you really don't want to get into this. So if you throw it away and rewrite it in your stack of choice, then that would make that problem go away. It's the same for programming languages, methodologies, whatever. So everything that you're not used to, you kind of immediately say, maybe let's not do that. The second thing also is someone else built that. So if you really like a certain architecture, let's say for instance you really dig functional programming but the current product is built in a very object-oriented fashion, then those two worlds might not align so well. And it always would feel like a hurdle to get acquainted with everything. You might already notice there's a pattern here and the pattern is not that necessarily tech choices, programming languages choices, architecture choices are better or worse. It's just that they're different. And that intrinsically if something is different to what we prefer, we have to fight that urge to just throw it away and turn it into something that we like. A colleague of mine once gave me the advice to wait 100 days until you make any bigger technical decisions once you join a new job. Because in 100 days you can learn a lot and also you can figure out whether a choice from the past is actually worse than what you wanted to use. Is it a problem that it's object-oriented? Should it be functional? Or whether this was just your initial gut reaction to a certain thing and you just didn't like it before. But probably 100 days in, you also got used to it and then maybe it's not that bad anymore. Personally I have to admit I always shortened that period to 50 days because especially when I was CTO I couldn't convince my CEO to just wait almost a third of a year before I start making bigger choices. Let's just start up life and do what makes sense at the time. Now let's get to the third part and that's the biggest egomaniac part of it. If you just throw away everything you can essentially work the way you want. You make up all the rules so this definitely feels comfortable. The big question to ask here is just is that something that makes sense in the long run? Is your way of working for instance compatible with how the company works? All of these things that I just outlined feel initially like good reasons to start from scratch and we can definitely talk ourselves into the benefits of either one of these but the question is do they really pay off? I think they rarely do. This is also where the trap comes in. Because initially if we look at these projects, I've drawn two made up lines essentially. The green line representing an existing software project and you don't rewrite it but you try to gradually improve it, work your way through the project and make it better. The purple line represents the complete rewrite and the X's are just time on the X axis and the Y axis is outcome like how much can you actually achieve. And obviously if you start a rewrite nothing's there, nothing's holding you back. You can push out a lot of stuff in the beginning really fast. So you could even then fool yourself into thinking, wow, this must have been the better choice. We're so fast, everything's moving fast, that's great. While the people who would stick with the existing code and try to gradually make it better first have to go through the weeds, learn the architectural choices, see what's there and gradually improve the running system essentially. And this is where the trap comes in really. That false sense of productivity in the beginning is what I think leads people to making those rewrite choices. And I give you a hint, maybe it's also good for your career to do this. Because I don't know how long the phases of 1, 2 and 3 are, but if you leave in phase 1 or at the end of phase 1, all you've done so far seems very productive. You've done a great job. So you might be the really good tech lead that just gets things done and that just does the things no one else wanted to do and it helps you progress. Maybe do that if it's good for you, but I'm not that person. But let's look at all these things in detail and start with phase 1. So I've already said the rewrite will be off the ground really fast. You crank out the first features, you do the basic stuff, have a sign-up flow, have the first big features going. Everything's rough because you work iteratively and do all the fluff and you can have always something to present to people. Whereas the people who stick with the existing code probably have little to present because they need to understand everything that's there. They need to figure out what belongs together, how do things really work, where are the edge cases and all of this stuff. So in phase 1 the rewrite feels good and you crank out stuff. Improving what is there doesn't feel so good because it feels like you're making no progress at all. These things start to look different if you enter phase 2 because what the rewrite completely disregarded was the problems and the edge cases that people have found before. That big legacy codebase probably wasn't that complicated just out of sheer coincidence but probably for a reason. So our big rewrite at some point will hit that part where the edge cases come in, the quirks come in, and you will notice that since you didn't learn about all these things upfront and you didn't even bother to understand why the old system became the way it was, you made a lot of assumptions building out your new software that just don't really fit that well with those edge cases and the other requirements you have. So you got to start reworking stuff, you got to somehow need to crank in other things. But since you've been so productive before, people have that assumption of stuff's going to move fast and they want it to move fast and eventually you'll end up adding tech debt again. You got to cut corners, try to keep that momentum moving, but eventually what will happen is your progress will slow down, outcomes will go down, and you're just going to navigate yourself into essentially the same mess that the old legacy system was before. Whereas the people who took the time to really understand what's there and figure out how to make it better and improve it gradually, in phase two these initial efforts will start paying off because by reducing complexity over time, by really doing the necessary refactors in a way that matches the system, they can actually start untangling stuff, maybe pulling monoliths apart into clear modules with good cohesion, not like decouple a system where everything's coupled to everything. And they will actually start getting some momentum now. So it's going to be faster to add new things, change things, because the people working on the project or product will now know better what's in there and how things work. And this leads us to phase three. In phase three, essentially, the big rewrite is now the new legacy system. Because you didn't bother to understand stuff and because you tried to keep your momentum up and navigated yourself also into a situation where the outside world kind of expected features fast from you, you've built up so much tech that now feature delivery kind of grinds to a halt. And I mean, if you're lucky you've left the project before. If you're not lucky, then probably this is also the time where some other tech lead will replace you. And chances are that the story will then just repeat with another rewrite. But the people again who stuck with what was there and gradually improved it might now just reap the big benefits. They might now be in a state where the software is pretty good. They understand, like a super good understanding of what's going on and they can just make changes in the parts that are really important. And also have now built out an architecture that supports those changes and supports also the software for whatever purpose it was intended to do. And yeah, this is essentially the trap here, right? That the initial euphoria around a rewrite kind of leads to its demise and then to the next rewrite. And once again, some career advice, try to leave at the end of stage one, because then you're the cool person who just got things done. And everything that happens after is then mostly accounted to you not being there anymore. I think, oh, now my name is Phil. Now Phil is gone. So the moment he left, everything just went to pieces. But that we now know wasn't really the case. But you can use that. I don't say you should use that, but you could. So let's look at what should we do instead. And I kind of outlined this already before. But once again, three key things. First of all, if you join a new project, try to defer judgment as much as possible. Always assume that the people who worked on the product before worked with the best intentions and just made the best choices they could at any given point in time. We all know that the best choice with the best knowledge five months ago is probably not the best choice now. But this kind of sets the stage for a climate where you want to learn, where you want to understand why certain things are the way they are. And you need to do this to actually really improve them. Because if you throw out something just because you don't like it, well, you're probably prone to make the same mistakes again. So learning, talking to people and understanding why things are a certain way is the first key takeaway here. Second thing is that you need to understand, or there are going to be pieces that you don't understand. And that's okay. Some stuff might just seem super weird and you can't come up with any good explanation of why would it be this way. And this is where you need to go in and really have discussions with people and say, you know, I, you know, and be, you know, also be clear about what you don't understand, what you, you know, what you think, how it should be, and that you'd like to learn, you know, why it's not a certain way, you know, in some situations or in a lot of situations, probably people will tell you that we just didn't have the time to do it properly. We know that we should have done it like this, but well, you know, the deadline was approaching. But these discussions help you get along with people, also get an understanding and probably also you learn that people really, really have a desire to build the right thing, but you know, the system around them just didn't give them the support they needed to actually do it. And, you know, as a leader, tech lead, CTO, whatever, this is now your call, essentially. You now need to make sure that you create such a system for them where they can do the proper work. Right. And sometimes you also, you know, should also talk to people outside the development area, right. Talk to sales, talk to marketing, because, you know, chances are they also had an influence on your current architecture and you need to understand why certain things were important, right. If you're in a startup, you know, money needs to come in, right. So sometimes you got to, you know, crank out the one thing that just going to sign the next deal, because otherwise the company wouldn't exist anymore. These are all good reasons to make some, you know, shortcuts in tech. I didn't really want to believe that earlier in my career. Now I think I understand some salespeople much better. And this is also the third thing, really. Get to know the system, right. You're a group of people and engineering or R&D department inside a company is not its own universe. There are factors around it that influence the system, the CEO, sales, marketing, you know, all of them essentially combined. And you need to understand how the company works in order to navigate it in a way that you can actually achieve, you know, success. And this is not an easy feat. This also requires talking a lot also to people outside of engineering and also understanding their needs. Like what do people need? And I'll give you the example, for instance, you know, when I joined the CTO, I couldn't believe what I was seeing, right. There were no tests. There was no definition of what the product was supposed to do. We didn't have any error reporting. And the CEO was always asking me for a plan for when the features are done. You know, I don't even know what's there. So I can't tell you anything about when the next thing will be done. And I tried to do a lot of explaining and essentially it didn't work out in the end. But this is also me not completely acknowledging the system there, right. Because the system with the existing CEO and the existing customer, the chief customer officer, kind of led to this, right. And I didn't bother understanding exactly what their needs are and trying to also always cater for that. I went in there and just with a complete tech kind of view on all of that, right. To say, okay, I know all of this is bad. We need to do this, right. We need to add monitoring. We need to add some tests. We need to add some automation around releases. And then we can just work our way through the weeds. And essentially, I also actively did not rewrite, but improved what was there. And with that, essentially failed in their view because it was really slow. And essentially, I got fired when the curve just started tipping up when we didn't have more bugs in than we could actually fix. Or when we actually started delivering features on a predictable schedule or more predictable schedule where then funny enough, someone else came in and the CEO hired someone who did a complete rewrite and they were super fast. And then I essentially got the news that, you know, look at them in a month, they've built that and in three quarters of a year, you didn't achieve that. So you're out, you know, understandable decision now. But still, right. This is what I mean, get to know the system, get to know the needs of people so that you can also cater for the respective needs that people have and give them a proper story. Right. You know how or you know how I think you should work, but you still need to create a story for all the other people that they also feel comfortable with that approach. So in essence, the conclusion is do what matters. Don't try to be busy. But focus on the things that actually create value. You got to move out of your comfort zone. And especially with that, I mean, you got to talk to a lot of people and you got to try to also understand needs of people that are so foreign to you. For me personally, it was always sales. How does this, this is a complete different world. You need to acknowledge that you just don't have all the answers. You don't know everything and that's fine. Learning is, you know, as you probably know, the biggest part of the job. And you need to, you know, you're going to have to learn a lot more. And in the end, once you've figured all of that out, you need to rebuild the parts that really matter. And this is a talk in itself, and you know, you're lucky because I did that talk already. It's called Embrace Legacy. You know, if you want to watch it, you find it on my blog, philpeaser.com. Okay, sponsorship segment over, but really, right. So a lot of the things that you want to redo in the beginning are just not important now, right? The parts that really matter are the parts that create value for your customers. And also creating value for customers could mean that essentially make you faster and make you or enable you to deliver high quality software faster. Right. And there is no standard rule how to find them. But I guess you will do this. I trust you. And with that, I'll send you off. So this was the rewrite trap. My name is Phil Giese. See you around.