A large payment company needs to replace the heart of their production system, the code through which all payments travel. A friend described repeated failed attempts replace it, now the 3rd attempt in a decade. Once again the programmers were building a “better” version of the code—better language, better programmers (just ask ‘em!), better practices. The plan? Build it better then switch it on.
I find it so hard to strike a good balance between coupling and cohesion. I never know when my design requirements can change and it can make so much implode in an instant.
Well, if it was easy, regular people could do it. When we discuss power laws in the next book I'll describe exactly how designs "implode in an instant". In the meantime, know that it is a property of the problem not your solution. Also, there's stuff you can do to mitigate the consequences, like tidying first (and after sometimes & just because sometimes).
Kent, I was wondering why they want to improve the system while doing the rewrite? Wouldn’t it be safer to build a duplicate of the old system using the new stack (bugs and all) and then once you convert over, start the improvements? Change one thing at a time principle.
Great question & I'll explain in greater detail in a future post. In short, because the bigger the circle you draw around the old system to be replaced, the more complicated & sensitive the "API" is. This includes people with implicit knowledge of how to operate the system, for example. There is a tradeoff space--the bigger the chunk being replaced, the bigger the surface area of interactions with the containing system (including people). That's why big integrator replacement projects fail. The interface is impossibly complex & you're going to get something critical wrong. Go too small with replacement, though, and you accrue value more slowly & pay for many cycles of replacement.
The Empirical Software Design approach is to go as small as necessary for safety, then expand from there as quickly as is safe.
This is what we do at Mechanical Orchard to replace constellations of mainframe apps. We move one at a time to the cloud, validating each move as we go.
Aug 24, 2023·edited Aug 24, 2023Liked by Kent Beck
Depending on how the existing system is designed and the overall architecture, that might be nigh impossible. Remember that microservice architectures (and other forms of lightly-coupled modular designs) weren't common until relatively recently. Silos and stovepipes ruled the day for a long time.
I suppose there's always Database As Architecture, temporarily to integrate new pieces with a legacy system... _I_ wouldn't really be excited to work on a migration like that.
If the current design makes it impossible to migrate incrementally, then our first task is to change the design so it is possible to migrate incrementally, no? For each hard change, make the change easy then make the easy change. That's the Empirical Software Design way, anyway. I'm not saying that's the only way.
That's one possible way to proceed. It's really just a form of decomposition, isn't it.
I've definitely seen some silos with walls so thick and reinforced through coupling that any design/architectural changes are very risky and/or difficult. I guess following the empirical design/decomposition strategy all the way down, you eventually find a place to get a foothold for evolution. Hopefully managers and bean counters will trust the development team enough to tolerate that journey. That comes down to relationships, doesn't it? Like I think you've written a few times, software development is really about relationships.
Busted. I have indeed. One challenge is the "leap to the new state" folks always sound so efficient & understandable. When it becomes clear that they aren't going to make it then it's way late in the game. That's why the team was on their 3rd try.
I've gone through a number of project rewrites over the decades. The "big bang" ones are always a lot harder than everyone thinks, going into them. The most recent one for me was a three year green field rewrite that became increasingly fraught as time went on and the final few months were... so unpleasant I would never do that again!
Since then, that same system has been completely rewritten again but we did it my way, staying on the JVM and switching languages, starting with small pieces at the bottom and incrementally working up the stack, replacing pieces as they needed changes to support the business. It took a lot longer -- close to eight years -- but there was always only one system to maintain, slowly evolving in small steps, until we got to the final step where the build/deploy process really changed. That was the point at which we finally saw a lot of the benefits I'd promised all those years ago. Now we have a fully-automated CI/CD process and we can take pieces of the system to production as and when we want, instead of having huge, monolithic, manual deployment windows.
I haven't tried an incremental rewrite that changed the core stack, but I think a similar, bottom-up approach could be used for that (but, at this stage in my career, I kind of hope I don't have to find out -- I've been on the JVM for 26 years now and I kind of like how it works).
If I have something working I will put tremendous effort into keeping it working while improving it. A dysfunction I've seen is where people have something "almost" working (where no one can quantify "almost") & yet are reluctant to "start over".
I'll write this as an "attack" for challenging it, but the attack is against myself first in that case.
Is thinking like this not just a consequence of anxiety? Not trusting that one could do the change, so try everything around, extra careful steps to fight that anxiety and risks?
Sure on a 5year work, it would fail, but are we not doing the same even for few weeks tasks that may have worked without all the care?
I tend to have this kind of answers because of anxiety. Test everything around, extract, strengthened, refactor, document, then document the old crappy behavior, to only at the end replace. Creating this under level bricks, that I can trust and move around because I am afraid otherwise.
At the end I might have replaced everything and taken way longer than it could have by rewriting first (let's say I would have succeeded at this part of the story) (and ignoring that this let me have a running system all the time).
But in practice, when I work like this, with extra care, I always get surprised how wrong some of the previous implementation was, how everything could have broken by replacing as it was not working as it was showing it is. Which allows me to notify users of the issues, prepare smooth transition, and break when I want to on a single commit, and not on the "everything was replaced" step we need to deal with all of it. Breakage still happens, but really rarely, and can often either revert, or be fixed in the same day on their side (only internal users on my case, so easier this way).
Issue I have with this, is that just supports my anxiety, say to keep having it as I am rewarded. How much of this is just self told story to justify my extra work on "real" reasons and not just on anxiety reasons?
A friend told me recently that he was kinda the star of his company, the cool guy, because he was able to deliver on time and regularly, when others were struggling. A guy same as me, devoured by anxiety, so using all the security measures from XP to be sure of what he was producing.
So sometime I am bit in an internal discussion between if it is really justified, or if the anxiety is really a mirror of reality. We will miss things, we will plan wrong, we will do mistakes, so must use all the XP tools to better master what you produce and be ready to change it, and we CANNOT do it without that.
And sometime, wondering if I am not just answering to my anxiety and not the real world constrain in that case.
In my case, anxiety made me put tools and systems in place that productively helped me cope with it. They were in practice also really useful for the product I was writing. Anxiety did not made me hide in a corner crying not knowing what to. I was also lucky to be in a place where time to put them in place was given to me. I just sometime ask myself when could I just not worry that much and still get good results, a balance that let me sleep at night.
an incremental development/deploy based on many small_steps/tiny_bangs might avoid a 4th attempt
Well that's what I was thinking but they didn't ask me.
I find it so hard to strike a good balance between coupling and cohesion. I never know when my design requirements can change and it can make so much implode in an instant.
Well, if it was easy, regular people could do it. When we discuss power laws in the next book I'll describe exactly how designs "implode in an instant". In the meantime, know that it is a property of the problem not your solution. Also, there's stuff you can do to mitigate the consequences, like tidying first (and after sometimes & just because sometimes).
Looking forward to reading it!l
Kent, I was wondering why they want to improve the system while doing the rewrite? Wouldn’t it be safer to build a duplicate of the old system using the new stack (bugs and all) and then once you convert over, start the improvements? Change one thing at a time principle.
Great question & I'll explain in greater detail in a future post. In short, because the bigger the circle you draw around the old system to be replaced, the more complicated & sensitive the "API" is. This includes people with implicit knowledge of how to operate the system, for example. There is a tradeoff space--the bigger the chunk being replaced, the bigger the surface area of interactions with the containing system (including people). That's why big integrator replacement projects fail. The interface is impossibly complex & you're going to get something critical wrong. Go too small with replacement, though, and you accrue value more slowly & pay for many cycles of replacement.
The Empirical Software Design approach is to go as small as necessary for safety, then expand from there as quickly as is safe.
This is what we do at Mechanical Orchard to replace constellations of mainframe apps. We move one at a time to the cloud, validating each move as we go.
Do I detect a subtle jab at RUP and the IBM products there? Anyone else notice? (not that I would find that distasteful :-)
I'm insulted. Subtle? 🤪
If they like to change to difference language, would be difficult to tinier steps ?
If you absolutely had to adopt a new language incrementally (say going from Ruby to Rust), how would you do it?
maybe rewrite the simplest part first, and communicate with the original system with micro-service approach? Or some other better suggestion?
Depending on how the existing system is designed and the overall architecture, that might be nigh impossible. Remember that microservice architectures (and other forms of lightly-coupled modular designs) weren't common until relatively recently. Silos and stovepipes ruled the day for a long time.
I suppose there's always Database As Architecture, temporarily to integrate new pieces with a legacy system... _I_ wouldn't really be excited to work on a migration like that.
If the current design makes it impossible to migrate incrementally, then our first task is to change the design so it is possible to migrate incrementally, no? For each hard change, make the change easy then make the easy change. That's the Empirical Software Design way, anyway. I'm not saying that's the only way.
That's one possible way to proceed. It's really just a form of decomposition, isn't it.
I've definitely seen some silos with walls so thick and reinforced through coupling that any design/architectural changes are very risky and/or difficult. I guess following the empirical design/decomposition strategy all the way down, you eventually find a place to get a foothold for evolution. Hopefully managers and bean counters will trust the development team enough to tolerate that journey. That comes down to relationships, doesn't it? Like I think you've written a few times, software development is really about relationships.
Busted. I have indeed. One challenge is the "leap to the new state" folks always sound so efficient & understandable. When it becomes clear that they aren't going to make it then it's way late in the game. That's why the team was on their 3rd try.
I've gone through a number of project rewrites over the decades. The "big bang" ones are always a lot harder than everyone thinks, going into them. The most recent one for me was a three year green field rewrite that became increasingly fraught as time went on and the final few months were... so unpleasant I would never do that again!
Since then, that same system has been completely rewritten again but we did it my way, staying on the JVM and switching languages, starting with small pieces at the bottom and incrementally working up the stack, replacing pieces as they needed changes to support the business. It took a lot longer -- close to eight years -- but there was always only one system to maintain, slowly evolving in small steps, until we got to the final step where the build/deploy process really changed. That was the point at which we finally saw a lot of the benefits I'd promised all those years ago. Now we have a fully-automated CI/CD process and we can take pieces of the system to production as and when we want, instead of having huge, monolithic, manual deployment windows.
I haven't tried an incremental rewrite that changed the core stack, but I think a similar, bottom-up approach could be used for that (but, at this stage in my career, I kind of hope I don't have to find out -- I've been on the JVM for 26 years now and I kind of like how it works).
I feel the “Plan to throw one away” from the MMM book is misleading or people like to use it as an excuse for building completely new thing.
If I have something working I will put tremendous effort into keeping it working while improving it. A dysfunction I've seen is where people have something "almost" working (where no one can quantify "almost") & yet are reluctant to "start over".
The sunk cost effect in full bloom - "I'd prefer not throwing this away since I have spent the better part of the sprint working on this"
I'll write this as an "attack" for challenging it, but the attack is against myself first in that case.
Is thinking like this not just a consequence of anxiety? Not trusting that one could do the change, so try everything around, extra careful steps to fight that anxiety and risks?
Sure on a 5year work, it would fail, but are we not doing the same even for few weeks tasks that may have worked without all the care?
I tend to have this kind of answers because of anxiety. Test everything around, extract, strengthened, refactor, document, then document the old crappy behavior, to only at the end replace. Creating this under level bricks, that I can trust and move around because I am afraid otherwise.
At the end I might have replaced everything and taken way longer than it could have by rewriting first (let's say I would have succeeded at this part of the story) (and ignoring that this let me have a running system all the time).
But in practice, when I work like this, with extra care, I always get surprised how wrong some of the previous implementation was, how everything could have broken by replacing as it was not working as it was showing it is. Which allows me to notify users of the issues, prepare smooth transition, and break when I want to on a single commit, and not on the "everything was replaced" step we need to deal with all of it. Breakage still happens, but really rarely, and can often either revert, or be fixed in the same day on their side (only internal users on my case, so easier this way).
Issue I have with this, is that just supports my anxiety, say to keep having it as I am rewarded. How much of this is just self told story to justify my extra work on "real" reasons and not just on anxiety reasons?
A friend told me recently that he was kinda the star of his company, the cool guy, because he was able to deliver on time and regularly, when others were struggling. A guy same as me, devoured by anxiety, so using all the security measures from XP to be sure of what he was producing.
So sometime I am bit in an internal discussion between if it is really justified, or if the anxiety is really a mirror of reality. We will miss things, we will plan wrong, we will do mistakes, so must use all the XP tools to better master what you produce and be ready to change it, and we CANNOT do it without that.
And sometime, wondering if I am not just answering to my anxiety and not the real world constrain in that case.
In my case, anxiety made me put tools and systems in place that productively helped me cope with it. They were in practice also really useful for the product I was writing. Anxiety did not made me hide in a corner crying not knowing what to. I was also lucky to be in a place where time to put them in place was given to me. I just sometime ask myself when could I just not worry that much and still get good results, a balance that let me sleep at night.