It's probably a big lesson to learn for DevOps/platform teams: just build what is necessary. And don't forget the corollary: always build for change.
It's a lesson software developers have learned (or to be exact, are still learning) but have evaded infra developers, probably due to they only have moved recently to more flexible (but more complex) components and their related technics: PaaS has replaced hardware, IaC has replaced switchboard. I just buy this a little however: My background was in CES manufacturing, and building for changes, with interchangeable parts, flexible assembly lines, and other Lean manufacturing concepts was primordial for surviving the crazy passing of the modern market. It makes no sense to me to hear a team telling me it will takes 2 men-months of work to simply update the infra to support a new build platform, when my old team of engineering were able to reconfigure a pipeline in a 1 week to take into account a new form factor with a new supplier.
I'm reminded of a project I worked on when Adobe acquired Macromedia (where I worked). We'd been rolling out collaboration servers (behind the Connect service) fairly regularly at Macromedia, and this was going to be our first rollout as Adobe. The first battle was getting our (standardized by this point) stack approved by Adobe IT -- because they had different standards for databases, app servers, and hardware -- and even once we'd gotten an "exemption" (since all our existing clusters used the same stack), the new IT folks said it would be a six-month project to build everything out for us. Rollouts had been on about a six-week schedule prior to the acquisition. I'd been at Macromedia for six years and had been really happy there because they were very flexible about pretty much everything. About six months post-acquisition I was unhappy, eight months in I was miserable, eleven months in I quit.
So I think why I like reading your writing is not because of the great lessons, but because of the humor. "For some value of never" is a witty dry funny statement that because I have lived it, I appreciate. It made me laugh out loud in a public place!
You raise a great point here, Kent! Makes me think of how the general rule is to do vertical scaling until it doesn't make monetary sense anymore, and only then do the more complicated horizontal scaling. Of course, when it makes sense to switch over depends on the company and the application, but that's your point. If you're Google or Meta then you need to start with horizontal scaling, but if you're a startup, horizontal scaling only makes deploying your product more complicated for no or very little gain.
This is the breakthrough we need in the 21st century software profession. The app is the first class citizen; the infrastructure is an annoying detail (as Uncle Bob once alluded to).
And how about longer timeframes and bigger orgs? Shall we start every new application from a clean slate, so its infra perfectly reflects its needs? Or shall we share the infrastructure and encourage the use of common abstractions at the cost of it accreting more and more over time?
With the former, how are we to steer towards some consistency and not reinventing the wheel? With the latter, how are we to avoid stagnancy and enable evolution?
Could we call this “just-in-time” infrastructure as Lean would suggest. Or does that imply more structure than ad hoc?
I completely agree that infrastructure needs to act as a playground for applications that are exploring. One way I have seen this work is through extra capacity.
An app wanted to try out location-based recommendations (and we don’t know yet if this product is going to survive), they could filter on latitude and longitude separately in a classical database. With a lot of spare processing capacity, small enough dataset, and a large enough cache, the classical DB worked fine. This required zero new infra development and enabled the app to try this feature.
Once the product started to get traction (and the pressure on the DB began to approach its limits), and in-memory spatial DB service was stood up. This reduced the cost on the classical DB by a few of orders of magnitude and worked really well while the dataset fit in memory. Sharding was introduced to scale beyond a single machine. When managing many shards became cumbersome, a cache-friendly on-disk indexing was developed.
The name matters, for sure. JIT suggests you start building on speculation. I’m talking about a fuzzier situation. We see the outline of the infrastructure we wish we would have had in one or more current apps. Then we pull it out. If we pull it out after one specific examples that’s maybe early. If we pull it out after one specific example but we wish we had it for the next app, that’s safer. After 3-4 examples it’s getting late but okay. After 20 examples ouch.
I find there's a slight variation of the path that doesn't so much involve regret as it does awakening and/or maturing. Instead of "I wish we had built this a different way...", it's "Hey, that's a useful capability that other apps could use!" or "Now that App X has proven and evolved that capability, we should share it with other apps." or some combination of all three.
It's probably a big lesson to learn for DevOps/platform teams: just build what is necessary. And don't forget the corollary: always build for change.
It's a lesson software developers have learned (or to be exact, are still learning) but have evaded infra developers, probably due to they only have moved recently to more flexible (but more complex) components and their related technics: PaaS has replaced hardware, IaC has replaced switchboard. I just buy this a little however: My background was in CES manufacturing, and building for changes, with interchangeable parts, flexible assembly lines, and other Lean manufacturing concepts was primordial for surviving the crazy passing of the modern market. It makes no sense to me to hear a team telling me it will takes 2 men-months of work to simply update the infra to support a new build platform, when my old team of engineering were able to reconfigure a pipeline in a 1 week to take into account a new form factor with a new supplier.
Optionality is worth money so it’s worth investing in.
I'm reminded of a project I worked on when Adobe acquired Macromedia (where I worked). We'd been rolling out collaboration servers (behind the Connect service) fairly regularly at Macromedia, and this was going to be our first rollout as Adobe. The first battle was getting our (standardized by this point) stack approved by Adobe IT -- because they had different standards for databases, app servers, and hardware -- and even once we'd gotten an "exemption" (since all our existing clusters used the same stack), the new IT folks said it would be a six-month project to build everything out for us. Rollouts had been on about a six-week schedule prior to the acquisition. I'd been at Macromedia for six years and had been really happy there because they were very flexible about pretty much everything. About six months post-acquisition I was unhappy, eight months in I was miserable, eleven months in I quit.
So I think why I like reading your writing is not because of the great lessons, but because of the humor. "For some value of never" is a witty dry funny statement that because I have lived it, I appreciate. It made me laugh out loud in a public place!
I often say things that disturb people. The humor helps. I can't really be funny on command, though. It has to come in the flow.
You raise a great point here, Kent! Makes me think of how the general rule is to do vertical scaling until it doesn't make monetary sense anymore, and only then do the more complicated horizontal scaling. Of course, when it makes sense to switch over depends on the company and the application, but that's your point. If you're Google or Meta then you need to start with horizontal scaling, but if you're a startup, horizontal scaling only makes deploying your product more complicated for no or very little gain.
I now have to find more places to use the term “Regret-Based Programming”.
I'm proud of that one.
Regret-Driven Development
This is the breakthrough we need in the 21st century software profession. The app is the first class citizen; the infrastructure is an annoying detail (as Uncle Bob once alluded to).
And how about longer timeframes and bigger orgs? Shall we start every new application from a clean slate, so its infra perfectly reflects its needs? Or shall we share the infrastructure and encourage the use of common abstractions at the cost of it accreting more and more over time?
With the former, how are we to steer towards some consistency and not reinventing the wheel? With the latter, how are we to avoid stagnancy and enable evolution?
Could we call this “just-in-time” infrastructure as Lean would suggest. Or does that imply more structure than ad hoc?
I completely agree that infrastructure needs to act as a playground for applications that are exploring. One way I have seen this work is through extra capacity.
An app wanted to try out location-based recommendations (and we don’t know yet if this product is going to survive), they could filter on latitude and longitude separately in a classical database. With a lot of spare processing capacity, small enough dataset, and a large enough cache, the classical DB worked fine. This required zero new infra development and enabled the app to try this feature.
Once the product started to get traction (and the pressure on the DB began to approach its limits), and in-memory spatial DB service was stood up. This reduced the cost on the classical DB by a few of orders of magnitude and worked really well while the dataset fit in memory. Sharding was introduced to scale beyond a single machine. When managing many shards became cumbersome, a cache-friendly on-disk indexing was developed.
The name matters, for sure. JIT suggests you start building on speculation. I’m talking about a fuzzier situation. We see the outline of the infrastructure we wish we would have had in one or more current apps. Then we pull it out. If we pull it out after one specific examples that’s maybe early. If we pull it out after one specific example but we wish we had it for the next app, that’s safer. After 3-4 examples it’s getting late but okay. After 20 examples ouch.
I find there's a slight variation of the path that doesn't so much involve regret as it does awakening and/or maturing. Instead of "I wish we had built this a different way...", it's "Hey, that's a useful capability that other apps could use!" or "Now that App X has proven and evolved that capability, we should share it with other apps." or some combination of all three.