Ha…

Read more

Why can’t tree-huggers & forest destroyers get along?

I lived in the wilds of southern Oregon for 17 years. I enjoyed many things about life in the forest. One of the things I loved most was watching the politics. Our community was polarized before polarization was cool. Loggers versus environmentalists, sure, but also weird alliances. Open-carry fisherman cozying up to tree huggers to save the salmon & steelhead. Rugged loggers cozying up to (relatively) soft townies with money.

My roots went deep in the area, one set of grandparents having moved there in 1943. The other grandparents (that grandfather was a former mayor & city councilor) moved there in 1933. I felt connected & invested, even if I didn’t usually stick my ideas out there.

Gridlock

The one time I did put my 2 cents in (this is back when we had cents) was when I tried to resolve the absolute hatred between environmentalists & loggers using what I was studying about incentive structures. I think the system I came up with was cool but it quickly disappeared, so now I want to put it in public permanently here.

Here’s the setup:

• Only 5% of the old-growth forest was left.

• Loggers wanted to harvest all of it.

• Environmentalists wanted all logging to stop.

• Second-growth forest was prone to catastrophic wild fires. (There’s a special kind of helpless feeling watching a fire approach your home.)

• Loggers wanted to harvest second-growth.

• Environmentalists wanted all logging to stop.

The result was a complete impasse. Forests burning. Mills closing. Crime & drugs up. Anybody with any ambition leaving. Nobody was getting what they wanted—loggers, environmentalists, workers.

(Or at least what they said they wanted—there seemed to be a bunch of psychodramas playing out.)

Incentives

I recast the forest tinning situation as an incentives problem (I was intensively studying incentives at the time). Once the loggers finally got permission to harvest a tract of second-growth, they were incentivized to take out every stick of wood with economic value, leaving further growth stunted, encouraging the growth of flammable underbrush.

Because environmentalists saw the loggers’ incentives, they were ever more incentivized to block all logging & put onerous restrictions on any activity that managed to sneak through. In Influence Diagram terms, more logging leads to more money & more damage, but more damage leads to more resistance leads to less logging.

Classic inhibiting loop. More logging leads to less logging. (We could go on & on mapping this system, but this will do to illustrate my pr of the many interventions we can make in a system is to speed or slow feedback. What if, instead of getting paid for this harvest, the loggers got paid for the next harvest. Today they’d thin the forest, with any valuable material going to be turned into products, but it wasn’t until 10 years later that they would be paid the proceeds of the next forest thinning. You would get paid more if the forest thrived over the next 10 years, less if it grew less.

(I think I kind of made up the notation for delay.)

Now we have a reinforcing loop. More logging. Less damage (because the loggers get paid in a decade). Less resistance. More logging (in the form of forest thinning.)

Priming the Pump

That first logger, how do they get paid? They are paying for diesel, salaries, depreciation today & won’t get money for 10 years. The right to be paid in 10 years can be turned into a financial instrument to be sold today (remember those soft townies with money?) Now you have monied interests who also care about the forest’s health.

And who is best suited to evaluate the health of the forest for future gain (and avoiding future loss from pests or fire)? Well, those environmentalists who care so about the forest are well positioned to act as consultants & auditors.

Mill workers would be back at work. Local capital would have another way to extract rents. Environmentalists would have healthier forests. Loggers would have trees to cut.

Chickening Out

I wrote the above up as a letter to the editor of the local newspaper. That got me invited to a “summit” of conservationists & loggers. When the microphone got around to me I had an attack of shyness, said something self-deprecating, and passed the mic on to the next person. So that was that.

Would it have worked? Maybe. Entrenched interests were more interested in staying entrenched than in making progress. That’s true today in many situations I see. It’s not as simple as “change the rules and the behavior will change”. But “don’t change the rules & the behavior will definitely not change”.

Kent partners with a handful of companies each year on editorial collaborations, speaking, and workshops. If that's interesting, let's talk →

I’m writing this in a creative storm, so it’s not polished. Thought it was worth getting out to y’all & getting your feedback.

The “natural” use of genies is to further isolate people who are already suffering (even if they enjoy it) from isolation. “Teams of one can now achieve infinite results!!!” Technical solution to a human problem. (There should be…

Read more

Folks are taking knee-jerk action around the advent of AI—slowing hiring, firing all the juniors, cancelling internship programs. Instead, let’s think about this a second.

The standard model says junior developers are expensive. You pay senior salaries for negative productivity while they learn. They ask questions. They break things. They need code review. In an augmented development world, the difference between juniors & seniors is just too large & the cost of the juniors just too high.

Wrongo. That’s backwards. Here’s why.

Oops, I buried my lede. Here’s my number one lesson—manage juniors for learning, not production.

The Valley of Regret

A junior developer is a bet. You’re paying money today for productivity tomorrow. The shape of that bet looks something like this:

That red zone—the “valley of regret”—is where things go wrong. You’re spending money. You’re investing senior attention. And every month the junior is in that valley, something can happen: they get a better offer, they decide engineering isn’t for them, your startup runs out of runway, there’s a layoff.

The longer & deeper the valley, the more likely you never reach the other side.

This is why, when I talk to engineering managers, I hear the same thing: “We’d love to hire juniors, but we can’t afford the ramp time right now.”

Shrinking the Valley

I’ve been watching junior developers use AI coding assistants well. Not vibe coding—not accepting whatever the AI spits out. Augmented coding: using AI to accelerate learning while maintaining quality. Remember, you’re managing for learning, not production.

Managed for learning, the magnitude of the junior bet changes:

The juniors working this way compress their ramp dramatically. Tasks that used to take days take hours. Not because the AI does the work, but because the AI collapses the search space. Instead of spending three hours figuring out which API to use, they spend twenty minutes evaluating options the AI surfaced. The time freed this way isn’t invested in another unprofitable feature, though, it’s invested in learning.

Learning fast can become a habit. When a task is “completed”, there is always the opportunity to squeeze more juice from it:

• How else could this have been done?

• Is there now a way to simplify the code?

• What are the tradeoffs?

• Are there more tests we can/should write?

• What is the performance envelope?

The genie is an endlessly patient tutor, but unlikely to volunteer learning opportunities. We’ll talk at the end of this piece about why learning like this is rare & what you need to change to realize the potential of juniors. It’s not just going to happen by itself.

First Order: More Value

Shortening the valley isn’t just “you get productivity sooner.” It’s that you survive the valley at all. (See also ’s Ergodicity.)

Junior developers have high turnover. Let’s say 20% annual attrition—low for tech. Run the numbers:

• Traditional 24-month ramp: About 36% of juniors leave before they ever become net positive. More than a third of your bets never pay off.

• Compressed 9-month ramp: About 15% leave before breakeven. You’ve cut your failure rate by more than half. Of the juniors who pass break even, more of that profitable area comes back to the company.

You’re not just getting productivity 15 months earlier. You’re dramatically increasing the probability that you realize profitability at all.

Second Order: Faster Growth

There’s another factor a simple model of junior productivity misses. Productive developers don’t just produce—they multiply.

A developer who reaches payoff productivity doesn’t just write code. They mentor other juniors. They build institutional knowledge. They take on projects that create leverage for the whole team. Every month you accelerate one developer’s ramp, you’re also accelerating the ramp of everyone they’ll eventually help.

What This Means

If you’re an engineering manager thinking about hiring:

The junior bet has gotten better. Not because juniors have changed, but because the genie, used well, accelerates learning.

Investment in tooling is investment in hiring. The ROI on getting your AI tooling right isn’t just “existing developers are more productive.” It’s “we can now make profitable bets on juniors that we couldn’t make before.”

The math favors action. In high-attrition environments especially, the expected value of a junior hire has shifted. You’re no longer betting against the valley of death—you’re shortening it.

But you have to actually do it. The compressed ramp doesn’t happen automatically. It requires intentional investment in how juniors learn using the genie. Not vibe coding—augmented coding. That’s a skill, and you have to teach it, encourage it, refine it.

The bet has gotten better. Better if you manage juniors for learning & not production.

Today’s post is sponsored by CodeRabbit. Boost your team’s code quality and shipping speed with CodeRabbit—the most advanced AI code review tool built for engineers. CodeRabbit delivers context-aware, line-by-line reviews, instant one-click fixes, and concise PR summaries, integrating right into your GitHub workflow so you spend less time diff diving and more time building.​

Ready to optimize your engineering workflow?

Try it Free for 14 Days

Join thousands of developers who’ve halved code review time and defect rates with CodeRabbit. Start your 14-day free trial and experience seamless AI reviews, actionable feedback, and effortless codebase learning.​

We’re going to play with graphs here for a second. The simplest way to look at our conundrum is to watch the features over time.

Rapid progress at first, then stagnation despite our best efforts. Bugs pile up. The build slows. Backwards compatibility imposes its own tax on progress. Original team members move on, while new members take time to acclimate.

What’s to be done? Is this just the price of progress?

Let’s take the first derivative of features, feature progress over time.

Starts out gangbusters, then devolves to occasional bursts of productivity. We’re no closer to a diagnosis or a treatment. We just know that coding with a genie compresses time.

Scatterplot

The typical time series plot fails to generate understanding because it implies that the only source of change in the response is the unrelenting, impassionate drumbeat of time. Show cause. Show effect. — reader of Edward Tufte’s site.

Edward Tufte in Visual Display of Quantitative Information laments that most graphics are time series. If you want to understand the relationship between two variables, you need to plot them with each other, not just as part of the same timeline.

We want to understand feature development, so that needs to be one of the axes. I’m going to put it on the horizontal axis, for reasons that will become apparent (I tried it every which way before settling on this—I’m giving you the abridged version).

If all we’re looking at is feature progress, we expect progress to be spread apart (fast) at first, then bunch up as time passes.

But what goes on the vertical axis?

Options

No big surprise here, my fellow Tidiers. We know that software embeds economic value 2 ways:

• Cash flow from current features

• Optionality for future features

Let’s see how this plays out. We start out with no features & lots of options:

The first feature we develop goes pretty far but inevitably it burns options.

The code is now more complicated, we have to either preserve backwards compatibility or tear out the feature before proceeding.

(I hear the “why not make it better on both axes at once?” crowd. Make it run then make it right. My brain isn’t big enough to do a good job of both at once. Not even my augmented brain.)

Okay, so we have one feature. We want more! Features feel good! Features make customers feel good!

It wasn’t much harder than that first feature. What’s that you say? More features & then more features? You got it!

You got it right until you don’t. If you look at my GitHub repos you’ll see BPlusTree, BPlusTree2, BPlusTree3, etc. In each case the genie & I drove the project out of options & had to start over. Somehow, though, I don’t learn my lesson. I don’t name the first project BPlusTree1—this time will be different!

Exhale Then Inhale

What’s the alternative? We can’t improve both axes at once. We can’t escape burning some options with each feature. The first feature is the same.

We do have a choice when we’ve finished that first feature, though. There is a space between features, should we choose to perceive it. In that space we can choose to invest in restoring & enhancing optionality.

Tidying first is one way to perceive this gap between features. What’s the next feature we are implementing? What about our current mess makes that harder? What can we tidy to make it easier? If implementing the feature is easier, then the option of implementing it becomes more attractive.

We can, should we so choose, continue this rhythm—feature, options, feature, options.

And Next…

That’s all I have time for today. I’ll use this same “Options Lead To Features” graph to describe a variety of problematic situations in development, augmented & not.

Thank you to Gene Kim for a long series of Tokyo breakfast conversations (long story) that helped refine this diagram.

Today’s post is sponsored by CodeRabbit. Boost your team’s code quality and shipping speed with CodeRabbit—the most advanced AI code review tool built for engineers. CodeRabbit delivers context-aware, line-by-line reviews, instant one-click fixes, and concise PR summaries, integrating right into your GitHub workflow so you spend less time diff diving and more time building.​

What Makes CodeRabbit Different?

• CodeRabbit provides AI-powered reviews that adapt to your team’s standards, enforcing style, spotting bugs and edge cases, and mapping out code dependencies automatically.​

• With multi-language support and over 40 linters and static analysis tools, it keeps your code clean, secure, and maintainable—no matter how complex your stack.​

• Real examples show dramatic impact: SalesRabbit cut bugs by 30% and boosted engineering velocity by 25% simply by adding CodeRabbit to all deploys.​

• Engineered to help junior and experienced devs alike, CodeRabbit catches issues even seasoned reviewers might miss, and its built-in documentation and reporting keep everyone informed and aligned.​

Ready to optimize your engineering workflow?

Try it Free for 14 Days

Join thousands of developers who’ve halved code review time and defect rates with CodeRabbit. Start your 14-day free trial and experience seamless AI reviews, actionable feedback, and effortless codebase learning.​

This is a visualization of a chain I’ve talked about before: Effort→Output→Outcome→Impact.

1. Effort—the work we do.

2. Output—the changes the user/customer sees.

3. Outcome—the changes in the user/customer behavior.

4. Impact—the profit we share with the customer.

My point was that the earlier in this chain you make your observations, the easier it is to observe (it’s easy to count hours spent programming) but also the easier it is to game the observations.

Conversely, the later you measure, the more difficult it is attribute changes to any one cause but the less subject the measures are to distortion. I can generate as many lines of code as I want, if that’s what I’m being judged on, but it’s hard (not impossible, but harder) to manipulate profitability.

Sponsored by CodeRabbit

Teams using CodeRabbit report 50%+ reduction in code review time. It’s not just automated linting—it understands your codebase, learns your team’s patterns, and provides contextual feedback that actually matters.

SOC2 Type II certified. Zero data retention.

Try it Free for 14 Days

Cycles

I was taken to task on LinkedIn for not considering feedback loops in my analysis.

There’s a beautiful phrase in English—don’t teach your grandmother to suck eggs. I’ve been using systems thinking since the 80s (1980s 😉), but I don’t trot it out unless I really need it. I don’t need systems thinking & feedback loops to confidently predict that “Percentage of code generated by an LLM” is a terrible target, guaranteed to incentivize destructive behavior.

The Toolbag

Aside from not assuming that your interlocutor is ignorant, my suggestion for Francisco above is yes to have a big bag of tools but not to reach too deep too soon. If a superficial analysis reveals a flaw, then go with that.

• Quick analysis takes less time & effort.

• More importantly, linear analysis is easier to explain.

By the time I’ve explained influence diagrams & positive & negative influence & reinforcing & inhibiting feedback loops, I’ve lost most of my audience. If full systems thinking is the simplest way to get my point across, I’ll do it. I’ll invest the time. I’ll risk losing some listeners.

In the case of software development, I don’t need feedback loops to confidently predict the results of measuring & judging lines of code (or PRs or hours worked). The measurement is just too early in the chain of delivering value.

We could spend all day & all night expanding the system to include more aspects of delivering value with software, but we don’t have to.

I once worked for a CEO who brought in a box of donuts every Saturday morning. If the donuts weren’t gone by Monday morning, we’d get a tongue lashing about how we didn’t care enough.

We set up a rota of employees who lived near the office to go in and throw away the donuts Sunday night.

Quick Quiz, No Tricks

How much water flows through a pipe of capacity 5?

5. Told you—no tricks. How much water flows through a sequence of pipes, both with capacity 5.

Same same. 5. (Yes, we can worry about turbulence & leakage at the boundary & all that. I’m making an analogy.)

What happens to the capacity of the system if we expand the capacity of the second pipe to 20?

What comes out of the second pipe? 5. Why? Because that’s all that went into it. Doesn’t matter how big the second pipe is. We can make it a million & all that will trickle out is 5.

To complete the picture, what if we expand the first pipe?

Capacity as measured at the end? 5. Again, doesn’t matter how much we expand the first pipe. We can’t put, as my Pappy would have said, 10 pounds of manure in a 5 pound bag. Well, Pappy wouldn’t have said “manure”, but we’re being professional here.

Application

As an executive you accept a trade. You live a life of service, no longer able to do the detailed work. For this loss of control you get wider influence, greater rewards.

One benefit of an executive position is that you can see the whole process, in ways those at the coalface cannot.

Product → Design → Engineering → Operations & Sales & Marketing is exactly one of these multi-pipe situations. Each department’s incentives align with improving their own capacity. You are uniquely positioned to look across the whole chain. (You are also responsible for improving across the whole chain.)

What can you do?

• Find the narrowest pipe. Look up-stream & you’ll find half done work piling up. You can do this. They can’t. They are all working to keep their pipe going.

• Make sure there’s enough buffer to keep the pipe busy, but no more. This may require reducing the output of upstream pipes. You can do this. They can’t. No one can voluntarily reduce their output.

• Expand the narrowest pipe. Don’t bother with the pipes that aren’t narrowest. At best expanding them won’t help. At worst you’ll flood the system. You can do this. They can’t. They are overwhelmed with daily work.

• Reduce demand on the narrowest pipe. Can you route some of its work elsewhere, even if that elsewhere isn’t ideal? Again, you can do this. They can’t. The folks getting their work re-routed won’t be happy. Boo hoo.

• Stop improvement & shift attention when a new pipe is narrowest. You can do this. They can’t. Your people got credit for improvements (if you did it right). They want more credit.

Mea Culpa

I read the “80 hours weeks or else” pronouncement, got mad, & clapped back.

Here’s why I got emotional. That kind of pressure doesn’t work to get more accomplished. There is a natural limit to how much work comes out of a system. Pressure stresses the system, it doesn’t change the system, not in any positive way. So pressure is low benefit. How about cost?

I’ve seen mental health breakdowns (& had one myself). I’ve seen divorces, kids scarred. I’ve lived through 2 colleague suicides. The cost of pressure is high & permanent. Also, & this is what pisses me off, the consequences accrue to those least able to push back.

So I applied Thinkie Complain & Propose & wrote the above.

Conclusion

Drucker introduces so many effective management skills. Deming too. Matching the pipes is just the start.

You’ve chosen a grave responsibility, with life-changing consequences for your decisions. I wish for you the toolbox to go with that responsibility.

Interested in executive coaching? Augmented coding creates a slew of new Match the Pipes problems to go with the ones we were already struggling with. You can get help.

This morning, you checked your OpenAI dashboard instead. You’re burning $50,000 a week in tokens. Your growth curve looks like a rocket launch. And you're going to hit your API limits in six days.

Whiplash

This is what success looks like. While you were exploring you optimized for cheap experiments. Less time, less money, more chances of survival. Now your hair is on fire. It’s time to shift gears.

Then last Tuesday happened. Some influencer posted about you. Your quiet little AI experiment went from 200 users to 20,000 users. Now it's doubling every three days.

Your cofounders are ecstatic. Your investors are throwing money at you. And you’re staring at the Graph of Death:

.Welcome to success. The only thing worse is failure.

Welcome to Your New Normal

This is your new job: seeing these kinds of cross-overs in time & doing just enough to dodge them. There’s good news:

• You don’t have to solve these problems, just push them out into the indefinite future.

• You have access to capital on a new scale to buy your way out of these problems.

There’s bad news:

• If you don’t manage to push this problem out into the future, it will kill you.

• Your instincts from the past 12 months are exactly wrong.

• Once you push this problem into the future a new one will arise.

Remember being proud when you could get an answer for a hypothesis for one tenth the cost? When you had that elegant queueing system that ensured fair usage?

Forget all of it. Those were exploration habits. You're in expansion now. Different game, different rules, different ways to die.

Your new reality:

• That sustainable architecture you planned? Too late. Usage patterns have shifted & it’s already obsolete.

• That token optimization project? Too slow. Time is now money in a newly existential way.

• That careful cost management? No longer relevant. Once you’re airborne, you’re suddenly immune to the length of the runway.

You have days, not months. Time to break some things.

The Only Graph That Matters

Let’s say you’re a supplier of tokens (the same logic works if you’re a consumer, but you’ll need to translate). Graph:

• Supply: How many tokens you can consume & generate?

• Demand: How many tokens your users are supplying & expecting?

Where they cross, your product dies. Not "degrades." Dies.

Remember—this is success. This is what you worked so hard for. So, now what do you do? Good news: you have 2 levers.

Kicking the Can Down the Road

Lever 1—bend supply up. Do what you need to do to survive. I heard stories from early Facebook of engineers going to Fry’s at lunch, buying servers, racking & stacking, & keeping the site up for one more day of peak usage. Efficient? No. Better than the alternative? Oh yes.

Lever 2—bend demand down.

Optimization is useful, if it can be achieved quickly (remember, time is now lots of money).

Delete expensive-but-not-critical features. I know this is hard. You love everything you built. Better to survive today & reimplement tomorrow.

Reduce users. Wait—you just spent a year praying for one user. Now you have thousands & you want to just drop them on the floor? Yep, again, it’s not a good option but it is the best option. Use invitation-only signups. A waiting list. Whatever to keep demand down below supply.

The Mental Shift That Saves You

In exploration, you were precious about everything. Every user mattered. Every feature was crafted. Every decision was reversible.

In expansion, you're at war. Wars aren't won by perfect soldiers with shiny boots. They're won by whoever doesn't run out of ammunition.

Your ammunition is tokens. Waste them, hoard them, steal them, borrow them—I don't care. Just don't run out.

Capital Isn't the Constraint Anymore

Here's the mind-bender: That runway you were watching? Meaningless now.

Investors smell blood in the water—the good kind. The "holy shit this is actually working" kind. They're begging to throw money at you. Take it. Take it all. You'll need it.

But money can't buy you tokens fast enough. API limits don't care about your Series A. Rate limits don't respect your growth rate. Your constraint shifted from cash to capacity, and that's a different problem to solve.

The Practical Survival Guide

This week:

1. Map every API limit you have. Not just the official ones—the real ones where things start breaking.

2. Split your infrastructure. Multiple accounts, multiple providers, multiple everything.

3. Implement the dumbest possible caching. Cache entire responses. Cache by user ID. Cache incorrectly. Just cache.

4. Turn off features. Start with the token-hungry ones only 5% of users touch.

5. Beg, borrow, and negotiate for higher limits. Today. Not after your "process review."

Next week:

1. Keep doing all of the above.

2. Start thinking about that one perfect solution you'll implement "when things calm down."

3. Laugh at your naivety. Things won't calm down.

4. Double down on the hacks that are working.

Next month:

1. Maybe—MAYBE—start building the real solution.

2. But probably you'll have hit the next bottleneck by then.

3. That's okay. That's what winning looks like.

The Permission Slip You Need

You're not a bad engineer for hardcoding API keys. You're not irresponsible for mixing providers with different capabilities. You're not short-sighted for building systems that will obviously break at 10x scale.

You're surviving expansion.

In six months, when you're in extraction, you'll rebuild all of this properly. You'll have beautiful token management, elegant fallback systems, and efficient prompt optimization. Your code will be clean, your architecture pristine.

But first, you have to live that long.

The Light at the End

I know it feels like you're building on quicksand. Like everything could collapse at any moment. Like you're betraying every engineering principle you hold dear.

You're right. You are.

And that's exactly what you should be doing.

Because in expansion, the only principle that matters is survival. Everything else—code quality, system design, cost optimization—is a luxury you can't afford.

Your investors understand this. Your users (the ones who stick around) understand this. The only person who doesn't understand this yet is you.

Welcome to expansion. It sucks. It's glorious. It's temporary.

Now stop reading and go bend those curves. You've got tokens to find and a product to keep alive.

Tomorrow's crisis is tomorrow's problem. Today, you survive.

First Principles

Why do we care about time in software development? Partly because time is money. Less time, less money spent. But that’s only part of the pi…

Read more

The One Startling Sentence for the presentation is, “Forest & Desert can’t talk.” Words with seemingly-similar mea…

Read more

I had a fantastic week of consulting here (hope I can tell you where “here” is later). It’s a growing org hitting scaling discontinuities. One diagram I drew Monday was used all week, so I thought I’d present it here.

Thanks to today’s sponsor Tuple.

Tuple is developer-grade screen sharing, for pairing and more. They interviewed me for the Distributed podcast, and I went deep on software design as an exercise in human relationships and distributed work. Check it out here.

Specialist Team

And so we do the logical thing, we hire experts & promote the best in-house experts, forming a specialist team.

Good! Now the work is being done by experts & the individual teams have less to do.

The incentives of the specialist team don’t perfectly align with the client teams, however. The specialist team wants testing or design or database administration or compliance or security to be done right, not the janky way it was being done. If the work is rushed, the specialist team is going to have to fix it later. The client team wants their requests to be handled quickly, to be unblocked. As long as the solution is good enough, worse sooner is way better than perfect later.

Buffers

As work piles up, a black market in specialist work emerges. “Hey, can you just…” Words to freeze a specialist manager’s heart. No, if we do you a favor we’ll have to do everyone a favor & then nobody will know what’s going on.

I know—legibility! We’ll put the requests in a clear, public queue, sort them by overall priority, & then work on the most impactful requests first.

Who could possibly object to doing the most impactful work first? What, you want us to do less impactful work?

Buffers Feed Themselves

There are always way more requests than we have specialists to service the requests. No matter how many more specialists we hire, client teams’ appetites instantly consume the available capacity & then some.

Self Service

You can go around the “bigger org→bigger buffer” loop many times without noticing that it isn’t providing the overall org, the client teams plus the specialist team, with more capacity. The request queue grows, wait times stretch, the incentive for the black market grows no matter the policing put into place. This is the problem that emerges when going from medium to big.

The key insight is that only resource that scales with the client teams are the client teams themselves. The next move is for the specialist team to make their expertise self-service. Build the tooling/frameworks/training that allows client teams to mostly satisfy their own requests.

Being able to draw this transformation easily doesn’t mean it’s easy to execute with real people in a real organization. Building self-service tools is bound to interfere with the execution of queued tasks, which were already being serviced too slowly. Client teams are going to need training & consulting to take back up the responsibilities they gave up with the specialist team formed. Client teams who did have their requests serviced had gotten used to bespoke solutions. They must learn to be content with the generic (but scalable) solutions now being provided.

Iteration will go a ways towards soothing the ruffled organizational feathers. Resist the temptation to self-service all the things at once, pausing all requests for an extended period. Find the costliest, most frequent requests to self-serve first. As the specialist team proceeds, they will need to re-work some of their earliest self-service efforts. That’s okay. It’s better than an organization-halting pause.

Conclusion

In the room this week, we just called this concept “the three pictures”. For general consumption, it makes more sense to me to use the analogy of the caterpillar forming a chrysalis & transforming into a butterfly. Staying in the chrysalis, familiar as the pain may become, prevents the whole organization from growing wings & flying.

I posted recently on LinkedIn about teams living without the daily experience of production defects:

As you can see, it stirred up a bunch of reaction, only some of which was because of the brevity of the original post. Here I want to:

• Explain myself more thoroughly &

• Reflect a bit on the negative half of the reaction.

No Bugs is not Zero Bugs

Here’s the contrast I’m making:

• Bugs are a daily fact of life. We track an ever-growing list. We’ll never fix them all. The arrival of high priority defects interferes with feature development. Versus:

• Bugs are a rare exception. When one is reported we drop everything to fix it. We also investigate how it happened & fix that. We make sure that this whole class of bugs can never again reach production. We make amends to everyone affected.

Bugs in The Forest are anti-holidays—coming up at random intervals & stopping development for their duration.

Desert Bugs

Defects are a daily fact of life in The Desert. Bless your heart if this is where you live. You have backlogs of bugs growing monotonically. Most of those bugs you’ll never fix. More bugs come in daily. Some of those bugs are severe enough to interrupt ongoing development. The bug overhang creates tension between parts of the organization—product, sales, and marketing want features but customer service wants fewer bugs.

In The Desert defects feel like an inevitable part of the landscape, like gravity or weather. Disheartening at times, yes, but whaddayagonnado?

Again, if this is you, good on you for being able to make progress while being tugged hard in multiple directions. You are responding well to a truly difficult set of constraints.

Those constraints are not inevitable.

Forest Bugs

One observation about Extreme Programming teams that shocked me is that many of them have almost no defects reported from production. It shocked me because “no reported defects” was not on my list of goals for XP. If I’d wanted “no reported defects”, I would have done something like Clean Room.

Once teams had systems in production for a year or two, though, many of them reflected that they’d had few or no reports of production defects. Teams embedded in organizations where projects typically had thousands of open defects observed that they received one a month, or one a quarter, or one a year. Or none.

Whu?

This set off a round of industry skepticism. After all, “Use a Bug Tracker” was a widely accepted “best” practice. XP teams weren’t using a bug tracker? Well then they must not be serious.

XP teams responded, “Why do you choose to have bugs?” Not perhaps the most diplomatic answer, but one with more than a hint of truth in it.

Okay, next explanation (since the obvious explanation of “these teams don’t put bugs into production” just can’t be true)—they must be lying. Some executive sent down a zero defect edict so they just stopped reporting.

Nope. With the customer right there with the team, dissatisfaction at that scale would be impossible to hide.

Maybe the teams just aren’t paying attention? Nope. Same reason.

Maybe software was just simpler back then. Some ways yes, some ways no. Forest teams are still reporting “incredibly” low defect counts.

How?

If “no” bugs (I’m just going to start saying “no bugs” & trust you to understand what I mean by “no”) wasn’t a goal of XP, it bears thinking about how it could “just happen”. Especially when everyone in The Desert expects bugs to be like weather.

For example—one team tracked down the six defects reported from production in a year & noticed that all of them had been introduced by someone programming alone. After that they were consistent & persistent about pairing—”Hey, whatcha working on there? I noticed you over in the corner.”

Pairing alone won’t do it, though. Defects are too complex a problem to be addressed by any one practice. There’s something about the mixture of:

• Continuous planning, so there’s enough time for implementation, for defect remediation, & for reflection.

• Continuous design, so you can prevent recurrences & mistake “proof” code.

• Continuous integration, so you step on each other’s toes less frequently.

• Continuous deployment, so defects that do get to production are more quickly identified & isolated.

• Continuous collaboration (pairing, ensemble), so a) bugs are more likely to be caught immediately & b) understanding of bugs (& safety) quickly percolates through the whole team.

• Continuous testing, so you have an immediate double check on implementation decisions.

• Customer on the team, so you can just ask instead of feeling pressure to move forward on shaky assumptions. Also, everyone on the team can see the actual face of the person affected by bugs.

Practically all the roots of The Forest work together to create the “bug as anti-holiday” development experience.

Reaction

Why is the possibility of teams living with no bugs so threatening to some folks?

First, this is just what it’s like trying to communicate from The Forest to The Desert. Assumptions in one realm seem absurd in the other.

Second, it’s clear that folks in The Desert are attached to the notion of production defects as just part of development. That feels like a coping mechanism to me—folks feel bad about production bugs so they soothe themselves by assuming they are inevitable.

Last, getting from a Desert perspective on production defects to a Forest perspective is going to be mighty hard work. You’re changing the plane’s engine in the air. But no bugs is not rocket surgery. And it carries benefits for everyone involved.

First, let me say that when I say “high consequence”, I don’t mean human life. Taking responsibility for human life in programming projects requires some of the same principles, but turned all the way up. I’ve done it, but it’s not the norm. Here I’m speaking of business consequences.

Structure of Expand

Back to 3X: Explore/Expand/Extract basics, Expand is the phase of the project/company/industry where you have explored & found a new growth engine. Expand is powered by a reinforcing feedback cycle where the more you grow, the easier it is to grow more.

This “hypergrowth” will be threatened by a series of near-catastrophes, each of which you must avoid to realize the potential of the growth loop you have so fortuitously discovered. And it isn’t hyperbole to call them catastrophes. As puts it, “irreversibility absorbs future gains”.

When you are truly in Expand, each near-catastrophe will come at you like a sharp curve on a foggy mountain road at night. You need to survive each one to experience the next. Attempts to solve all future problems through careful & thorough engineering will backfire. The extra effort & delay increase the chances of an irreversible disaster. Expand is the time for duct tape & zip ties.

Rate Limiting Resource

Each crisis is brought about by the intersection of the supply of some critical resource & the (rapidly rising) demand for that resource.

There was some resource—network bandwidth, CPUs, trucks, programmers, insurance—that was plentiful enough not to worry about. Demand for it has skyrocketed because you’ve been zipping around that growth loop (love those exponentials). Some day in the future (you hope it’s the future), you’re going to need more of the resource than is available. That is going to be a terrible, horrible, very bad, no good day.

In Theory

Your job in Expand is to put that day off into the indefinite future. Not “solve” the problem. Not beautifully & elegantly finesse it. Nor yet ignore it. Put it off so you can be ready for the next looming catastrophic curve crossing.

You do this by some combination of bending the supply curve up & bending the demand curve down.

No more imminent death!

How? Increase supply by:

• Throwing money at the problem. You’ve been frugal while exploring. Relax that constraint for the moment. Capital sniffs out expansion. Spend some of it.

• Quick engineering. Since you haven’t been focused on this resource, perhaps there’s some quick way to supply more of it.

Decrease demand by:

• Limiting growth. Think of all the “invitation only” projects. It’s good marketing but it’s also good engineering. If you know what your safe growth curve looks like, don’t take on more customers than that.

• Load shedding. Delete features that aren’t critical to the growth curve. Some of your pet features may need to be shelved. Boo hoo.

• Optimizing. Use less of the resource per user. Delay kills, though, so be tactical.

Structure

We’re almost ready to discuss the alternative to “can’t fail explorations”. The conceptual leap comes from observing that Explore/Expand/Extract curves are fractal. What looks like a smooth line in a macro view is made up of a bunch of little S-curves.

The overall Explore/Expand/Extract curve:

Let’s, um, expand the Expand segment, showing that it is made up of little 3X curves:

Now we can expand one of those curves to see the whole structure:

And the Explore portion of that Expand sub-project looks like any exploration any where. We try a bunch of things. Most we abandon. The one that works surprisingly well we expand on:

High-Risk Management

That is “managing substantial risks”, not “managing so as to increase risk”.

Panic is your nemesis while expanding. The temptation is to see a looming resource supply/demand cross-over & try to “solve” it. As your initial solution encounters problems, sunk cost fallacy breeds with terror to produce insane hours, little progress, & compounding risk of irreversible failure.

Acknowledge the situation. The price of success is seemingly insurmountable problems. That you don’t know how to solve the problem is a good sign. It means fewer competitors. But the nature of risk doesn’t change just because it scares you. The way forward is by acting appropriately for the 3X phase you’re in.

You’re in Expand overall, but for this particular looming resource shortage you start in Explore. If you’ve been reading along with 3X you know the rules for Explore:

• Creativity & diversity. You don’t want your project to die just because you didn’t look around.

• Small experiments, quickly discarded.

• Experiment in parallel. Why not? You have the money.

• Collaborate closely. The team should still be small at this point, which makes this easier. (Hey, if it was easy easy then regular people could do it.)

Once you’ve found a promising idea to increase supply and/or reduce demand, focus on that idea alone (Expand & then Extract) until this crisis has been averted. Then keep your eyes open for the next looming catastrophe.

Oh, and once there’s only hard work ahead, no catastrophes, then you’re ready to shift into Extract mode. Extract is its own deal. For another day.

Conclusion

When stakes are high & the consequences for failure expand, that is exactly the time to use good project hygiene. It’s not heroics. It’s smart engineering in an environment with strange & unpredictable constraints.

Some day the limitations to growth will kick in. Not today, Satan.

Republished from the copy published 25 March 2016. Republished from the original published on 19 July 2016.

Because I keep careful notes these days I can identify the precise moment when I asked the question I should have asked twenty years ago: what if those waterfall folks aren’t wrong, what if they are solving a different problem than I’m solving? What problem is that?

The past four months have been fecund beyond belief for me. I have changed how I approach software development but also life in general. Different risk profiles call for different approaches, but I can sometimes choose which risk profile I use in a situation.

Trying to communicate my insight has been frustrating. I feel like a puppy on a hike. I’ve run ahead, found an awesome stick, brought it back, dropped it at my masters’ feet, and now I’m barking madly without anyone paying to attention to just how awesome the stick really is.

I awoke in the middle of the night with a new approach, a new metaphor to go with the new vocabulary I’m using. Here goes.

Explore/Expand/Extract

Product development proceeds in three phases:

• Explore–the risky search for a viable return on a viable investment. Successful exploration is unpredictable, so the highest expected value strategy is to reduce the cost of experimentation and put a little investment into many, uncorrelated experiments. If you’re lucky, one of these experiments turns out to be unexpectedly successful, which leads to:

• Expand–now things are going nuts (think Pokemon Go or Facebook Live Video). Unanticipated bottlenecks appear. All you have time for is to eliminate the next bottleneck just before it derails you. Once growth becomes routine, it’s time to:

• Extract–now the shape of the problem and solution spaces are clear. One euro in equals three euros out. Playbooks emerge: here’s how you roll out the service in a new city. Economies of scale matter: delivering the service at lower cost is more profitable.

There’s hours and hours of cool stuff implied by this model, but that’s the basics.

Lessons from the Triathlon

• Each phases requires different equipment, different technique, different training. Software product development requires different tools, techniques, and value systems in the three phases. (The answer to the ??? in my notebook is “Efficient Extraction”.)

• You can’t mix the phases. Just because biking is the fastest phase in kilometers per hour doesn’t mean you can go faster by taking your bike into the water. (I was dealing with a project prematurely moving from explore to expand just yesterday.)

• The transitions are awkward, but they count. If a project goes from one phase to another and the team doesn’t notice, it’s like trying to swim on a bike. Not so good.

• Only organizations that can execute all three phases can complete a race. If you just don’t feel like running, then it’s not a triathlon.

• Unlike a triathlon, the same people don’t have to complete the phases. I specialize in exploration. As soon as expansion starts, I’m ready to start exploring again. As long as there is someone I can hand off to, this works fine.

• The relative progress of the three phases roughly maps. Exploration (swimming) seems slow. Experiment after experiment fails. False hopes are dashed. Repeatedly. Then comes the “overnight success” of expansion (biking). Progress that would have taken months now happens in days or hours, just like a mile on a bike is way shorter than a mile swimming. Extraction (running) is more of a slog than expansion.

• ??? [hoping for suggestions from actual triathletes or poets]

Application

A few highlights:

• Choose exploration. If you’re extracting and starting to feel hopeless or burned out, deliberately switch to exploration.

• Accelerate experimentation. If you’re following all the rules and your experiments aren’t getting anywhere, break the rules to experiment faster. If experiments aren’t likely to last, quick and easy won’t cost you in the long run.

• Transition. If things aren’t going smoothly, see if you’ve changed phases without noticing. If so, change the rules to match the phase you’re in.

I’m only four months in, so I’m sure there’ll be lots more to come. I hope I’ve convinced you that this really is an awesome awesome stick.