The Trough of Despair

The price of improving structure

Things always get worse before they get better

—The Sad Truth of Software Design (also lots of other activities)

You have a design. You need it to be better at supporting new features. How do you get from here to there?

Here’s the Sad Truth of Software Design—it always gets worse before it gets better. We can ignore the Sad Truth, try to leap directly to the better design, but reality always gets a veto. Leaping creates risk.

Software design improvement always looks like this:

1. We got to the design we have by (attempting to) improve the previous design.

2. When we set out to improve the design, things will always get worse before we see improvement.

3. We choose the size of the steps we take to get, eventually, to improvement.

It is the designer’s job to envision the desired state. It is also the designer’s job to design the shape of the curve, subject to reality’s annoying veto.

Vision

The obvious job of the software designer is to imagine what a better design would look like. The key question for the designer is, “What would the system’s structure need to be so that <some feature> would be no harder to implement than necessary?” (It’s a bit surprising when designers don’t ask this question, instead simply asking, “What should the design look like?”—for what purpose?)

How high do you reach? You as a designer have a wide range of options.

But this is only the beginning of the designer’s job.

Worse

That initial dip is the price you inevitably pay for improvement. If you built a factory you’d expect to pay first, see widgets later. How much you pay & when, how many widgets you see & when, those are the parameters that determine whether you’ve made a good investment. So with software design—how much you pay & when & how many features you see & when, those are the parameters that determine whether you’ve made a good software design investment.

First you have to get through the Trough of Despair:

This is the period where everybody but the programmers are saying, “Why are we slowing down? We need to go faster, not slower.” The Trough is characterized by 2 factors under the influence of the designer:

How much worse is it going to get? How long is it going to get worse?

The designer might decide to reach for a lower eventual goal in order to rein in how much & how long things are going to get worse.

Better

The Trough isn’t infinite, at least if you remain employed. Eventually the design gets better. The key factors under the software designer’s influence are:

• How soon until things are at least better than they were?

• How quickly do things improve after that?

The designer can’t fully control either factor, but they can change the sequence of the project to change time-to-first-value and rate of improvement.

Steps

The final factor under the designer’s influence is the step size of the transformation. How much of the design will be changed between (for example) deployments. (The Empirical Software Design answer is, “Tiny steps. No, smaller than that. Then get good at taking steps quickly.”)

Succession

All of these factors add up to the design skill I call “succession”. For a desired end state:

• What are the steps to get to that end state?

• How are the steps ordered?

Software designers can influence design transformations by:

• Choosing the improvement in structure based on what features are coming, how certain those features are, how cheap they need to be to implement, & how the organization will react to the design transformation, especially the Trough of Despair.

• Choosing a succession to make the “getting worse” phase acceptable.

• Choosing a succession to make “how much worse” acceptable.

• Choosing a succession that achieves first value in an acceptable time, and for the “right” features.

• Choosing a succession that achieves acceptable rate of improvement.

• Choosing a step size that balances risk, human factors, & overhead.

Software design isn’t just designing software. Anyone who says different is selling something.