Caterpillar to Butterfly

The Evolution of Specialist Expertise

As you grow from small to medium, you’ll notice some work getting done that requires specialist expertise—database administration, design, quality assurance, compliance, security. When the organization was small, it was fine for this work to be done kinda okay. When the organization grows to medium size, though, the gap between experts & hobbyists starts to show.

I had a fantastic week of consulting here (hope I can tell you where “here” is later). It’s a growing org hitting scaling discontinuities. One diagram I drew Monday was used all week, so I thought I’d present it here.

---

Thanks to today’s sponsor Tuple.

Tuple is developer-grade screen sharing, for pairing and more. They interviewed me for the Distributed podcast, and I went deep on software design as an exercise in human relationships and distributed work. Check it out here.

---

Specialist Team

And so we do the logical thing, we hire experts & promote the best in-house experts, forming a specialist team.

Good! Now the work is being done by experts & the individual teams have less to do.

The incentives of the specialist team don’t perfectly align with the client teams, however. The specialist team wants testing or design or database administration or compliance or security to be done right, not the janky way it was being done. If the work is rushed, the specialist team is going to have to fix it later. The client team wants their requests to be handled quickly, to be unblocked. As long as the solution is good enough, worse sooner is way better than perfect later.

Buffers

As work piles up, a black market in specialist work emerges. “Hey, can you just…” Words to freeze a specialist manager’s heart. No, if we do you a favor we’ll have to do everyone a favor & then nobody will know what’s going on.

I know—legibility! We’ll put the requests in a clear, public queue, sort them by overall priority, & then work on the most impactful requests first.

Who could possibly object to doing the most impactful work first? What, you want us to do less impactful work?

Buffers Feed Themselves

There are always way more requests than we have specialists to service the requests. No matter how many more specialists we hire, client teams’ appetites instantly consume the available capacity & then some.

Self Service

You can go around the “bigger org→bigger buffer” loop many times without noticing that it isn’t providing the overall org, the client teams plus the specialist team, with more capacity. The request queue grows, wait times stretch, the incentive for the black market grows no matter the policing put into place. This is the problem that emerges when going from medium to big.

The key insight is that only resource that scales with the client teams are the client teams themselves. The next move is for the specialist team to make their expertise self-service. Build the tooling/frameworks/training that allows client teams to mostly satisfy their own requests.

Being able to draw this transformation easily doesn’t mean it’s easy to execute with real people in a real organization. Building self-service tools is bound to interfere with the execution of queued tasks, which were already being serviced too slowly. Client teams are going to need training & consulting to take back up the responsibilities they gave up with the specialist team formed. Client teams who did have their requests serviced had gotten used to bespoke solutions. They must learn to be content with the generic (but scalable) solutions now being provided.

Iteration will go a ways towards soothing the ruffled organizational feathers. Resist the temptation to self-service all the things at once, pausing all requests for an extended period. Find the costliest, most frequent requests to self-serve first. As the specialist team proceeds, they will need to re-work some of their earliest self-service efforts. That’s okay. It’s better than an organization-halting pause.

Conclusion

In the room this week, we just called this concept “the three pictures”. For general consumption, it makes more sense to me to use the analogy of the caterpillar forming a chrysalis & transforming into a butterfly. Staying in the chrysalis, familiar as the pain may become, prevents the whole organization from growing wings & flying.