First publishing February 2016

The above was an argument for pre-emptive, speculative performance tuning. If this product is successful, goes the thinking, then data write performance is going to become a bottleneck. We don’t want to have pause user growth to switch to a new database. Let’s just fix the bottleneck now.

I can empathize with the sentiment, but I think this line of reasoning creates risk & reduces profit.

Review

Going from exploration to expansion always creates the risk of uncovering new bottlenecks. Fixing bottlenecks quickly so extraction can commence is a more realistic goal.

To review, product development proceeds:

1. From exploration—the risky search for a viable return for a viable investment

2. To expansion—the elimination of bottlenecks to growth

3. To extraction—where profitable growth continues

Can’t Jump To Expand

The system design rules change between the three phases. In exploration, anything goes as long as it reduces the cost of experimentation. Use infrastructure that doesn’t scale if it accelerates experimentation.

The transition from exploration to expansion is tricky. The activities & values that resulted in successful exploration become dangerous during expansion. Exploration requires diverse, tangential thinking and experimentation. Expansion requires singular focus on removing the next bottleneck just before it chokes growth. Continuing to experiment distracts from this focus.

The activities & values that make for successful expansion, however, endanger exploration deployed prematurely. Doing a better job preparing for future growth slows experimentation, reducing the chance of success.

Success

The lament above, that during traction is a “bad time to stop feature development” is perfectly understandable. You’ve been experimenting for months or years. You’ve begun to despair of those experiments ever paying off. Suddenly you’re on a hot streak. Everything you try works.

Who wants to stop during a hot streak? (In poker we call this “playing the rush”.)

You can’t create infrastructure that eliminates all bottlenecks. You don’t know the exact circumstances of those bottlenecks. You don’t know what data distributions look like, usage patterns geographically or by time of day or day of week.

Universal infrastructure is under-constrained, does work it needn’t do. That extra work perversely creates risk in the precise situations we need to overcome now that users have shown us what those situations are.

Conclusion

The best we can hope for is:

• To repair emerging bottlenecks quickly so we can get on with extraction. If this requires that we pause or throttle growth so we survive, that’s the price of success.

• To permanently repair bottlenecks that “rhyme” with past bottlenecks, but this as an Extract project.

This article was first published in 2022. It has become newly relevant because remote development environments turn out to be key enabling infrastructure for remote genies.

When I joined Gusto 3 and a half years ago I got to program production Ruby for the first time. I was excited to get started. Alex, a senior developer, walked me through the steps required to get tests & an IDE running on my laptop. All was well for a few days.

Then I updated the repository & the tests wouldn’t run. “Oh, yeah,” Alex said, “You have to delete this file & re-run that script. No problem.” I did. No problem.

Come the next Monday, though, it was, “Hmmm, I haven’t seen this before.” Type type type. “Let me ask somebody.” Type type type. “There, all fixed.”

Every few days, my development environment got borked. Often it was a quick fix. Sometimes not. But always time taken from one of the most productive programmers (Alex, not me.)

Welcome to new sponsor CodeRabbit. As I continue to write more code with AI, I long for a "what all did the genie do?" tool.

CodeRabbit offers this visibility, providing code reviews right in the IDE (for individual coding insights) and on PRs (to ensure consistency and quality across your org). CodeRabbit fits into existing toolchains and git-based workflows, can be personalized for individuals and teams, and can be used in conjunction with some of our favorite AI agents (including Claude Code, Augment Code, Cline, etc.).

Sign up today and see how CodeRabbit can cut bugs and review time in half (while also letting you keep tabs on your augmented coding projects).

Structural Problem

Developers spend hours a week just being able to program on laptops. Laptops have advantages as a programming location: control, latency, flexibility. However, we just don’t seem to be able to keep them working without leaving the floor covered in yak hair. We’re programmers. Can’t we just solve this problem?

No. As I was getting acquainted with GitPod the other day, it struck me that there is no direct way around the problem of a stable, predictable development environment on a personal machine.

Variability, Interconnection, State, Irreversibility

I learned the following model from Professor Enrico Zaninotto, at the time the Dean of Economics at the University of Trento. Large, complicated system aren’t controllable when they combine 4 properties:

• Variability — the environment around the system changes out of control & visibility of the system.

• Interconnection — the elements in the environment affect each other.

• State — the system can be in one of many states at any given time.

• Irreversibility — changes to the system, once made, cannot easily be unmade.

When all 4 properties are present, there is no way of predicting how the system will behave. Change the system, there is no way of predicting how the overall behavior will change. You can’t understand the system well enough to fix this problem.

His first example was mass production gaining control by reducing the number of states. All Model T Fords were black not because Henry Ford liked black so much, but because “paint or no paint” is easier to manage than “how much of each color of paint to do we have/need?”.

(Professor Zaninotto was trying to understand the effectiveness of Extreme Programming. Software development has all 4 properties. XP, in this model, works by introducing reversibility to software development decisions.)

Applied to Development Environments

How does this model apply to development environments?

• Variability — Yes. Today’s style of development takes advantage of a long & complicated supply chain. All the systems we rely on change independently.

• Interconnection — Yes. The parts of our system rely on each other.

• State — Oh my yes. The current state of a development environment is enormous. What’s been downloaded, initialized, running, messed up?

• Irreversibility — Surprisingly, yes. “Oh, upgrading this messed up that? Just reinstall the previous version.” Yeah, right.

How does this help with my Monday morning laptop blues? We still want to take advantage of variability. We aren’t Forth programmers writing all of our own stack down to silicon. We can’t avoid interconnection.

That leaves us with state and irreversibility. Pre-built development environments of the sort offered by GitPod give everyone using the same environment the exact same state. If today’s pre-build doesn’t work, we can easily load yesterday’s & get back to a known-good environment.

And Then…

I’m excited to see how this plays out. The first advantage for programmers is the time they save. I’ve seen different numbers for the current cost of keeping a development environment working, but it’s certainly tens of percent.

More important, I suspect, will be all the programs that folks will try writing because they are no longer afraid of wasting another 4 hours before giving up. One of those programs that wouldn’t have been started will turn out to be exceedingly valuable.

Interesting problems remain. How does remote development retain the latency of local development? How do you share a remote environment? How do you satisfy the needs of beginner, intermediate, & expert programmers at the same time?

Republished from July 2014.

"Jean-Louis wants to see you in his office." My boss Eagle Burns' bald head disappeared from the door to my office. Something about his tone suggested that righteous indignation, which I had been nursing for several days, was not the right attitude to pack for my trip. I started getting scared.

It was Apple 1988.

was head of engineering. He had a reputation as a fire-eater, someone who chewed fools up and spit them out. I had been foolish, it suddenly occurred to me. Do the math.

I was a young, cocky engineer. I was sure I knew better than anyone what needed to be done. When a decision came down that I disagreed with, I shot off an email to JLG that made it clear just how stupid the decision was. A day later, I was summoned.

JLG's office was on the same floor as mine, but I had never approached the sanctum. He was at least four levels up from me in hierarchy. Rumor had it that decorating his office had cost $100K, which was a fortune since everybody else was stuffed in cubes.

When his admin passed me in, though, JLG was avuncular. "I read your message. I'd like you to think about it from my perspective for a moment. What am I going to do if I agree with you?"

No idea. He must have been used to dumb silence.

"I'm going to ask my report for a proposal. He's going to ask his report and so on [yes, completely male. Plus ca change...]. Eventually Eagle is going to ask you what to do. In all those layers, who knows what's going to happen to the question.

"Here's what I want you to do. Rewrite your message and make a proposal for what we should do about it. That way you'll save us a week and you'll get to solve the problem you want to solve."

I did. I don't remember what the issue was or what I wanted to happen or what actually happened. That was lesson number one—it's really not that big a deal, or at least hardly ever.

Lesson number two was to always bundle a proposal with a complaint. I've used that style daily in the intervening quarter century. If I need to vent or I need to think something through, I do it in private. If I'm going increase the scope of a communication, I always say what I would like to happen.

Lesson three was the most important. Those words, "Think about it from my perspective," were a revelation. You mean you have a point of view and it's just as valid as mine and it might be different? I've spent that same quarter century trying to absorb and apply that lesson. Communicating begins by putting myself in the place of my reader, not demanding that they put themself in my place.

Complain and propose.

Bonus lesson for when I was the old guy: don't get caught up in young hotshots' emotions. Assume good intentions and lack of skills. Thanks, JLG.

This is from a new archival source—the original WikiWikiWeb. The page doesn’t have a creation date that I could find but it would be around 2000.

The topic seems apropos to The Forest & The Desert. In The Desert you don’t dare let go of the yoke because you already don’t have enough control. In The Forest, you can need less steering & it turns out that less steering leads to greater satisfaction all around.

An airplane's direction and altitude is controlled primarily by a 2D steering wheel called a "yoke" (plus other inputs like the rudder pedals, which don't bear on this story). When my ex-father-in-law first showed me how to fly, I immediately grabbed the yoke with both hands. We bounced around a bit. Then he showed me how much smoother I could fly straight and level with only one hand on the yoke.

An airplane is much more sensitive to controls than a car. Small inputs create large changes. Coordinating the pressure of two hands generates many, many small inputs. Counter-intuitive as it feels at the time, 10,000 feet up in a tiny little cigar tube, letting one hand go is the right way to solve the problem.

Now, when you do a maneuver, you do hold the yoke with both hands. Then you need the power provided by both armsful of muscles. At least you want that power available, and you are willing to pay the price in a little wiggliness, at least for the duration of the turn.

An ExtremeTeam is sensitive to small inputs. Much better to let one hand go and trust the self-organizing culture they have developed. Most of the time.

Another example of this is my rolling suitcase. The wheels are too close together, so it often starts oscillating between the wheels, eventually falling over. I tried grabbing tighter, which started the oscillation sooner and toppled the suitcase sooner. The solution is to pull the dang thing with one finger. Occasionally (but much less often) it still begins to wobble. I slow down until it stops.

I replaced the suitcase with a 4-wheeled Tumi. Problem solved.

First published Dec 8, 2021. The topic of background work came up in the context of the way LLMs make such work so much cheaper than it used to be. The same rabbit hole that would have taken a week to explore can now be excavated in an hour or a minute. Background work should become more prevalent.

A student had a code review where the reviewer said, “This variable needs to be explicitly initialized to nil.” The usual course for a junior engineer would be to change the code. Instead, the student did some research, then responded, “According to the language spec, the variable is initialized automatically.” That research was more work than strictly called for but it created value for both parties.

“Background Work” is the work you do over and above what is strictly required to complete a task. It may be done for learning, to satisfy curiosity, or just because you forgot to stop working when you were done. I’ve noticed that the most accomplished programmers make a habit of background work.

When I saw the exchange above it brought to mind times when I have seen engineers I admire dig into the background of a topic. “I wrote a little prototype PHP-to-JavaScript translator and here’s the performance data,” “I thought this might come up, so I re-read Knuth on the topic,” “I wasn’t sure what I thought so I wrote this essay.”

At first I was going to call this, “extra work,” conjuring visions of late nights and weekends, but on reflection that’s a misleading name. First, leaders seem good at managing their energy, so if they do some background work “off the clock”, they find ways to recharge. Background work also has a habit of paying off quickly, so that the leader accomplishes more in the ensuing day or week or month.

My own experience of background work, though, is that sometimes my curiosity gets the best of me at inopportune times, like 4 AM, and I need to just go do the work. Greek rhetoric teaches that persuasion comes from a combination of ethos, logos, and pathos, appealing to listeners on the basis of personal credibility, logic, and emotion. Background work levels up logos but I think its biggest effect is in establishing credibility, ethos. Someone who habitually digs deeper than strictly necessary is worth listening to. When they know, they will speak with authority. When they don’t know, they’ll say so and then, if they are curious, go do the work to find out.

First published May 2017. I’ve been trying to write about authority, responsibility, accountability, & other properties of productive, functional relationships for a couple of decades. I hope I’m getting close, because there’s a chapter in the new book labelled “Authority & Responsibility”.

Just before writing this I had been on a teaching trip in central & southern Africa. Met some life-long (well, the remainder of it anyway) friends, made some music, & learned about my narrow understanding of people & culture.

I was terrified much of the time I was walking in Lagos. Traffic was chaotic. Drivers were hyperaggressive. Roads were narrow.

There was one time I wasn’t worried: bareheaded on the back of a motorcycle, dodging and weaving through that self-same traffic.

Partly it was because the first thing I did on arriving in Lagos was go to church (a whole other story), which I figured improved my chances of surviving and at least got me into a desirable afterlife should things go sideways. Partly, though, it was because of principles that I find both useful when followed and frequently violated in organizations as they scale.

Thought Experiment

Staying on a motorcycle for the moment, let’s say I ask you to take my bike to the pizza place to pick us up some deep dish. I’m in a hurry, though, so while you can steer, I’m going to control the accelerator from here on the couch. How do you feel about this arrangement?

Okay, little tweak. I’m still going to control the throttle, but I’m going to ride the bike with you. Feel different? Now you get to control both steering and throttle. Different?

Authority and Responsibility

If you work for yourself, by yourself, you don’t even need concepts for authority and responsibility. You make the decisions and do the work and there’s no one else who possibly could. You experience the consequences, all of them, and there’s no one else who possibly could. Becoming aware of authority and responsibility results from scale.

Two Principles

The problem comes because people are people. We are all subject to loss aversion (where losing $1 feels worse than gaining $1 feels good [ed: see also ergodicity for more perspective]). We are all tempted to free ride (let others do the work while accruing the benefit ourselves). At the same, we all want to be significant, to do important work and be recognized for it.

Two organizational principles reduce the chaos caused by our natural human tendencies:

• Responsibility aligns with authority. If I have the authority to make a decision, then I experience the consequences of that decision.

• Responsibility comes as a package. If I get the benefits of a good outcome, I also feel the pain of a bad outcome.

A friend called me the other day. She’d been asked to lead a big project to convert the company’s CRM. “Okay, what are the business goals? I’ll be glad to get to work on this.” “No, no, you have to use Microsoft Dynamics CRM.” Bzzzzt.

This situation violates alignment of authority and responsibility. My friend will certainly be punished if the project goes badly, but the single most consequential decision has already been made. Also, this smells like a case where if the project goes well the executive sponsor will get the kudos/bonus/promotion but my friend will get fired if the project goes badly.

She pushed back, pointed out that she was in a no-win position, and told them that without authority to match her responsibility, she wasn’t going to do it. After whining about her not being “a team player”, they gave her the authority.

Steering and Throttle

Now let’s put you back on that crazy remote control motorcycle. When I control the throttle from the couch, I have authority without responsibility. I can make decisions and take action, I benefit from the upside (pizza sooner), but I don’t share in the pain if you crash. Put me on the bike with you and at least we both suffer road rash together. You’d rather control steering and throttle yourself, but it’s better if we share a fate than if we don’t.

That was my thinking in Lagos. If I’m walking on the side of traffic, the worst that could happen to the person who hits me is that they’ll have to repair a dent. On the back of a bike, though, I’m guaranteed that there is one person in the world as interested in my safety as I am.

Followup

Organizational alignment is a huge topic. I’ve only chipped the paint here. More topics for future notes:

• How to realign authority and responsibility

• My shaky relationship with responsibility

• Accountability is a good thing and it’s not the same as blame

First published August 2016. Content warning: divorce & raw emotion.

I pick pieces to pull out of the archives on impulse. If I remember something I said, or if I point someone else to it for whatever reason, then the piece goes in the queue to be republished. I recently passed the eighth anniversary of rebooting my life & remembered this essay.

I’m flying to Menlo Park with two guitars, a big suitcase, and the backpack I got in Facebook Bootcamp five years ago. The rest of my stuff forlornly occupies a third of a 10x10 storage unit. The nest empty, I am flying the coop (to nearly mix a metaphor).

I’m writing about my divorce here because I need to write and I’m not comfortable going public public. I’m sure I’ll have lessons learned, cautionary stories, and sage advice later. Right now all I have is an emotional grab bag. Stick my hand in, pull it out, and see what’s there. Fury. Sadness. Elation. Fury again. It’s like eating a sadistic ice cream sundae blindfolded.

Okay, one lesson: presence. I spent the last week doing things for the last time: my last Wild River Brewing root beer, my last pat of the old farm dog Patch, my last whiff of the Rogue River, a smell that carries me instantly, effortlessly back a half century.

Here’s the thing. Doing these things (and a hundred like them) the last time is exhausting. I’d tear up, blubber, and eventually let the feeling pass and move on. About the tenth time in an hour, though, and the energy tank is dry dry. I have to truncate.

I’m not drinking Wild River root beer for the last time, I’m just drinking root beer. And it’s damn good. I’m patting my dog. I’m feeling the river caress my fingers. Call it “insight by exhaustion”. Too tired to continue living in a future where I’m no longer doing things I love, I live in the present moment because I have exhausted (literally) all the other possibilities.

At first I was afraid I’d be missing something. This is the last time, dammit. No. Turns out tears and self-pity spoil the flavor of root beer. It tastes better straight.

I still float back to the future sometimes, like just now as I was writing and got all watery. However, I’ve had moments of the joy of pure experience. Right now, Lamy in hand, I’m enjoying the hospitality of United Airlines.

What’s next for me? If you’re asking that, apparently you didn’t read the rest of the note. If I’m answering it, apparently neither did I. I’m going to spend two months in NYC finding out if I’m a coach, a team coach, a manager, a product manager, a corporate shill, or something else I haven’t thought of. But for now, one moment at a time.

I heard a comedian joke that the correct response to, “I’m getting divorced,” should be, “Congratulations!” However bad it is, it must be better than what went before. And so it was for me. I had wild mood swings but the average was much, & remains much, better than before.

And to those asking why I didn’t leave sooner if it was really that bad, you don’t leave when it gets bad, you leave when you’re ready to leave.

First published June 2017. Apparently I was already toying with the tidying metaphor, even if I could have sworn I didn’t start using it for another 4 years. Memory is a funny thing.

I have been tidying up server code and loving it. I’ve tidied up for thousands of hours over my career, and I have some rules for making this style of cleanup safe, fun, and efficient.

Tidying up works through a series of small, safe steps. In fact, Rule #1 is If it’s hard, don’t do it. I used to do crossword puzzles at night. If I got stuck and went to sleep, the next night those same clues were often easy. Instead of stressing about the big effects I want to create, I am better off just stopping when I encounter resistance.

Tidying up is concave [ed: this was also when I was digging in to 3X: Explore/Expand/Extract] in the sense that you have a lot more to lose by a mistake than you have to win by any individual success (more on that later). Rule #2 is Start when you’re fresh and stop when you’re tired. Get up and walk around. If I don’t come back refreshed, I’m done for the day.

Tidying up can happen in parallel with development, but only if you carefully track other changes (I messed this up with my latest diff). Rule #3 is Land each session’s work immediately (after review, natch). Unlike feature development, where it sometimes makes sense to land only when a chunk of work is done, tidying up is time based.

Tidying up requires little effort for any step, so I am willing to discard any step at the first sign of trouble. For example, Rule #4 is Two reds is a revert. If I tidy, run the tests, and encounter a failed test, then if I can fix it immediately I do. If I try to fix it and fail, I immediately revert to the last known good state.

Tidying up works even without a vision of the shiny new design. However, sometimes I want to see how things might play out, so Rule #5 is Practice. Perform a sequence of tidyings and revert. The second time will go much faster and you’ll be more familiar with which bumpy spots to avoid.

Try it. Move the declaration of a temp adjacent to its first use. Simplify a boolean expression (“return expression == True” anyone?). Extract a helper. Reduce the scope of logic or state to where it is actually used.

The Rules

1. If it’s hard, don’t do it

2. Start when you’re fresh and stop when you’re tired

3. Land each session’s work immediately

4. Two reds is a revert

5. Practice

Coda

I’ve made architectural changes strictly by tidying. I’ve extracted frameworks strictly by tidying. You can make big changes safely in this style. I think this is because, while the cost of each tidying is constant, the payoff is power-law distributed. I need both data and a model to explain this hypothesis.

Is your business looking to juice the value they get from software development. Consider my new workshop—Lead Geeks With Purpose. It makes an excellent holiday gift for the CEO who has everything.

Leading Geeks with Purpose

Leading Geeks with Purpose: The Brass Ring Approach is a one-day workshop for non-technical executives who want to get the full returns from their technology investment.

By Kent Beck

First published June 2017. I had been coaching at Facebook for 6 years at this point. I wanted to reflect on how the most effective people got to where they were.

After coaching hundreds of engineers, I got to thinking about why more people don’t reach the top levels. Part of it is the curve—the top 3% is always going to be the top 3%. Part is bias—from childhood through education through a career some talent is repressed based on gender, melanin, or just the vagaries of geography.

Even with all the privileges, most folks don’t make it to the elite level. Some folks do make it, though, but to get there they have to pass through the Gates of Paradox:

• Longevity/diversity. Elite engineers stick with projects long enough to see the consequences of their decisions. Insight is best mined deep in maintenance. At the same time, elite engineers have enough diversity of projects to separate context-dependent lessons from more general ones.

• Success/failure. Elite engineers need to succeed enough to maintain confidence and initiative but they also need to fail enough to question themselves and their assumptions when doing so is valuable.

• Mentored/self-directed. Most elite engineers can point to one or more mentors who changed the trajectory of their career. Elite engineers are also self-directed learners, trusting their curiosity as a compass pointing towards future growth.

• Urgency/slack. Elite engineers work hard, but lots of folks work hard. Elite engineers have the habit of investing the margin in personal growth. When that odd hour or two isn’t likely to yield a great new feature, it can still yield useful learning.

While all elite engineers face these contradictions, there are as many paths through them as there are engineers. In conversation, some patterns emerge:

• Longevity/diversity. Pick projects with a short Time to Production Feedback (T2PF). Structure existing projects to slash T2PF. Once you’ve learned your lessons, move on to a contrasting project.

• Success/failure. Cut your losses but learn your lessons. Focus on what you could have done differently.

• Mentored/self-directed. Build and maintain relationships with engineers you admire. Trust your curiosity as a compass pointing to your future growth.

• Urgency/slack. Work hard on your main responsibility, but take time to learn when that’s the best use of marginal effort. You’re worth it.

First published in 2017. I was already wrestling with the question of “when” as the critical design decision. Also, you can see the roots of the “empirical” in Empirical Software Design here.

Commission me to deliver a custom talk to your development organization. Your choice of format, topic, & location.

My intention in this note is to reclaim the phrase ad hoc from those who use it as a pejorative, especially as applied to infrastructure. Instead, building infrastructure ad hoc is the safest, most efficient strategy. It carefully balances the risks inherent in creating infrastructure, stages investment, and realizes economies of scale.

“Infrastructure”?

Imagine we have lots of code like this:

value = memcache.get(key)
if (! value) {
  value = database.get(key)
  memcache.put(key, value)
}
...value...

This code requires three network calls in the case of a cache miss, expensive network calls going from the front end servers to the data storage servers. Suppose we replace these 3 calls with a single call:

value = cache.get(key)

(Assuming we set up the cache to know how to fetch its underlying data.)

The front end no longer has to care about how to fill the cache. The cache machines can be located close (in the network sense) to the database machines, so a cache miss no longer incurs as much of a penalty.

Once the cache is in place, we can continually improve cache hit rates, reduce latency on misses, and improve behavior for “hot” keys all without changing the front end code at all. A small improvement to the cache’s behavior results in a large return on investment because of scale.

That’s what infrastructure is: a (mostly) isolated bundle of related functionality achieving economies of scale. Those economies may come from reduced capital expenditure (fewer servers needed for the same data) or reduced operating expenditure (shorter mean time to repair because the functionality can be economically refined).

So far I haven’t said anything controversial (I hope). But…

When should such infrastructure be built? What goes into the infrastructure? These are questions that provoke furious debate.

The Myth

The myth is “build the infrastructure and then build the apps on top of it”.

This trick never works (for some value of “never”). Building the infrastructure is a never-ending project. When is it good enough for apps? Tomorrow. Always tomorrow. What should go into the infrastructure? More. Always more.

Infrastructure is supposed to be built more carefully than apps. Apps gain value from exploration & thus emphasize latency of experiments. Infrastructure gains value from scale. Mistakes or inefficiencies scale too. However, infrastructure is not built with perfect knowledge. Infrastructure needs to grow & adapt.

“When?” should, then, be answered with “some sooner, some later”.

Ad Hoc Infrastructure

I’ve heard people critique as system as “ad hoc”. Hang on!

ad hoc /ˌad ˈhäk/
adverb
when necessary or needed.
adjective
created or done for a particular purpose as necessary.

I like both those definitions. Why would you create infrastructure before it is necessary? Why would you create infrastructure that’s not needed?

The “ad hoc” answers to the infrastructure question:

• When? After an application already has the functionality, but in a clearly inferior form.

• How much? Just as much as will let the application return to only doing its job.

Ad hoc infrastructure achieves economies of scale by using regret as the primary motivation. “I wish we had built this a different way...” leads to building a small part of the application in a better way, a way that just so happens to be useful for other apps after a bit of tweaking.

Subscribe now

First published July 2017

I was twelve the first time I got paid to work. I was helping renovate a house, out in the burning sun, a wooden-handled scraper/wire brush heavy in my hands.

My first target was an outdoor bannister. I gave it a few swipes, got hot, and went inside to find a cooler job. Dick called me back out. “You need to scrape and brush until all the loose paint chips are gone. Otherwise, we’ll paint it, it’ll look good, but next spring it’ll start flaking and we’ll have to do it again.”

I got back to work, finding, despite the heat and the sweat, satisfaction in doing a job that would last.

Test Coverage

Studying better engineering is based on a collective realization that we (sometimes, more on that later) need to shift our development efforts towards the long term. Quality matters. But how, with a measurement-oriented culture, do you measure quality?

The consequences of code quality are either hard to measure or lag far behind changes of quality:

• How easy is it to add features?

• How many SEVs did it cause?

• How quickly can people learn it?

• (Ultimately) How much community does it build?

Test coverage, on the other hand, is easy to measure and measurable today. In the absence of changes to design practices, however, test coverage is at best worthless and at worst a distraction from more important issues.

Production Predictor

The ideal test suite is an instant production predictor. If the tests run, production will be fine if you push. If the tests fail, a production push would fail. The test suite will never be a perfect production predictor, but you can continually make it better.

The problem is that by the time we get to the test suite, it’s already too late because of the Universal Law of Testing:

The code is composed of large elements (functions, classes, files, modules), unnecessarily-shared mutable state, complex control structures, vestigial code, duplicated code, and implicit coupling. You don’t have bugs because you don’t have tests. You have bugs because you write code that makes it easy to have bugs. You have a design problem, not a testing problem.

Prescription

• Improve cohesion by reducing the size of elements

• Unify duplicated logic

• Replace duplicated conditionals with polymorphism

• Use immutable data structures where feasible to eliminate aliasing errors

• Eliminate dead code

• Rationalize naming so the right names are likelier to be guessed

• Simplify control flows where possible

Once the system is closer to being composed out of cohesive, loosely coupled elements, then and only then does test coverage begin to correlate with the predictive power of the test suite. As long as changing a line has a significant chance of unexpected consequences, whether that line is covered or not doesn’t really matter.

Priorities

Another way of interpreting efforts to improve engineering is that they recognize that parts of the code base have made this shift to longer half-lives. The cultural challenge is that de-coupling is hard work, provides no immediate payoff, can disconnect from actual progress, and is hard to measure. Test coverage, on the other hand, is easy to measure, easy to make progress on, and easy to justify short-term.

I’m 100% in favor of slowing down to speed up, where we know code will be valuable long-term. I’m in favor of better test coverage. But you have some hard, sweaty, unglamorous work to do first. Otherwise, you risk testing the hell out of error-prone code, only to have to test the hell out of tomorrow’s error-prone code. And the next day’s. And the next.

P.S. One Exception

One scenario where test coverage provides value today is answering the question, “You know this line I just changed? Is it being exercised?” If it isn’t, it’s time to ask, “Am I comfortable with this change first being executed outside of my direct control when it’s in production?” Maybe yes, maybe no, but test coverage eliminates a class of errors where you accidentally didn’t even check.