Recruiters and sales people also game the system. If recruiter is measured by lead time and closure of positions, we end up hiring the below-average people to fill in the slots. And talk about system dynamics from there.
This is great! Measuring output is an artifact from the pre-digital industrial age, when output was a decent predictor of outcome. Operational risk was paramount. (If we can produce a black Model T efficiently enough, the middle class will buy.)
Entire companies are structured around optimizing efficiency of output. This makes measuring efficiency of outcomes difficult. Engineering teams may very efficiently build features customers don’t care about. This means you need other disciples involved. Company structure makes this difficult.
"If you consider why a startup moves so fast and executes so well, it’s because they have to do so out of necessity, even if they do not measure this." - Given that 90% of startups fail (couldn't get specific numbers for software startups), I am not sure that most startups actually end up delivering positive customer impacts even though they might deliver a customer facing thing per week.
At the end of the day, Goodhart's law will always prevail.
I would like to agree with others that pointed out that measuring and rewarding of people work can lead to undesirable results in all fields not just in software engineering. Book "The tyranny of metrics" by Jerry Z. Muller has great examples (I like one where surgeons wouldn't operate patient with even small risk so they won't get bad grades).
Most often when people try to measure dev productivity, they use other fields and industries for comparison like engineering, sales, and support. I think this is what leads to the over arching confusion. I would love to see someone do a study on how to measure productivity in, say, a research and development field like medical developments, or even a field that is more artistic like writing or painting. In my opinion, those fields are much closer in nature to software development than the ones people typically use. It would be interesting to see, say, how a company measures the "performance" of a mystery novel writer or someone who writes screen plays, or how to encourage "better performance" from a research scientist who is trying to develop a new vaccine or a cure for some currently incurable illness. If one of those fields has ways of measuring their employees performance, then perhaps we would have a reasonable baseline of comparison for our industry.
Overall, great article by the way. I'm looking forward to reading part 2!
Thank you for the article. Could you please explain why you put DORA metrics to outcomes? The fact we deployed smth could have no impact on customer behaviour.
Completely agree, or as a friend would have said: "Measure business, not busyness".
I have worked with (outsourced) IT support who were measured on number of tickets closed and that was useless.
The incentive to close as many tickets as possible often resulted in no resolution or a bad one. Or even a closed ticket with a message telling me to open another ticket on another department.
Thanks for sharing your point of view. Very insightful.
There are a few moments in this article that are confusing me though:
1. When talking about how HR metrics fit in the proposed mental model, you mention in the text that “number of heads filled” is categorized as impact but list it in the “outcome” box in the illustration. It feels more like outcome than impact so I am assuming there is an error in the text. Or am I missing something?
2. When you talk about the Space framework, you mention that the people behind it put a warning on “effort and outcome” metrics. Did you mean “effort and output” there? Some outcome metrics could need a warning as well depending on how people could game them though.
3. When you propose to measure the “please the customer once per team, per week”, you imply that this is an “output” metric, but it feels more to me like an “outcome” one. Can you enlighten me on why you’d consider this an output metric?
Nice read. Thanks for outlining the issues with measurements in the software industry, it's a big problem. In any industry.
With highly productive and skilled engineers we can very productively deliver products nobody buys or uses.
The productivity problem is so much engrained in our Taylorism style of management. McKinsey et al still fit into this style of management. We as management are incapable of learning to solve problems, therefore we hire consultants to tell us what to do.
If you want to measure (which is a huge investment for many), then measuring (the right) outcomes in combination with team behavior (e.g. collaboration). Put incentives only on (team) behaviors. Never on outcomes. Lean OKRs is a good framework (but I'm biased).
Separate note: software engineers are often abstracted from outcomes and even more so from impact. Other disciplines are generally making decisions about what to build and how it should function.
Many engineers are simply asked to build the thing. I think at that point the outcomes ICs can be responsible is the output of their work: the thing they were asked to build. Whether or not is has actual business impact is generally someone else's decision to hold to account. From an impact perspective, the developer's customer is the requestor of the feature - the PM or PO or whathaveyou.
I'm not saying this is the way to do it, just kind if taking the temperature of this kind of system for making software. The engineering team in this type of system is NOT responsible for many business outcomes because they're not asked to deliver business outcomes. They're asked to deliver features.
Sadly, they're even less likely to be asked to create impact.
There might be many issues with the Mck article but they try to 'prescribe' a solution which can be operationalized. Unless the software / product industry comes up with better prescription CEOs will continue to listen to the consulting giants. Everyone understands it's important to measure outcomes, but how many orgs are able to do this at scale (it doesn't make sense to count start ups only).
I absolutely agree that measuring outcomes and connecting them to impacts is the way to go in the software business (Jeff Patton has been shouting that from the rooftops for years now). I also noticed that a lot of companies don't even know where to start, and articles like McKinsey's don't make it any easier. One thing I noticed about your article is the conspicuous absence of product management, which is directly focused on this outcome and impact correlation thing, including making sure that product strategy supports business strategy (impacts). I think that may be as much of a blind spot for many engineering leaders as the importance of focusing on teams rather than individuals in product development is for many non-technical senior executives (both are business leaders, and the very distinction between engineering and "the business" is at the root of the problem). I am not saying that engineers can't be great product managers, only that engineering and product management are two very big things that at any but the smallest scale call for a team of specialists working closely together, as everything else in product development.
I feel like you missed the two main reasons we care about developer productivity:
1. Developers want to be more productive. Part of the reason we got into the field is to make things better and more efficient, to solve problems faster using automation. That applies to our own jobs as well, so developers really don't like inefficient processes in their own workflows. Yet if we don't measure this it's hard to know where to focus on improving.
2. Teams want to be more productive and ship more valuable software. We want to add value to customers using our skills at engineering, so if we can do that more quickly that's great. If one team ships 2x as much as another team that could reflect some superior process the former team has that could be more widely adopted.
The problem only comes when you misuse these metrics. Thus it's the "becomes a target" part of Goodhart's law that is bad, not the act of measuring and trying to improve. As soon as we say "lines of code should go up 10%" or "your lines of code was lowest on the team so you're fired" we've created a problem, but that doesn't mean in general it's useless to measure lines of code. For example if we notice aggregate LOC across the company is trending down that could be a worrying sign that we've put in place some processes to slow down development that bear looking into.
Srikanth, you've raised a crucial point. The temptation to game the system exists in various domains, and the consequences can be significant. As we explore the complexities of measuring productivity, your insight underscores the need for thoughtful considerations to avoid unintended outcomes. It's a delicate balance between incentivizing performance and preserving the integrity of the outcomes. Looking forward to delving deeper into these dynamics in Part 2.
This is spot on, great job of balancing unequivocal criticism of the McK "article" (which is really a thinly veiled sales pitch), without getting petty.
Robert (Bob) Lewis has been writing about the perils of metrics for many years. Originally in InfoWorld and then his own online outlet. Here's a search of his content for "metrics" that yields dozens of articles: https://issurvivor.com/?s=metrics
Recruiters and sales people also game the system. If recruiter is measured by lead time and closure of positions, we end up hiring the below-average people to fill in the slots. And talk about system dynamics from there.
This is great! Measuring output is an artifact from the pre-digital industrial age, when output was a decent predictor of outcome. Operational risk was paramount. (If we can produce a black Model T efficiently enough, the middle class will buy.)
Entire companies are structured around optimizing efficiency of output. This makes measuring efficiency of outcomes difficult. Engineering teams may very efficiently build features customers don’t care about. This means you need other disciples involved. Company structure makes this difficult.
"If you consider why a startup moves so fast and executes so well, it’s because they have to do so out of necessity, even if they do not measure this." - Given that 90% of startups fail (couldn't get specific numbers for software startups), I am not sure that most startups actually end up delivering positive customer impacts even though they might deliver a customer facing thing per week.
At the end of the day, Goodhart's law will always prevail.
I would like to agree with others that pointed out that measuring and rewarding of people work can lead to undesirable results in all fields not just in software engineering. Book "The tyranny of metrics" by Jerry Z. Muller has great examples (I like one where surgeons wouldn't operate patient with even small risk so they won't get bad grades).
Most often when people try to measure dev productivity, they use other fields and industries for comparison like engineering, sales, and support. I think this is what leads to the over arching confusion. I would love to see someone do a study on how to measure productivity in, say, a research and development field like medical developments, or even a field that is more artistic like writing or painting. In my opinion, those fields are much closer in nature to software development than the ones people typically use. It would be interesting to see, say, how a company measures the "performance" of a mystery novel writer or someone who writes screen plays, or how to encourage "better performance" from a research scientist who is trying to develop a new vaccine or a cure for some currently incurable illness. If one of those fields has ways of measuring their employees performance, then perhaps we would have a reasonable baseline of comparison for our industry.
Overall, great article by the way. I'm looking forward to reading part 2!
Thank you for the article. Could you please explain why you put DORA metrics to outcomes? The fact we deployed smth could have no impact on customer behaviour.
@Kent Beck
If developers productivity can't be measured then look at the implications:
- Promotion , PIP low performer's and Layoffs.
are done arbitrarily.
- organizations won't accept this publicly because they are scared of lawsuits.
Completely agree, or as a friend would have said: "Measure business, not busyness".
I have worked with (outsourced) IT support who were measured on number of tickets closed and that was useless.
The incentive to close as many tickets as possible often resulted in no resolution or a bad one. Or even a closed ticket with a message telling me to open another ticket on another department.
Thanks for sharing your point of view. Very insightful.
There are a few moments in this article that are confusing me though:
1. When talking about how HR metrics fit in the proposed mental model, you mention in the text that “number of heads filled” is categorized as impact but list it in the “outcome” box in the illustration. It feels more like outcome than impact so I am assuming there is an error in the text. Or am I missing something?
2. When you talk about the Space framework, you mention that the people behind it put a warning on “effort and outcome” metrics. Did you mean “effort and output” there? Some outcome metrics could need a warning as well depending on how people could game them though.
3. When you propose to measure the “please the customer once per team, per week”, you imply that this is an “output” metric, but it feels more to me like an “outcome” one. Can you enlighten me on why you’d consider this an output metric?
Thanks again!
Nice read. Thanks for outlining the issues with measurements in the software industry, it's a big problem. In any industry.
With highly productive and skilled engineers we can very productively deliver products nobody buys or uses.
The productivity problem is so much engrained in our Taylorism style of management. McKinsey et al still fit into this style of management. We as management are incapable of learning to solve problems, therefore we hire consultants to tell us what to do.
If you want to measure (which is a huge investment for many), then measuring (the right) outcomes in combination with team behavior (e.g. collaboration). Put incentives only on (team) behaviors. Never on outcomes. Lean OKRs is a good framework (but I'm biased).
Separate note: software engineers are often abstracted from outcomes and even more so from impact. Other disciplines are generally making decisions about what to build and how it should function.
Many engineers are simply asked to build the thing. I think at that point the outcomes ICs can be responsible is the output of their work: the thing they were asked to build. Whether or not is has actual business impact is generally someone else's decision to hold to account. From an impact perspective, the developer's customer is the requestor of the feature - the PM or PO or whathaveyou.
I'm not saying this is the way to do it, just kind if taking the temperature of this kind of system for making software. The engineering team in this type of system is NOT responsible for many business outcomes because they're not asked to deliver business outcomes. They're asked to deliver features.
Sadly, they're even less likely to be asked to create impact.
There might be many issues with the Mck article but they try to 'prescribe' a solution which can be operationalized. Unless the software / product industry comes up with better prescription CEOs will continue to listen to the consulting giants. Everyone understands it's important to measure outcomes, but how many orgs are able to do this at scale (it doesn't make sense to count start ups only).
I absolutely agree that measuring outcomes and connecting them to impacts is the way to go in the software business (Jeff Patton has been shouting that from the rooftops for years now). I also noticed that a lot of companies don't even know where to start, and articles like McKinsey's don't make it any easier. One thing I noticed about your article is the conspicuous absence of product management, which is directly focused on this outcome and impact correlation thing, including making sure that product strategy supports business strategy (impacts). I think that may be as much of a blind spot for many engineering leaders as the importance of focusing on teams rather than individuals in product development is for many non-technical senior executives (both are business leaders, and the very distinction between engineering and "the business" is at the root of the problem). I am not saying that engineers can't be great product managers, only that engineering and product management are two very big things that at any but the smallest scale call for a team of specialists working closely together, as everything else in product development.
> Who wants to measure productivity, and why?
I feel like you missed the two main reasons we care about developer productivity:
1. Developers want to be more productive. Part of the reason we got into the field is to make things better and more efficient, to solve problems faster using automation. That applies to our own jobs as well, so developers really don't like inefficient processes in their own workflows. Yet if we don't measure this it's hard to know where to focus on improving.
2. Teams want to be more productive and ship more valuable software. We want to add value to customers using our skills at engineering, so if we can do that more quickly that's great. If one team ships 2x as much as another team that could reflect some superior process the former team has that could be more widely adopted.
The problem only comes when you misuse these metrics. Thus it's the "becomes a target" part of Goodhart's law that is bad, not the act of measuring and trying to improve. As soon as we say "lines of code should go up 10%" or "your lines of code was lowest on the team so you're fired" we've created a problem, but that doesn't mean in general it's useless to measure lines of code. For example if we notice aggregate LOC across the company is trending down that could be a worrying sign that we've put in place some processes to slow down development that bear looking into.
Srikanth, you've raised a crucial point. The temptation to game the system exists in various domains, and the consequences can be significant. As we explore the complexities of measuring productivity, your insight underscores the need for thoughtful considerations to avoid unintended outcomes. It's a delicate balance between incentivizing performance and preserving the integrity of the outcomes. Looking forward to delving deeper into these dynamics in Part 2.
This is spot on, great job of balancing unequivocal criticism of the McK "article" (which is really a thinly veiled sales pitch), without getting petty.
Robert (Bob) Lewis has been writing about the perils of metrics for many years. Originally in InfoWorld and then his own online outlet. Here's a search of his content for "metrics" that yields dozens of articles: https://issurvivor.com/?s=metrics