Recruiters and sales people also game the system. If recruiter is measured by lead time and closure of positions, we end up hiring the below-average people to fill in the slots. And talk about system dynamics from there.
This is great! Measuring output is an artifact from the pre-digital industrial age, when output was a decent predictor of outcome. Operational risk was paramount. (If we can produce a black Model T efficiently enough, the middle class will buy.)
Entire companies are structured around optimizing efficiency of output. This makes measuring efficiency of outcomes difficult. Engineering teams may very efficiently build features customers don’t care about. This means you need other disciples involved. Company structure makes this difficult.
"If you consider why a startup moves so fast and executes so well, it’s because they have to do so out of necessity, even if they do not measure this." - Given that 90% of startups fail (couldn't get specific numbers for software startups), I am not sure that most startups actually end up delivering positive customer impacts even though they might deliver a customer facing thing per week.
At the end of the day, Goodhart's law will always prevail.
I would like to agree with others that pointed out that measuring and rewarding of people work can lead to undesirable results in all fields not just in software engineering. Book "The tyranny of metrics" by Jerry Z. Muller has great examples (I like one where surgeons wouldn't operate patient with even small risk so they won't get bad grades).
Most often when people try to measure dev productivity, they use other fields and industries for comparison like engineering, sales, and support. I think this is what leads to the over arching confusion. I would love to see someone do a study on how to measure productivity in, say, a research and development field like medical developments, or even a field that is more artistic like writing or painting. In my opinion, those fields are much closer in nature to software development than the ones people typically use. It would be interesting to see, say, how a company measures the "performance" of a mystery novel writer or someone who writes screen plays, or how to encourage "better performance" from a research scientist who is trying to develop a new vaccine or a cure for some currently incurable illness. If one of those fields has ways of measuring their employees performance, then perhaps we would have a reasonable baseline of comparison for our industry.
Overall, great article by the way. I'm looking forward to reading part 2!
Completely agree, or as a friend would have said: "Measure business, not busyness".
I have worked with (outsourced) IT support who were measured on number of tickets closed and that was useless.
The incentive to close as many tickets as possible often resulted in no resolution or a bad one. Or even a closed ticket with a message telling me to open another ticket on another department.
Thanks for sharing your point of view. Very insightful.
There are a few moments in this article that are confusing me though:
1. When talking about how HR metrics fit in the proposed mental model, you mention in the text that “number of heads filled” is categorized as impact but list it in the “outcome” box in the illustration. It feels more like outcome than impact so I am assuming there is an error in the text. Or am I missing something?
2. When you talk about the Space framework, you mention that the people behind it put a warning on “effort and outcome” metrics. Did you mean “effort and output” there? Some outcome metrics could need a warning as well depending on how people could game them though.
3. When you propose to measure the “please the customer once per team, per week”, you imply that this is an “output” metric, but it feels more to me like an “outcome” one. Can you enlighten me on why you’d consider this an output metric?
Nice read. Thanks for outlining the issues with measurements in the software industry, it's a big problem. In any industry.
With highly productive and skilled engineers we can very productively deliver products nobody buys or uses.
The productivity problem is so much engrained in our Taylorism style of management. McKinsey et al still fit into this style of management. We as management are incapable of learning to solve problems, therefore we hire consultants to tell us what to do.
If you want to measure (which is a huge investment for many), then measuring (the right) outcomes in combination with team behavior (e.g. collaboration). Put incentives only on (team) behaviors. Never on outcomes. Lean OKRs is a good framework (but I'm biased).
Separate note: software engineers are often abstracted from outcomes and even more so from impact. Other disciplines are generally making decisions about what to build and how it should function.
Many engineers are simply asked to build the thing. I think at that point the outcomes ICs can be responsible is the output of their work: the thing they were asked to build. Whether or not is has actual business impact is generally someone else's decision to hold to account. From an impact perspective, the developer's customer is the requestor of the feature - the PM or PO or whathaveyou.
I'm not saying this is the way to do it, just kind if taking the temperature of this kind of system for making software. The engineering team in this type of system is NOT responsible for many business outcomes because they're not asked to deliver business outcomes. They're asked to deliver features.
Sadly, they're even less likely to be asked to create impact.
There might be many issues with the Mck article but they try to 'prescribe' a solution which can be operationalized. Unless the software / product industry comes up with better prescription CEOs will continue to listen to the consulting giants. Everyone understands it's important to measure outcomes, but how many orgs are able to do this at scale (it doesn't make sense to count start ups only).
I absolutely agree that measuring outcomes and connecting them to impacts is the way to go in the software business (Jeff Patton has been shouting that from the rooftops for years now). I also noticed that a lot of companies don't even know where to start, and articles like McKinsey's don't make it any easier. One thing I noticed about your article is the conspicuous absence of product management, which is directly focused on this outcome and impact correlation thing, including making sure that product strategy supports business strategy (impacts). I think that may be as much of a blind spot for many engineering leaders as the importance of focusing on teams rather than individuals in product development is for many non-technical senior executives (both are business leaders, and the very distinction between engineering and "the business" is at the root of the problem). I am not saying that engineers can't be great product managers, only that engineering and product management are two very big things that at any but the smallest scale call for a team of specialists working closely together, as everything else in product development.
This is spot on, great job of balancing unequivocal criticism of the McK "article" (which is really a thinly veiled sales pitch), without getting petty.
Robert (Bob) Lewis has been writing about the perils of metrics for many years. Originally in InfoWorld and then his own online outlet. Here's a search of his content for "metrics" that yields dozens of articles: https://issurvivor.com/?s=metrics
As a CEO with developer backround, running a 50 people SaaS software company, I can highly relate to this article. Especially to the tradeoff of measuring different metrics. Comparing it to sales and recruitment, I find those two functions (sales and recruitment) very similar in process and type of outcome that need to be achieved. The main difference to engineering, based on my experience performing all of those three functions myself too, is that sales and recruitment are much more dependent on external factors than engineering roles. I feel developers (or business analysts etc) are much more in control of achieving a result by their own means than relying on external factors like sales and recruitment teams do. So, if you do the right thing as an engineer, the result will follow, if you do the right thing in sales, a customer could
still decline your offer based on uncontrollable factors.
The pressure on the Eng teams to quantify everything in metric is outdated, while at the same "necessary" for non-technical C-management who only go by the $$$ however as mentioned this remove the team towards a goal to everyone for himself changing the whole team dynamics.
In some narrow context, measuring productivity works. E.g. save AWS cost and increase performance measured by benchmarks.
The primary advantage of measuring this ROI is better prioritisation.
In a broader context, having any crude metric that will work is very hard. Though having readily available data helps, you have to mesh it with subjective assessments.
The problems of our industry in bigger org, the incentives go against productivity. Having a smaller team of 4 full-time instead of 12 doing the same thing will rarely give any upside other than intrinsic satisfaction. Managers get rewarded for building empires, and engineers for building more complex systems. Over-engineering over-hiring gets awarded for many years, and then execs hope one trick will fix it.
Only a better culture of organisation has managed to solve this challenge so far.
Measuring developer productivity? A response to McKinsey
Recruiters and sales people also game the system. If recruiter is measured by lead time and closure of positions, we end up hiring the below-average people to fill in the slots. And talk about system dynamics from there.
This is great! Measuring output is an artifact from the pre-digital industrial age, when output was a decent predictor of outcome. Operational risk was paramount. (If we can produce a black Model T efficiently enough, the middle class will buy.)
Entire companies are structured around optimizing efficiency of output. This makes measuring efficiency of outcomes difficult. Engineering teams may very efficiently build features customers don’t care about. This means you need other disciples involved. Company structure makes this difficult.
"If you consider why a startup moves so fast and executes so well, it’s because they have to do so out of necessity, even if they do not measure this." - Given that 90% of startups fail (couldn't get specific numbers for software startups), I am not sure that most startups actually end up delivering positive customer impacts even though they might deliver a customer facing thing per week.
At the end of the day, Goodhart's law will always prevail.
I would like to agree with others that pointed out that measuring and rewarding of people work can lead to undesirable results in all fields not just in software engineering. Book "The tyranny of metrics" by Jerry Z. Muller has great examples (I like one where surgeons wouldn't operate patient with even small risk so they won't get bad grades).
Most often when people try to measure dev productivity, they use other fields and industries for comparison like engineering, sales, and support. I think this is what leads to the over arching confusion. I would love to see someone do a study on how to measure productivity in, say, a research and development field like medical developments, or even a field that is more artistic like writing or painting. In my opinion, those fields are much closer in nature to software development than the ones people typically use. It would be interesting to see, say, how a company measures the "performance" of a mystery novel writer or someone who writes screen plays, or how to encourage "better performance" from a research scientist who is trying to develop a new vaccine or a cure for some currently incurable illness. If one of those fields has ways of measuring their employees performance, then perhaps we would have a reasonable baseline of comparison for our industry.
Overall, great article by the way. I'm looking forward to reading part 2!
@Kent Beck
If developers productivity can't be measured then look at the implications:
- Promotion , PIP low performer's and Layoffs.
are done arbitrarily.
- organizations won't accept this publicly because they are scared of lawsuits.
Completely agree, or as a friend would have said: "Measure business, not busyness".
I have worked with (outsourced) IT support who were measured on number of tickets closed and that was useless.
The incentive to close as many tickets as possible often resulted in no resolution or a bad one. Or even a closed ticket with a message telling me to open another ticket on another department.
Thanks for sharing your point of view. Very insightful.
There are a few moments in this article that are confusing me though:
1. When talking about how HR metrics fit in the proposed mental model, you mention in the text that “number of heads filled” is categorized as impact but list it in the “outcome” box in the illustration. It feels more like outcome than impact so I am assuming there is an error in the text. Or am I missing something?
2. When you talk about the Space framework, you mention that the people behind it put a warning on “effort and outcome” metrics. Did you mean “effort and output” there? Some outcome metrics could need a warning as well depending on how people could game them though.
3. When you propose to measure the “please the customer once per team, per week”, you imply that this is an “output” metric, but it feels more to me like an “outcome” one. Can you enlighten me on why you’d consider this an output metric?
Thanks again!
Nice read. Thanks for outlining the issues with measurements in the software industry, it's a big problem. In any industry.
With highly productive and skilled engineers we can very productively deliver products nobody buys or uses.
The productivity problem is so much engrained in our Taylorism style of management. McKinsey et al still fit into this style of management. We as management are incapable of learning to solve problems, therefore we hire consultants to tell us what to do.
If you want to measure (which is a huge investment for many), then measuring (the right) outcomes in combination with team behavior (e.g. collaboration). Put incentives only on (team) behaviors. Never on outcomes. Lean OKRs is a good framework (but I'm biased).
Separate note: software engineers are often abstracted from outcomes and even more so from impact. Other disciplines are generally making decisions about what to build and how it should function.
Many engineers are simply asked to build the thing. I think at that point the outcomes ICs can be responsible is the output of their work: the thing they were asked to build. Whether or not is has actual business impact is generally someone else's decision to hold to account. From an impact perspective, the developer's customer is the requestor of the feature - the PM or PO or whathaveyou.
I'm not saying this is the way to do it, just kind if taking the temperature of this kind of system for making software. The engineering team in this type of system is NOT responsible for many business outcomes because they're not asked to deliver business outcomes. They're asked to deliver features.
Sadly, they're even less likely to be asked to create impact.
There might be many issues with the Mck article but they try to 'prescribe' a solution which can be operationalized. Unless the software / product industry comes up with better prescription CEOs will continue to listen to the consulting giants. Everyone understands it's important to measure outcomes, but how many orgs are able to do this at scale (it doesn't make sense to count start ups only).
I absolutely agree that measuring outcomes and connecting them to impacts is the way to go in the software business (Jeff Patton has been shouting that from the rooftops for years now). I also noticed that a lot of companies don't even know where to start, and articles like McKinsey's don't make it any easier. One thing I noticed about your article is the conspicuous absence of product management, which is directly focused on this outcome and impact correlation thing, including making sure that product strategy supports business strategy (impacts). I think that may be as much of a blind spot for many engineering leaders as the importance of focusing on teams rather than individuals in product development is for many non-technical senior executives (both are business leaders, and the very distinction between engineering and "the business" is at the root of the problem). I am not saying that engineers can't be great product managers, only that engineering and product management are two very big things that at any but the smallest scale call for a team of specialists working closely together, as everything else in product development.
This is spot on, great job of balancing unequivocal criticism of the McK "article" (which is really a thinly veiled sales pitch), without getting petty.
Robert (Bob) Lewis has been writing about the perils of metrics for many years. Originally in InfoWorld and then his own online outlet. Here's a search of his content for "metrics" that yields dozens of articles: https://issurvivor.com/?s=metrics
As a CEO with developer backround, running a 50 people SaaS software company, I can highly relate to this article. Especially to the tradeoff of measuring different metrics. Comparing it to sales and recruitment, I find those two functions (sales and recruitment) very similar in process and type of outcome that need to be achieved. The main difference to engineering, based on my experience performing all of those three functions myself too, is that sales and recruitment are much more dependent on external factors than engineering roles. I feel developers (or business analysts etc) are much more in control of achieving a result by their own means than relying on external factors like sales and recruitment teams do. So, if you do the right thing as an engineer, the result will follow, if you do the right thing in sales, a customer could
still decline your offer based on uncontrollable factors.
Great article on Effort/Output vs Outcome/Impact.
The pressure on the Eng teams to quantify everything in metric is outdated, while at the same "necessary" for non-technical C-management who only go by the $$$ however as mentioned this remove the team towards a goal to everyone for himself changing the whole team dynamics.
An interesting article of [Itamar Gilad](https://itamargilad.com/velocity-vs-impact/) approach the outcome/impact metric and help put a value on it for those directors.
In some narrow context, measuring productivity works. E.g. save AWS cost and increase performance measured by benchmarks.
The primary advantage of measuring this ROI is better prioritisation.
In a broader context, having any crude metric that will work is very hard. Though having readily available data helps, you have to mesh it with subjective assessments.
The problems of our industry in bigger org, the incentives go against productivity. Having a smaller team of 4 full-time instead of 12 doing the same thing will rarely give any upside other than intrinsic satisfaction. Managers get rewarded for building empires, and engineers for building more complex systems. Over-engineering over-hiring gets awarded for many years, and then execs hope one trick will fix it.
Only a better culture of organisation has managed to solve this challenge so far.