In the previous post, I described Imhotep, our scalable, efficient, and fast open source data analytics platform. Imhotep helps us use metrics at Indeed for fast, iterative experimentation, which drives improvement to our products.
Improving process and coaching people
We use the same tools and techniques to improve development processes, following a measure-question-learn-improve cycle:
- Measure everything we possibly can.
- Learn by asking questions and exploring the data we’ve collected.
- Use our learnings to try to improve.
- Measure continuously to confirm improvement.
Beyond process improvements, this approach can also work for people. Data can help us understand our own work and coach others.
- How much am I getting done?
- How am I engaging with other teams?
- How has my work changed over time?
- What are my blind spots?
Is measuring processes and people a good idea?
You might be skeptical of using this approach for improving process and measuring people. It’s good to be skeptical. To truly benefit from this approach, you must proceed with caution.
Gaming the stats (Goodhart’s Law)
The first caution is Goodhart’s Law, which states that “When a measure becomes a target, it ceases to be a good measure.” For example, your manager might say: “Our measurements show that our team productivity is declining. Let’s set a target to increase features completed by 20% next quarter. If we hit it, big bonuses all around!”
Okay, but your team might then start “gaming the stats” — making changes that improve the metric without improving productivity. Now your measure is meaningless for gauging productivity, and you’ve rewarded your team for counterproductive measures that don’t advance your goals.
The Number Six Principle
The second caution is something I’ve named the Number Six Principle (inspired by a classic TV character and his famous line): Don’t reduce people to a set of numbers.
No one enjoys being judged entirely by numbers. If you tell people you’re measuring them, you run the risk of seriously damaging morale. Many people will assume you’re not considering qualitative performance elements.
It’s how you use them
You can avoid these pitfalls if you’re careful. Consider the example above in which your team is concerned about slipping productivity metrics. If you take a close look at the numbers, understand them in context, and diagnose the situation, you can have a productive dialog about how to improve.
Perhaps your team tackled more complex features, therefore completing fewer. That might be okay, or you might agree as a team that you could have done a better job of simplifying the feature work.
Or maybe you look at a different metric and see that your overall support load went up 50% due to growth in your customer base. You can then decide to live with that balance or try to augment your team’s capacity to handle support while developing new features.
Starting with the measurements, a considered discussion can lead to tangible process improvement. In the next post, I describe a process improvement we validated and measured with Imhotep.
Read the full series of blog posts:
- Imhotep: Scalable, Efficient, and Fast
- Using Metrics to Improve the Development Process (and Coach People)
- Metrics-Driven Process Improvement: A Case Study
- What’s Up, ASF? Using Imhotep to Understand Project Activity
- The Benefits of Hindsight: A Metrics-Driven Approach to Coaching
Using Metrics to Improve the Development Process (and Coach People) cross-posted on Medium.