Values first, numbers second! We need to steer our projects, and to do that we need data.

Here is an excerpt from Gavin's book which talks about measurement:

Measure things, don't just use guesswork and "intuition". If you say to a project manager "I spent 4 hours improving performance" and you don't give any figures, then you might as well have been playing CounterStrike! … Get into the habit of providing figures whenever possible.

This helps us to make rational, informed decisions about our projects. It can, however, be extremely difficult to pick what to measure and how much stock to place in those measurements. In this article, we look at things to bear in mind when defining metrics for software projects.

We are looking at project-level metrics here. We will consider organisation-level metrics in a future post.

Bad Metrics - The Danger Zone!

Ever been in this situation?

If you've ever heard something like this, where people are trying to game metrics, then you might be in the danger zone!

(source: quickmeme.com)

Uh oh! Looks like a case of bad metrics!

In this example, the budget for the project is tied to the time spent, but it is not being honestly reported. There are many reasons why this may be, but management cannot make informed decisions if the metrics are poorly chosen or inaccurately reported... How has this happened and what can we do to prevent this kind of misuse and gaming of metrics?

Avoiding potential metrics pitfalls

Let's take a look at three potential pitfalls of poorly applied metrics and what we can do to avoid these dangers.

Danger 1: misunderstanding what a metric is

Metrics may been seen to be "the thing itself" rather than a measure of something. A thermometer in water does not show the temperature of all the water, rather it shows the temperature at a single point.

An example misunderstanding is common in agile development. "Oh no, only 20 points got delivered this week, we normally do 40!"... This does not necessarily mean catastrophe; the estimates could have been off, or someone could have been sick. Points are very useful, but are not "real things", they're just used for sizing and planning.

Preventing misunderstandings - understand that the measurement is not the thing!

Metrics are helpful only so long as it is entirely grasped that a metric is, at best, a view of an aspect of a thing and is NEVER the thing in and of itself. For example, yes, it's worth paying attention to story points (we pay VERY close attention to them!), but they are not bricks, they are not cement, they are very abstract and we see them for what they are.

Measurements, metrics and KPIs

Let’s have a look at how we can sensibly approach metrics and define our terms.

I want to draw a distinction between 'metric' and 'measurement'. A measurement is a value over time, and a metric is a measurement with an attached set of decision-making constraints. That is, we can measure stuff, but not all of what we are measuring should be used to guide the project. So, in this pyramid, not all measurements become metrics.

To help us work out what measurements to use as metrics, one approach is to can measure and track absolutely everything we can, going back as far as possible, then explore the data. See what looks interesting, find unexpected correlations, then try to derive metrics from that.

The most critical metrics, we can use as KPIs (Key Performance Indicators). These are the things that tell us whether our project is succeeding or failing. We have to pick these incredibly carefully, as we will see later, but when wisely chosen, they can give us a lot of insight into what is going on.

Approaching it this way means that we don’t get overwhelmed, and we keep in sight what our project-guiding metrics are without getting bogged down in the ones we just record in case they become useful later.

Danger 2: poorly chosen metrics

As we saw in the pyramid, not all measurements may actually be useful. It is important to continuously evaluate - is this metric telling us anything useful? The caution here is not to assume that just because something is recorded, it has value. With that said, something might prove to have value later, so it’s worth recording, just don’t sell the farm based on what a single measurement is telling you.

Choosing good metrics

The metrics should not be the focus of the project; rather they are a way of measuring what is going on and having telemetry into the project. Far more important are the values that drive us - what kind of project are we going to be? What matters to us? What matters to this project?

For example, on one recent project, Radify worked with the client to define three values that were most important to them, We then implemented metrics against these values. Here are those values:

1. Predictability
2. Continuous feedback
3. Quality

As stated before, we had been measuring everything we could against the application from the outset, and it was therefore quite easy to pick out representative measurements and turn them into metrics. We're not going to go into huge detail here but here are some of the things we picked:

Metrics for Predictability

1. Track progress against estimates and report back on every sprint
2. Expected vs actual velocity
3. Scope creep (changes to sprint after start)

Metrics for Continuous feedback

1. Number and timing of deploys
2. ApDex, error rate, uptime, CPU, memory and disk utilisation
3. Application-specific metrics (e.g. number of orders placed today, number of abandoned orders)
4. Continuous integration build results

Metrics for Quality

1. Number of bugs in production
2. Code coverage

We then create project dashboards using DataDog so we can see at-a-glance what our metrics are telling us. We iterate on these metrics in every sprint retrospective and are always fine-tuning what goes on the dashboard. Hopefully, you can see that we've picked metrics based on what is actually important to the project based on what is important to the client.

Danger 3: badly applied metrics resulting in toxic culture

An organisation or project that focuses on metrics with too narrow a gaze runs the risk of poisoning its own culture.

Where metrics are used individually and narrowly or as a stick to beat people with, this results in people “gaming” the metrics to appear more productive. Humans being humans, we tend to reverse engineer management’s perceived desired outcomes.

Worse still, toxic metrics can cause serious disenchantment. Toxic KPIs can be particularly harmful as they and can encourage people to focus narrowly on their KPIs to the exclusion of overall cohesion and cooperation (see this article, for example) . This is why linking metrics to individual reparation and performance reviews is dangerous - many people will pull up the drawbridge and focus on only what is tied to their pay reviews, not what is best for the project as a whole. This might mean that they spend less time helping colleagues or thinking outside the box.

One of Radify’s main driving principles is "no silos", because we want everyone to understand the project as a whole so we really want to avoid toxic metrics.

Preventing toxic culture - think team!

Metrics must NEVER be used as a stick to beat people with.

Singling people out creates silos as each person looks to their own interests rather than thinking about the project or organisation holistically. Metrics should never be individualized; the project team as a whole improves together or falls behind together. This means that people will feel free to report non-technical metrics honestly and openly because they don’t feel like they will be persecuted for the results!

Characteristics of good metrics

We have seen that bad metrics are poorly understood, badly chosen and toxically applied. Conversely, good metrics have three characteristics:

1. Well understood: the team understands what the metric means and what it is and isn't telling them. The team understands the relative importance of KPIs and other metrics.
2. Well chosen: the metric reflects the values of the project.
3. Non-toxic: the metric is applied positively to guide the project, not as a stick to beat people with.

Benefits of good metrics

The whole company and everything that happens within it can be decomposed into a set of systems. Complex systems are easy to create and hard to understand. The more things we measure and the longer the timeline, the better we understand the system. Eventually, our understanding of the system(s) will be almost intuitive, as the impact of changes will be immediately observable (for varying values of 'immediate', depending on the system). This will allow us to steer our project with greater confidence.

Well-designed KPIs provide quick insight into trends and summary information, while supporting drill-down into more detailed metrics. This allows an organization to see where it's doing well and where it requires improvements and/or course adjustments.
- Jonathan D. Becher

When to use your metrics

There are two main points at which we use our metrics in a project:

1. Reactively: if something goes bad, especially in production, we get notified right away! See Four Principles Of DevOps and The Bot Who Cried Wolf for details.
2. Proactively: we look at our dashboard in every sprint retrospective, and take time to interpret what it is telling us. This allows us to steer what we do. For example, if performance is bad, we may allocate some time in the next sprint to this as a non-functional requirement.

Wrapping up

We hope you’ve found our ideas on metrics thought-provoking. When properly chosen, metrics can be extremely valuable in guiding your project. When poorly chosen or applied, they can be toxic to morale and can give false confidence or misleading information.

We’ve got a lot more to say on the subject, so if you want to talk, please get in touch! If your organisation needs help getting measurable results, we can help you!

What do you measure? Is it helpful? Have you ever been a victim of toxic metric culture? Were you able to fix it? Let us know in the comments box below!