Forecast or Fantasy? The Dangerous Allure of Vanity Metrics

Forecasting Fantasies

The crystal ball…
How much easier would delivery planning be if we all had one? While the idea sounds amazing – perhaps even a little tempting – the reality is, we don’t.

Instead, plans are often built on more traditional methods: guesstimates and vanity metrics. While some long-standing practices in the delivery world still hold value, others have reached a point where they’re no longer fit for purpose. In my view, delivery plans based on guesswork and cosmetic metrics need to step aside and make room for something more reliable.

It’s time to move away from fantasy forecasting.

Vanity metrics

Before we explore a better alternative, let’s take a moment to understand what I mean by vanity metrics.

Examples include: number of likes, number of followers, lines of code, page views, app downloads – to name just a few.

While vanity metrics may look impressive, they offer little real insight or value for decision-making. They create the illusion of progress but don’t help you understand what’s actually working – or guide you toward improving your process or outcomes.

You might notice I didn’t include story points or T-shirt sizing in that list. Some may argue they aren’t vanity metrics. I’d challenge that – depending on how they’re used.

If they’re used for relative estimation or rough prioritisation during high-level roadmapping, they don’t belong in the vanity metric bucket. But in my experience, while they often start out this way – as intended – they don’t tend to stay there.

All too often, story points or sizes are converted into time-based commitments and tracked or reported as productivity measures. Once T-shirt sizing is used to commit to a date, it transitions from being a useful guide, to a vanity metric – aka a devil in disguise.

It’s easy to fall into the trap of thinking a story point – or T-shirt size – can provide a good indication of when a work item (or worse, multiple items) will be delivered. I’ve been there and worn the T-shirt-sizing T-shirt.

These vanity metrics – initially rough guesses – quickly evolve into trusted delivery forecasts tied to large investments or spend. Soon enough, dependency planning builds on what was originally just an estimate. And then – ta-da! – a beautiful yet misleading plan is born.

The Plan

You can see it now. It’s so enticing that it draws you in, as well as everyone around you. It practically glistens.

But while the plan may glisten, the dates behind it are a world away from glitter and gold. The dates plotted are deterministic - “X feature will be delivered on X date.” But forecasts, by their very nature, were never meant to be deterministic.

Take the weather, for example. A forecast shows a probability - an 80% chance of rain, for instance - and it’s updated frequently. Imagine if it weren’t. That could have devastating consequences in some parts of the world. If you’re familiar with Hurricane Sandy, you’ll understand the point.

The same applies to delivery plans. We should forecast the way we forecast the weather - probabilistically, short-term, and frequently.

Still, the plan - full of risk - often becomes the centre of stakeholder conversations. And in those conversations, there’s usually one or two individuals who remain unmoved by its shimmering presence: the outliers.

While everyone else is dazzled by the plan, the outliers are quietly raising their hands, trying to be heard. But let’s be honest - it’s hard to be the voice of reason in a room full of people mesmerised by “the plan.” You can almost picture the little red devil on their shoulders whispering, “Ignore those outliers. They don’t know what they’re talking about. The plan is great!”

But those outliers often have lived through the pain of fantasy-based planning - and have gone off on an Indiana Jones–style adventure in search of something that actually works. Sometimes, the outlier is trying to tell you something important. It’s worth paying attention.

Forecast or Fantasy?

I can honestly say I have yet to see a project based on estimates actually hit its forecasted dates. And chances are, I never will. It’s easy to fall into a false sense of security.

Why don’t estimates work?
Most of us have experienced situations where a best guess is put forward - then either padded out through fear of being wrong, or shaved down because “it can’t possibly take that long.”

Beyond that, there are other common pitfalls:

• We’re individuals, working at different paces and with different levels of experience. What takes one person two days might take another four.
• “That’s a five-minute job.” Even when the person estimating is the one doing the work, things happen - people get sick, priorities shift, bugs emerge.
  What begins mid-flow can be derailed by production issues, urgent requests, or organisational changes.

The only thing we can count on is change - that’s our one true constant.

Eventually, the enchanting, glistening plan begins to fall apart, and people start scratching their heads, wondering what on earth happened.

So why do we keep doing this? Familiarity? Maybe. But whatever the reason – there is a better way.

Enter: Data

Not just any data – past data. Data that reflects your current working reality - not the holidays or a change freeze, but your normal cadence.

With this, we can start answering questions like:

• “When will the next 15 items be completed?”
• “How many items can I complete by [X date]?”

And you don’t need fancy tools (though they help); a simple spreadsheet can do the job.

This approach uses a Monte Carlo simulation (MC).
An MC runs thousands - or even millions - of simulations to converge on a likely outcome, presented as a probability range, depending on the level of risk you’re willing to take.

It might look like this:
“85% chance of completing 30 items by July 2nd or sooner.”

This takes the guesswork out of planning and replaces it with something we can trust.

Caveats and Conditions

For MC forecasts to work effectively, you need a stable system. Some variation is normal, but too much reduces confidence.

Ask yourself:

• Is work entering the system at the same rate it’s being completed?
• Are you managing Work in Progress?
• Are you keeping track of item age?

If yes, you’re in a good place. (System stability is a whole topic on its own, but this gives you a basic understanding.)

Also like the weather, you must reforecast regularly. Change is ongoing.

Shifting from traditional planning (using story points or T-shirt sizing for date-driven commitments) to data-driven forecasting doesn’t just bring confidence - it enables you to control your risk.

It puts the business back in control. And before you know it, that little red devil sitting on people’s shoulders in the room? He’s disappeared.

In a Nutshell:

Vanity metrics - such as story points and T-shirt sizing used for time-based commitments - can be dangerous and add significant risk to your delivery plans. They may feel comforting and produce a glistening plan, but it’s a mirage.

If we want to grow our businesses and make informed decisions, we need confidence in our delivery forecasts. That means reforecasting regularly and embracing change.

If there’s one thing I will take a deterministic stance on, it’s this:

Change is inevitable. We need to plan accordingly.