Posts Tagged ‘measurement’

Anjali’s blog post on ‘Measurement versus engagement‘ made me think.

We’ve spent a great deal of effort over the last two years smashing painstakingly integrating Agile thinking into the strategy and design practices, but at the other end – the delivery end of the process – Agile is too often neglected soon after launch.

This shouldn’t be surprising.

It’s difficult, if not impossible, to keep all that innovationism going for extended periods of time. Often, the innovation team within a client organisation moves off onto new projects and hands the service to less risk-friendly teams, where people have KPIs set by line-managers who don’t know about very much about the project and almost certainly have no idea of probably much less understanding of the Customer Value.

Sometimes the fervent adoption of Agile methodologies that happened during the Vision and Service Definition phases turns out to be superficial and temporary. Who doesn’t like Agile when there’s tons of new code and excitement being demonstrated on a daily basis, and when you can get things done with less pain and far more rapidly than you ever did before? It’s definitely more difficult infinitely more challenging to adopt Agile as a culture, over the long term. “Iterative” and “emergent” are best intention ideas that make total sense during the Vision phase – but are VERY difficult to do on an ongoing basis, especially if the culture at large within the client business doesn’t get it, which is a big ask let’s face it!

But occasionally, even on innovation-geared projects, it’s the metrics what kills it.

Typically, there may have been a conversation early on in the project where you were asked to make some guesses about traffic and registered users. You may have been reassured that these would not be treated as promises, and you probably made the point that it’s very risky to make those kind of predictions with an innovative and emergent service like yours, I mean – how could you possibly predict these things, and they’re not the right things to measure anyway..? Whatever. A metrics time-bomb has started ticking.

Agile gets rid of fixed scope/fixed budget which is great – but that means nothing if you replace them with some dumb fixed metrics. So, I’m arguing here not for less measurement or no measurement, but for an Agile approach to measurement – which, obviously, I absolutely understand the need for.

Agile measurement should focus on simplicity and Customer Value. It should make it easier to measure the things that matter to make the service better for its users – to improve the value exchange. Instead of asking what success looks like, ask what value looks like.

It follows then that Agile measurement should measure fewer things. There is a risk – especially with all the new tools and dashboards and diagnostics – of measuring far too much. Measurement is in danger of becoming an end in itself. A smaller set of metrics and diagnostics – “just enough”, in exactly the way as Agile methodologies treat other types of documentation – should help. Even trying to find the “one key metric” that rules them all – and is tied to the economics investment and supported by executive management – is suggested in this awesome paper I found online and have plundered ruthlessly for this blog post. That makes so much sense.

You should also try and measure “outcome” rather than “output”. It’s not about big numbers, it’s about having the biggest impact on Customer Value. A small community of incredibly engaged, high-value customers is much more valuable than 1 million people who registered to win a car. Obviously, you’ll have first needed to have defined upfront what Customer Value actually is – for you, your project and the business sponsoring it. If you’ve been working Agile from the outset, you should have a pretty good idea of your customer/end-user and their needs, and the value exchange that’ll get them returning hopelessly addicted.

Maintaining a living, ongoing dialogue with users will certainly help – and is essential where the one key metric is for example a softie, like brand perception. But you should generally focus on a small set of engagement health indicators: like the number of visits per user per month, dwell time,  participation, recommendations to friends, positive buzz across other social networks leading to sustained referrals. I’ll also add receiving offers of help from partners and people wanting to be involved, and good ideas from users – and the holy grail: people start using the site for purposes never envisaged. Remember, EBay realised there was an online market for cars when they found customers selling grown-up cars in the toy section.

Lastly, you should be able to review targets, metrics and diagnostics to optimise your ability to understand Customer Value as the service evolves. Within an emergent, iterative innovation-geared project, you may not even understand the true value and how to achieve it until much later.

What does anyone else think?

6 Comments