Are you watching the wrong metrics for agile?

As projects shift to agile methodologies, many have not shifted their metrics

Are you watching the wrong metrics for agile? A project methodology sets a course for project success, with metrics that must be gathered and tracked. When a project chooses a new methodology, it has a new course – and new metrics.

Agile methodologies have become the standard approach for projects that face today’s evolving business needs, according to an online survey of development and IT professionals. Agile methodologies have a dynamic adaptability that is often critical for projects driving digital transformation.

But many teams have tried to adopt agile methodologies without adopting agile project metrics to keep them on course for success. Grant Thornton Advisory Senior Manager Scott Dalessio said “Often, project teams are tracking legacy metrics, and they’re collecting the wrong data to measure an agile project’s performance against desired outcomes.” When projects track the same metrics they used with a traditional waterfall methodology, they not only measure the wrong results, they set the wrong targets.

Why do teams wander off course? It can be difficult to change legacy reporting requirements – and stakeholder expectations. Stakeholders who support an agile methodology might still expect to compare metrics between agile projects and past waterfall projects.

Often, the metrics in an organization’s legacy reporting requirements have become the basis of understanding for people across the organization. People are uncomfortable with changing the metrics because they will need to learn a new basis of understanding. But many legacy metrics are based on waterfall or predictive planning approaches and do not accurately measure an agile project’s performance.

How can I tell if our metrics are wrong? With a wide range of project metrics for different work and environments, what metrics indicate that a project might not be following an agile methodology?

The project management “iron triangle” says that projects must balance three factors to achieve success: scope, schedule, and cost. When one of the factors changes, a project must adjust one or both of the others.

Waterfall projects begin with a fixed scope, in the form of detailed requirements. Waterfall project metrics (such as “percentage complete”) often report on the progress toward that scope over the course of the entire project, to show if or when the project needs to adjust its schedule or cost.

Agile projects consist of multiple two-to-four-week “sprint” iterations, where the cost and schedule are fixed for the duration of the project, instead of the scope. And an agile methodology takes a different approach to scope, measuring the team’s progress against a list of high-level business needs rather than detailed requirements. An agile project manager's question is not “How much of the blueprint have we built?” but rather “How many needs have we met?”

Grant Thornton Advisory Managing Director Wei Tang said “You need to measure against what you can produce in a two-to-four-week sprint.” Tang also warned that some teams apply agile labels to waterfall schedules. “A true sprint should not be longer than four weeks. And you should not take two, three, four sprints to achieve a deliverable. Each sprint should achieve a tangible benefit.”

Dalessio said that completion estimates, Gantt charts at the task level, and other metrics that illustrate long-term progress toward detailed requirements can be indicators that the team is tracking the wrong metrics. Dalessio added that earned value management (EVM) using traditional measurement can also be misleading in the agile paradigm. “In order to do EVM, you need a stable set of requirements to measure against – which agile does not provide. EVM can be done with agile, but you need to look at the scope differently.”

How can I tell if our metrics are right? To evaluate an agile project’s effectiveness, teams typically start with the amount of work they can deliver. “It’s a question of ‘velocity’ – when you look at the amount of work a team can deliver within a time box, you are measuring productivity or capacity in each sprint,” Tang said. This is an important metric, because agile projects don't estimate – they derive time to complete.

The speed and effectiveness of a team’s work can also be captured in metrics like “cycle time” (time to completion) or “product flow.” Dalessio explained “For ‘product flow,’ we want to understand: How effective is a team at taking something from an idea through to deployment so that it adds value to customers?”

While velocity is important, value is the ultimate measure for meeting business needs, and every project activity should be attached to a business value. “It’s a question of: How effectively are we meeting business needs?” Dalessio said.

How to stay on course One of the chief benefits of an agile methodology is that teams can make key adjustments after each sprint, based on metrics that report the velocity and value of the work from the sprint.

This continuous improvement is key – if a team does not perform effectively and does not review its metrics and take action, it will never improve and get back on course.

An agile project’s work can be tracked with different metrics for different types of work, environments and audiences. The annual CollabNet VersionOne State of Agile report found that “Business value, on-time delivery of projects and customer/user satisfaction have remained the top three measures” for agile project success over the past few years, reinforcing the agile focus on velocity and value rather than granular requirements.

The key to successfully using an agile methodology is to start with the right understanding and mindset. Once a project’s team and stakeholders understand how the methodology is designed to deliver value and dynamically adapt, they can better understand the best metrics and measures to evaluate the project at each stage.

Wei Tang
Managing Director, Advisory
T +1 703 562 6597

Scott Dalessio
Senior Manager, Advisory
T +1 703 637 2933