Smarter ideas worth writing about.

The Proof is in the Pudding: How Agile Teams Can Leverage Evidence Based Management

Sally’s team just completed their Sprint Review where they sat with stakeholders and inspected the increment of work the team had built over the last two weeks. There was a lot of good feedback and minor suggestions for improvement, but overall the stakeholders were very happy and congratulatory over the work Sally’s team did in exceeding planned velocity. Stakeholders agreed it was time to release to production and start seeing customers use this new feature. So after a few weeks of the feature being live, Mark, the VP of Operations, asked in a Sprint Review, “How’s the user response to the new reports being delivered?” This was exactly the question Sally was hoping to avoid, as only about 25% of the user base they were predicting to use the reports had actually started using them.

Unfortunately, Sally’s situation is all too common for teams using Agile/Scrum as a framework. In fact, it is still found that most software projects using agility do not deliver on the value originally intended. While this is a drastic improvement over traditional delivery methods (which have a close to 90% failure rate in some studies), it is nowhere near what talented teams are capable of. It begs the question, why do we still have this problem?

The Problem

Organizational structure destines us for failure. What do I mean by this? In most organizations, we have “stakeholders” that we are comfortable with, that we accept as the voice for the user. While this sounds good in theory (because getting actual user feedback is difficult, let alone actually understanding user needs), it has the danger of being nothing more than an exaggerated game of telephone where there are multiple places of abstraction from user to the team. This leads to many assumptions that are often untested getting baked into the process. Organizations then develop a desired timeline for getting this vision done. Sadly, many organizations still focus more on this delivery (scope, schedule, and budget) than they do on discovery. In this context, I mean the process of continuous discovery, of constantly collecting actual user feedback and letting that be the guide that we measure ourselves on.

The Solutions

As the old proverb goes, the proof is in the pudding. We can’t truly know something is good until we try it. That being said, we can’t even hope to get close to success if we have multiple layers of abstraction between the teams building the product and the users using the product. We have to close that gap to make our products better. Better understanding will help us have a better design, leading to better outcomes. Then we can then take these results and analyze and adapt iteratively. This is often counter to conventional thinking where we assume we know with certainty what we are getting to in the end. That mindset makes Sprint Reviews a formality to get minor tweaks suggested from our stakeholders. But the reality is, if we have a better and deeper understanding of what our user’s needs and goals are, we can often develop far better, quicker, cheaper, more valuable solutions than what we would have thought of on our own. This is what innovation is.

A popular example of this is Henrik Kniberg’s example of MVP:

In this example, Kniberg illustrates that having the end goal as the only thing in mind is not the solution. We can often start to achieve customer satisfaction by meeting one need at a time. Consider Kniberg’s example: With the skateboard, we allow somebody to travel a few blocks quickly. Through the scooter, we help give the user better coordination. By introducing the bike, we start to give the user more speed. With the motorcycle, we save their legs from fatigue. Finally, with the car as end state solution, we allow them to go far distances and still enjoy the wind in their hair. The key here is that there is an iterative level of satisfaction that we are building on, not just iterative delivery


How do we ensure this happens? There are two critical components:

  1. Collect the right data
  2. Measure the right things

There are many tools that are available today for businesses to track how their users are using their applications. It will require minimal investment, but will provide invaluable results in understanding users. This can be used as a catalyst for driving conversations with real users to get their feedback and understand how they are doing their job, so your team can make it better. Data doesn’t just have to be numbers on a spreadsheet, but by seeing your users face-to-face and empathizing with them, it can make a world of difference in discovering better solutions.

Stop putting so much emphasis on measuring delivery metrics (velocity, allocation, etc.). While there is still value in measuring these (for continuous improvement), they are irrelevant to the end result of organizational value. Instead, a better use of measurements is measuring the current state of how your organization delivers value and planning how to improve that. Some examples of these measurements could include:

  1. Employee satisfaction (Engaged employees that know how to maintain, sustain and enhance the software systems and products are one of the most significant assets of an organization)
  2. Customer satisfaction (Sound management, solid software, and creative, fulfilled employees)
  3. Release frequency (The time needed to satisfy the customer with new, competitive products)
  4. Cycle time (The time (including stabilization) to satisfy a key set of customers or to respond to a market opportunity competitively)
  5. Usage index (Determines a product that is burdensome and difficult to use and excess software that must be sustained even though it is rarely used)

Reference: Evidence Based Management

The proof is in the pudding, and the proof of our success is our customer delight. By doing continuous discovery of understanding our users and addressing their needs, we will be able to hypothesize what they want, quickly deliver solutions, and then quickly learn whether it met our hypothesis or not. If it does, excellent! On to the next need! If it does not, we learn what needs to change, and make it happen. If Sally’s team does this, they will be well on their way to delighting their users, and in turn, delighting their stakeholders.


About The Author

Project Services Principal Consultant

Chris is a Principal Consultant with Cardinal Solutions in Cincinnati. In his career, he has served in a wide variety of roles ranging from product management, agile coaching, business analysis, and just about everything in between. Chris' drive is to make organizations and people as effective as they can be, and sees agility as an excellent tool for doing that.