Estimating Defects Leads Teams Down a Rabbit Hole

Over the years, I’ve witnessed various approaches to handling software defects, many of which unintentionally led teams down a rabbit hole of wasted time and effort. These missteps often included excessive planning and estimation for defects, misguided defects prioritization, unclear team velocity charts, complicated burndown charts, and a harmful obsession with aged defects. These practices can eventually divert an organization’s attention away from its primary goal of delivering content. In this article, I will delve into a specific aspect of defect management: the art of (not) estimating defects.

The root of these issues often lies in our intuitive but misguided approach to defects. We all aspire to maintain proper software quality, though the definition of “proper quality” can be subjective and organization-specific. Proper quality might require having no release-blocking defects, limiting major defects to a certain threshold (e.g., less than X), or striving for other quality metrics.

Once we establish these quality guidelines, the primary goal should be to enable our software teams to continue delivering content while upholding the defined quality standards. Ideally, we should aim to enhance content velocity while maintaining the desired quality.

The Alternative

Now, let’s challenge the conventional wisdom that demands effort estimation for defects. Why do we feel compelled to estimate defect efforts? Typically, the answer is to understand the work invested in defect resolution. But why do we need this metric? Often, it’s to determine how many defects to include in a sprint, apparently to improve product quality.

Yet, I propose a different perspective: improving product quality doesn’t depend on fixing a specific number of defects. Instead, it involves managing the defect trend so that it aligns with the defined quality standards. To achieve this, we don’t need to estimate defects; we just need to balance the content we include in each sprint with our quality goals.

Balancing content with quality works as follows: If we need to reduce the defect trend, we pull in less content than our previous average velocity. This approach allocates more time for defect resolution. After the sprint, we review the defect trend. If it’s still unsatisfactory, we reduce the committed content further in the next sprint. We repeat this cycle until the defect trend aligns with our quality objectives. Once satisfied, we can gradually increase the committed content, closely monitoring its impact on the defect trend. This way, we can achieve the desired quality without the overhead of estimating defects.

Defect Estimation Drawbacks

Now, suppose some still insist on estimating defects. Why? To gain visibility into the effort spent on defect fixing. However, estimating defects presents several drawbacks:

• Understanding Effort: Often, 80% of defect effort involves diagnosing the root cause, making upfront estimation challenging. Some organizations turn to setting default values, which may not be optimal and may not reflect reality.
• Double Counting Efforts: Many defects are tied to stories already delivered and accounted for in effort estimation. Assigning additional effort estimation for defects in completed stories can distort the accuracy of velocity metrics.
• Defect Discovery Timing: When a defect is found while developing the story, most people will say we should not estimate. However, determining whether to estimate defects in delivered stories becomes arbitrary. Does a defect found an hour after the story was closed deserve estimation? What if the defect is found a day, or a week later? Determining to estimate defects based on how far they were discovered after the story closure lacks consistency.
• Estimation Based on Defect States: Defects can lead to various outcomes, such as fixed, duplicated, or unreproducible. Estimating all defect states can be misleading. Removing estimation from certain states of defects complicates our workflow.
• Reduced Visibility: consider the aspect of visibility — a common argument for estimating defects. Paradoxically, estimating defects can reduce rather than enhance visibility. Take the burndown chart, for instance: When defects are introduced mid-sprint, it becomes challenging to detect whether this shift is a result of scope changes or defect inclusion. Furthermore, when we dedicate time to defect resolution, the distinction between content delivery and quality issue resolution blurs.
  Similarly, inspecting the velocity chart is more complex as it becomes harder to understand the ratio of content delivery to defect resolution efforts. In response, teams may end up using external tools and dashboards to gain visibility back, departing from the simplicity of managing everything within a single place — the Scrum board.

Considering these drawbacks, it’s essential to consider the benefits of estimating defects against the drawbacks, including energy wastage, reduced visibility, and cumbersome processes.

Hidden Assumptions

Now, let’s delve deeper into the “why” game. We mentioned we want to gain visibility into the effort spent on defect fixing. But why? So we can make decisions to reduce defect resolution time. But why? So we can allocate more time to content delivery. The question is — why do we assume that reducing defect resolution time is the primary driver of increased content delivery?

A more effective approach to increase content delivery might be to just measure content delivery, closely monitor content delivery, and conduct retrospectives on content delivery results each sprint. This approach avoids the hidden assumption that defects are the primary impediment to our quest for improved content delivery. When we factor in the time wasted on defect estimation, it becomes evident that it’s a lose-lose proposition, where we consciously complicate our processes for the chance that it might be the main drive for boosting content delivery. This approach doesn’t yield a promising return on investment.

Discovering Bugs Effort Without a Formal Estimation

Addressing bugs often involves a focused effort due to their typically small and well-defined nature. Team members tend to work on bugs from start to finish without interruption, as each bug represents a specific issue to be resolved. Consequently, the cycle time for addressing bugs closely aligns with the actual time spent working on them. Leveraging a cycle time chart provides valuable insights into the cumulative time invested in bug resolution. By filtering the cycle time chart to display only bug-related activities, you can observe the number of bugs addressed and the average time dedicated to resolving each one. This approach offers a practical means of understanding the effort invested in handling bugs, without the need to manually set effort estimation for them.

Conclusion

In conclusion, challenging the status quo of estimating defects is not about dismissing the importance of software quality but rather finding more efficient and effective ways to achieve it. Balancing content with quality, backed by data-driven decisions, can often be a more practical and agile approach than investing time in estimating defects with uncertain outcomes.