Why Exact Dates Don’t Fit in a Software Domain

The use of exact dates for delivery (along with fixed content and budget) in the software engineering domain often proves challenging due to the inherent complexity and uncertainty associated with software development. The Cynefin framework, developed by Dave Snowden, provides insights into the nature of different problem domains, categorizing them into the following: Simple (or Obvious), Complicated, Complex, and Chaotic. In the context of software engineering, the development process often falls into the Complex domain.

Why Exact Date Predictions Do Not Work

In the Cynefin framework, the Complex domain is characterized by unpredictability, emergence, and a lack of clear cause-and-effect relationships. Software development is often complex because of the numerous variables, changing requirements, and the dynamic nature of technology. Traditional, linear approaches (like the Waterfall model) assume that the problem is clear and predictable, but in the software domain, this is often not the case.

The Waterfall model, popularized in earlier decades, followed a sequential and linear approach to software development. Gantt charts were commonly used for project planning, detailing tasks and their timelines. However, this approach assumed that all requirements could be defined upfront and that the development process would proceed predictably. In reality, software development often involves evolving requirements and discoveries during the development process and after the development process, rendering the Waterfall model, Gantt charts, and exact date predictions less effective.

Software engineering resides in a Complex domain because it involves numerous unknown unknowns — factors that are not only unpredictable but are also not fully understood or even knowable in advance. The complex nature of software development makes it challenging to provide precise estimates or fixed delivery dates.

Exact Dates Work When Waste Is Introduced

During the Waterfall era, which spanned several decades, there was a victorious belief that software development could be planned and executed in a linear, sequential manner, with fixed requirements and predetermined timelines.

However, this approach faced significant challenges in the inherently unpredictable and complex nature of software development. The Chaos Report, published by the Standish Group, has consistently highlighted the difficulties in achieving project success during this era. In the Chaos Report 2015, it was reported that only 11% of projects were deemed successful, with success defined by meeting scope, time, and cost goals.

In some cases where exact dates were seemingly met, it often involved incorporating extensive buffers in the planning. Parkinson’s Law suggests that work expands to fill the time available for its completion. This meant that even with buffers, teams might not use the additional time efficiently, leading to increased costs and potential inefficiencies.

The practice of setting exact dates in project plans resulted in significant waste. When changes occurred — whether in requirements, technology, or other factors — updating future dates in the plan became a time-consuming and resource-intensive task. The need for constant adjustments highlighted the inflexibility of the Waterfall model in adapting to changing circumstances.

The failures and challenges observed during the Waterfall era played a significant role in the emergence of Agile methodologies where fixing scope, time, and budget is not feasible.

Why Do We Keep Using Exact Dates?

The desire for exact date predictions in project management, including software development, is a natural desire. Human beings often seek certainty and clarity in planning, and having precise timelines can provide a sense of control and confidence. However, this desire for certainty has, in many cases, proven to be a double-edged sword, creating a false sense of confidence and security.

While the desire for exact date predictions is understandable, it’s crucial to recognize the limitations and risks associated with this approach, particularly in the context of software development. Embracing a more flexible and adaptive mindset, as promoted by Agile methodologies, allows teams to respond to change, manage uncertainties, and deliver value in a more sustainable and effective manner. The acknowledgment of uncertainty and a focus on collaboration and continuous improvement are essential for navigating the complexities of software projects successfully.

Agile Approach for Predictions

Agile methodologies, such as Scrum, emphasize adaptability and embrace the inherent uncertainty in software development. Instead of fixed timelines, Agile relies on prioritization and re-planning. By focusing on delivering high-priority features first and adapting to changes as they occur, teams can respond more effectively to evolving requirements and unforeseen challenges.

Agile frameworks leverage metrics like velocity (a measure of the amount of work completed in a sprint) and cycle time (the time taken to complete a specific task). These metrics provide more reliable indicators of progress and can be used for forecasting. However, even these metrics are subject to change as new information emerges, and priorities shift.

Instead of committing to an exact delivery date, a more dynamic and adaptive approach in software development involves expressing timelines in terms of sprints or iterations. This method leverages the team’s historical performance, known as velocity, to provide stakeholders with a reliable estimate of when a specific work item will be completed.

Teams typically analyze the delivery velocity observed over the past 3–4 sprints, calculating an average to establish a baseline for future performance. This average velocity becomes a key metric in predicting the completion time of work items in the backlog.

For instance, let’s consider a scenario where there are 20 items in the backlog, collectively requiring 50 working days for completion. If the team’s velocity is determined to be 10 working days per sprint based on historical data, it implies that the team can deliver the entire backlog in approximately 5 sprints.

Now, if a stakeholder is seeking a prediction for when a specific item, say item number 15 in the backlog, will be delivered, the team can employ this velocity-based estimation. By summing up the working days of the first 15 items, let’s say it amounts to 38 working days. This suggests that delivering item #15 is likely to take approximately 4 sprints (38 working days divided by the team’s velocity of 10 working days per sprint).

This approach not only provides a more realistic and data-driven estimate but also acknowledges the inherent variability in software development. By focusing on sprints and velocity, teams can communicate delivery timelines in a manner that aligns with the iterative and adaptive nature of Agile methodologies. It also allows for a more transparent and collaborative relationship between teams and stakeholders, fostering a shared understanding of project timelines based on historical performance and empirical data.

Conclusion

In summary, the complex nature of software development, with its unknown unknowns and dynamic requirements, makes exact date predictions challenging and most times not feasible unless waste is introduced into the process. The Waterfall model’s rigid planning and Gantt charts with fixed dates proved inadequate in the face of this complexity. Agile methodologies, emphasizing priority, adaptability, and iterative development, offer a more effective approach. Metrics like velocity along with backlog priority provide more realistic insights into progress, allowing for better-informed predictions.