Please note that this blog is archived and outdated. For the most current information click here!

How Accurate Are Software Estimates? Traditional vs Scientific Estimates

One of the first questions asked when undertaking any type of project is usually how much will it cost? It's a reasonable question to ask, before making a decision it's only natural to want all the facts at hand. There are some industries where this has become the norm - the painting and construction industries come to mind. However, this is vastly different in the software industry.

This article is designed to unpack the software estimation process and the different approaches used in the industry.

The industry

Generally the next follow up question asked is why can’t you tell me exactly how long it will take? Software differs to other industries as it deals predominantly with unknowns. Every piece of software is designed to be different in one way shape or form, otherwise there is no unique value proposition. Couple that with the fact that technology continues to improve rapidly, with a new ‘industry standard’ emerging every couple of years and it becomes incredibly unlikely that a software developer will build the same application, the same way, twice.

The problem with software estimates

There is a societal problem when it comes to estimating work. In many instances the cheapest estimate wins the work. This wouldn't be an issue provided the estimate was accurate and did not impact the quality of the project. So if the project blows out or the quality is not up to scratch, does the blame rest with the development company for low-balling the project or the customer that chose the cheapest option? With a greater knowledge and understanding of the estimations process, we can move away from this cycle.

When to estimate

There are many competing lines of thought here with different strategies recommended based on which phase the project is in. For the purpose of this article we'll break a project into the common stages; ideating, scoping and developing.

Ideating

Before the project is properly scoped it is still in an ideation phase. This means all the requirements are not yet known.

Bracketing

This is one of the more popular approaches to undertake when the software is at such an early stage. Based on the software developer's prior experiences and gut feeling they may give a broad bracket as to the length of time based on a few central requirements.

Historical comparison

If the complexity and size of the project sounds similar to a past project than the estimate given may simply be the time it took to build the past project.

The problem with giving an estimate this early on in the process is that there are so many unknowns. Generally, the client will budget and set their expectations based on a very early stage estimate (if it has been given). What happens when the scope is unpacked and the initial estimate is inaccurate? It creates tension and impacts the trust between the software development company and the customer.

No development estimate

WorkingMouse's preferred approach to estimating a project before it is scoped is simple. We don't. We can estimate the time it will take to properly scope the project based on its complexity but we do not have enough information to estimate development time.

Scoping

T-Shirt sizes

This exercise is one of relativity. By marking functionality as XS, S, M, L, XL (and so on) we can group together similar pieces of functionality. It's generally much faster than traditional time based estimates.

Once the tickets are grouped, estimation values can be put to each group. For example, an XS ticket may on average take 2 hours to complete. In a short space of time, the development team and product owner can start gauging the relative size of the application. Keep in mind, the estimate can still be inaccurate at this stage.

Fibonacci estimations

As mentioned above, software estimations are inherently quite difficult. The Fibonacci-type approach to estimations tries to simplify things. It is based on the theory that the bigger something is, the less precise we can be. It is reasonable to assume that a small piece of functionality can be estimated down to the hour. However when we start looking at bigger pieces of functionality, for example complex API integrations, which have a high degree of complexity and difficulty, we cannot be that precise. Something may be estimated to take a week or 3 days but it would be bold and ultimately unwise to estimate that it will take 1 week, 1 day and 2 hours.

It is recommended to wait until the end of scope, once all the immediate functionality is known to hold an estimations session. Without stakeholders aligned on the acceptance criteria of each piece of functionality and its impact across the wider application, it is difficult to ensure the accuracy of the estimates.

Planning poker

This is a slightly different way of running the Fibonacci approach to estimating. Everyone is given a set of cards with lengths of time. It's designed to ensure that one team members estimate (the first to speak) doesn't influence other team members estimates.

Development

During a planning session

During development, planning sessions occur at the beginning of each iteration (or sprint). It gives the development team an opportunity to learn from earlier iterations. It also gives them an opportunity to feed learnings back into the estimations.

Let's say for example after two iterations a developer was able to leverage a React library more than initially anticipated. That might bring the estimates for some functionality down, allowing more to be completed during the iteration. On the other hand, there may be a complication that means certain functionality takes longer than initially expected. By elaborating on estimations during the planning session, the development team has the most recent information available to make an estimate.

Types of estimations

While all the techniques listed above are helpful, it's more important to distinguish between traditional estimations and how they've been refined to embrace a more scientific approach.

Traditional software estimations

These are quite simply estimates against functionality. By asking how long "X" will take and doing some simple addition, that will give you the traditional project estimate.

The issue with this approach is that it fails to capture a number of other factors that have a major impact on time. For example, it is unrealistic to believe that 100% of a developers day will be spent working on a project. There are planning meetings, morning huddles and company-wide meetings that impact productivity.

Scientific software estimations

By understanding and measuring the impact that other factors have on development time, we can be more scientific in our estimations. After running an experiment over a number of projects we found a few factors impacted development length. These factors will be addressed in more detail below.

Each of these factors have a multiplier based on the impact that they have on estimates. As mentioned above, a rushed project can detrimental to the quality of the application. It is absolutely necessary to take these factors into consideration in order to accurately set expectations.

Risk is the most complex factor to measure. As a general rule, some projects are riskier than others. So then, how do we capture risk? Our approach is to separate risk into two contributing factors; complexity and unfamiliarity. These are measured on a 1-5 scale for each piece of functionality. The higher the risk, the greater the estimate for that functionality.

What is the average estimate for a feature?

As mentioned earlier, it is notoriously difficult to estimate a software project before it is scoped. However, there are a range of learnings that can be made by looking at historical data for trends. To give you as much insight as possible, we’re sharing novel data, taken from past projects at WorkingMouse.

Table 1: Average size of past tickets at WorkingMouse

This table tells us a few things.

Firstly, there is risk in leaving tickets large (over 1+ day in length). It is common for developers and development agencies to break these tickets down into smaller, more manageable pieces. Rather than a single 2-day ticket which might encompass a few components, there is less risk in 3x 5-hour tickets that are smaller in size and more focused in functionality.

Secondly, there are minimal tickets that can be completed in under 4 hours. Between feature development, writing tests and releasing, it’s difficult to complete a ticket to a high quality in under 4 hours.

Finally, there is safety in the pack. This can be a result of sequence bias or the golden mean fallacy. The sequence bias is the likelihood that the prediction before will influence the prediction after. So, if the first few tickets are medium sized tickets, then the likelihood that the next ticket is also estimated as a medium ticket is raised. The golden mean fallacy is the perception that the truth lies somewhere in the middle. Hence the idea that the right estimate is somewhere between 4 and 8 hours in length.

How to increase the accuracy of a software estimate

The million-dollar question; how do we improve as an industry and increase the accuracy of software estimates?

This comes down to refining the way that we estimate. We approach this problem with the view that no matter what the solution is, it will never be perfect. But we can continue to try get closer and closer to perfection.

The best scientific formula we’ve created to date takes into consideration:

• Feature development,
• Testing,
• Allocation for other tasks,
• Discovery,
• Peer review,
• Delivery/releasing.

This article represents our current approach to software estimations. In the past these factors and the influence they have on estimates differed. It’s only through a mode of continuous learning that we can increase the accuracy of software estimates.

To summarise

If you’ve gotten this far, well done. While the science behind software estimations may not excite many, there is no question that they have a huge impact on the success of a project. Start with an inaccurate estimate and you’ll find yourself on the back-foot, often at the expense of quality.

My advice is to ask your development company how they approach the estimations process. If they’re willing to give a quote or estimate before the scope is fully fleshed out and agreed upon, that should be a red flag. Also ask about the allowances they have for necessary work that isn’t feature development.