WorkingMouse has developed an experimental framework to continue to refine our processes as we learn and adapt to new technologies and ways of working. Our approach allows anyone within the company to propose an experiment, trial it and document the results. Some software development experiments succeed and are adopted widely across the company while others fail. The purpose of this article is to share the findings of one of our recent successful experiments concerning using unit-test driven development for complex features.

Summary

Based on our scientific structure, the experiment was summarised by looking at why, what and how. The current landscape is predominantly centred around end to end testing which simulates user behaviour.

We believe that writing unit tests before implementing a change will result in faster implementation and less risk of problems with the change.

We will be experimenting by writing unit tests for a change involving complex logic before implementing the change itself.

We will know this experiment is a success when the unit tests are passing, the change is implemented and passes UAT.

End to end tests vs unit tests

Firstly, lets re-visit the differences between unit testing and end to end testing. The idea of a unit, is to break down the code into small, easily testable parts. It could be a single function, class or in our case, complex algorithm. This differs from an end to end test where you're testing the entire application. The point of an end to end test is to simulate how a user would use the application. Scratch creates a good template around this. If I do X then I expect to see XYZ.

Method

To evaluate the success, test driven development was adopted as part of four pieces of functionality on a complex public sector project. The team implemented a number of unit tests for the four pieces of functionality while still creating a significant number of end-to-end tests.

Evaluating Unit Test-Driven Development

• During the experiment, our end-to-end tests routinely failed with a high number of false negatives, whereas the unit tests were rock solid - no false negatives.
• The amount of maintenance required for the unit tests was very low - almost none.
• The amount of maintenance for the existing end-to-end tests was very high - usually in the order of a day or two of effort a week.
• Our confidence in the areas covered in the unit tests was high.
• The unit tests on the Jenkins board were always green.
• They are quick and easy to run compared to the end-to-end tests.

Going forward

Unit tests work well as another tool in our toolkit to compliment our end-to-end tests.

They are not intended as a replacement for end-to-end tests, but given they're more reliable, faster and lower maintenance, they are a good option in some situations.

The biggest benefit is for long-running products with frequent maintenance (eg. large public sector projects), or for areas of code where the logic is complex and difficult to decipher when results go wrong.

Everyone should keep an eye out for appropriate opportunities where unit tests would provide value.

Industry theory on unit testing

Before the experiment, we gauged how the industry perceived unit testing. Google's Mike Wacker summed up the value of unit testing in this blog. Unit tests are fast, reliable and help isolate features.

The other key benefit that we saw reflected in our results was the speed to finding and fixing bugs. With end-to-end tests, you have to wait for the software to be built, then for it to be deployed, and finally for all end-to-end tests to run.

Since the bugs are found early in unit testing, it helps in reducing the cost of bug fixes.

While this is a great tool to have in our toolkit for complex logic, there is still significant value in running end to end tests.