The Advantages of Using a Traceability Matrix as a Testing Metric

SOFTWARE DEVELOPMENT

Believe it or not but soft­ware test­ing is a di­vi­sive topic. Given that ap­pli­ca­tion qual­ity is one of the most dif­fi­cult char­ac­ter­is­tics to mea­sure, the way in which tests are writ­ten and the tools used to run and mea­sure those tests are of­ten un­der the mi­cro­scope.

Now from a prod­uct owner per­spec­tive you might think, this is all good and well but I just need an app that works. The is­sue with this ap­proach is that there’s no such thing as per­fect soft­ware. There are too many vari­ables and edge cases to cover every sce­nario. Taking a naïve ap­proach means that you only know there’s a bug when it’s too late (a user has found it and has had a frus­trat­ing ex­pe­ri­ence).

This is where test­ing and in par­tic­u­lar, the trace­abil­ity ma­trix, comes into play.

Why do we test?

We’ve al­ready touched on some of the rea­sons why test­ing is im­por­tant. Not only does it pro­vide a proac­tive re­sponse to qual­ity as­sur­ance, it also gives you a snap­shot of the health of your ap­pli­ca­tion.

I won’t spend any more time dis­cussing why test­ing is im­por­tant, this is gen­er­ally ac­cepted by every­one in the in­dus­try. Rather, we’ll look at dif­fer­ent ap­proaches to test­ing.

The dif­fer­ent test­ing met­rics: code cov­er­age vs trace­abil­ity

One of the most com­mon test­ing met­rics is look­ing at code cov­er­age.

Code cov­er­age looks at the per­cent­age of your code that is tested. Let’s say within your ap­pli­ca­tion, 2000 lines of code are linked up to a test and 2000 lines of code aren’t. You would have 50% code cov­er­age.

There are a few rea­sons why we be­lieve code cov­er­age is a poor met­ric to base the health of your soft­ware off:

  • It does­n’t link with the whole of the ap­pli­ca­tion. If a line of code fails, what does that mean for the us­abil­ity of the ap­pli­ca­tion.
  • It’s naïve in as­sum­ing that if code is run dur­ing test ex­e­cu­tion, then it is tested. This is not al­ways true.
  • Chasing 100% code cov­er­age is in­cred­i­bly time con­sum­ing — on a big enough ap­pli­ca­tion it’s al­most im­pos­si­ble to test every branch of every piece of logic. Think of the dif­fer­ent ways you could test a free text field with a 100 char­ac­ter limit (the vari­a­tions of in­puts are enor­mous!)

While un­der­stand­ing code cov­er­age is bet­ter than no met­ric, we be­lieve there are more ef­fi­cient ways to get value out of your test suite. This comes in the form of a trace­abil­ity ma­trix.

What is a trace­abil­ity ma­trix?

Where code cov­er­age looks at the pro­por­tion of lines of code that have been tested, trace­abil­ity looks at what func­tion­al­ity has been tested.

We be­lieve that code is a by-prod­uct of fea­tures. At the end of the day, we only re­ally care about fea­tures work­ing. If we go back to why we test; it’s about proac­tively mon­i­tor­ing the health and qual­ity of your ap­pli­ca­tion for your end users.

The trace­abil­ity ma­trix works best when you only in­clude tests that ac­tu­ally mean some­thing. If the test fails will it tell you some­thing about the qual­ity of the ap­pli­ca­tion? Will you re­spond to it or ig­nore it?

Data is only as valu­able as the in­sights we draw from it.

We’ll dive deeper into the ad­van­tages shortly but the key pur­pose of a trace­abil­ity ma­trix is that it tells you whether a fea­ture has a test as­so­ci­ated with it, and if so, how many.

There’s a mock ver­sion of the ma­trix in the im­age be­low which will help il­lus­trate how it works.

Importantly, you don’t need to be a de­vel­oper to un­der­stand the trace­abil­ity ma­trix. From our mock ma­trix we know that 64% of re­quire­ments have tests that are pass­ing, 25 re­quire­ments have failed and 65 re­quire­ments do not have any tests as­so­ci­ated with them (red flag).

The pass­ing cov­er­age breaks down the re­quire­ment name and ID (imported from Jira) and the num­ber of tests as­so­ci­ated with each re­quire­ment, along with the sta­tus of those tests.

Just as no ap­pli­ca­tion is ever ‘done’, no test­ing suite is ever fin­ished. Regression tests (after the fact tests that check none of the ex­ist­ing func­tion­al­ity is bro­ken) should be added and mapped against fea­tures in your trace­abil­ity ma­trix.

Advantages

There are some clear ad­van­tages from us­ing the trace­abil­ity ma­trix, as well as a few trade-offs (or cau­tions).

Visibility for the prod­uct owner

By cre­at­ing a dash­board/​re­port ab­stracted out­side of de­vel­oper tools, it gives the prod­uct owner a level of in­sight they would­n’t oth­er­wise have.

While it’s im­por­tant to trust your de­vel­op­ers, it’s also dan­ger­ous to have no vis­i­bil­ity. With a high-level overview, you’ve got in­sight into the met­rics that mat­ter with­out risk­ing an­a­lyt­ics fa­tigue.

You ac­tu­ally know what your code is test­ing

We know that a fail­ing test is a bad thing, but why? What im­pli­ca­tions does it ac­tu­ally have for the ap­pli­ca­tion and for the end user?

The con­cept be­hind the trace­abil­ity ma­trix is that we know, for ex­am­ple, the fail­ing test re­lates to the lo­gin page, there­fore users won’t be able to lo­gin un­til the bug is re­solved. This fea­ture is a crit­i­cal part of the ap­pli­ca­tion so we might sound the alarms and get all hands-on deck un­til the bug is re­solved. Contrast this to lesser used func­tion­al­ity that may not need an ur­gent res­o­lu­tion, we may de-pri­ori­tise the bug in this case in favour of some­thing else.

Better un­der­stand­ing of test den­sity

Rather than sim­ply hear­ing 80% of the ap­pli­ca­tion code is tested, you know the den­sity of tests mapped against each re­quire­ment.

This comes in handy if you know there’s a par­tic­u­larly com­plex fea­ture that needs a higher level of test­ing against it. Going back to our mock ma­trix, we ex­pect the CRUD Admin pages to be more com­plex than the lo­gin page, there­fore we’re com­fort­able that there are ex­tra tests against that fea­ture.

Cautions

Beware of test qual­ity

Nothing in the trace­abil­ity ma­trix ac­counts for test qual­ity. It’s a rep­re­sen­ta­tion of tests matched against fea­tures. You still need de­vel­oper dis­cre­tion and strong com­pe­tency in writ­ing good qual­ity tests.

For WorkingMouse we ad­dress test qual­ity con­cerns through our de­f­i­n­i­tion of done (criteria that must be ticked off be­fore clas­si­fy­ing a ticket as done).

Tagging takes time

In or­der to cre­ate a trace­abil­ity ma­trix every test needs to be tagged with the re­quire­ment it re­lates to.

As you might ex­pect this can be time con­sum­ing when you’re com­plet­ing hun­dreds of tick­ets as part of a pro­ject. It’s highly rec­om­mended to ‘tag as you go.’ It’ll add a minute (or so) to each ticket but it’ll save it from be­ing a big ac­tiv­ity/​ex­er­cise at the end.

Takeaways:

  • As a prod­uct owner, un­der­stand­ing the health of your ap­pli­ca­tion is im­por­tant.
  • The trace­abil­ity ma­trix is a bet­ter and more in­sight­ful met­ric than code cov­er­age.
  • Metrics are only as valu­able as the in­sights you draw from them. Remove use­less tests from your trace­abil­ity ma­trix.
  • Proactively mon­i­tor your tests and con­tinue to aug­ment.

Discover Software
Secrets

ABOUT THE AUTHOR

Yianni Stergou

Marketing en­thu­si­ast and FIFA ex­tra­or­di­naire

Get cu­rated con­tent on soft­ware de­vel­op­ment, straight to your in­box.

Migration vs Rebuild

12 November 2018

The top tech­nol­ogy frame­works you can use to build a mo­bile app

21 January 2020

How does end of life soft­ware im­pact you?

10 November 2020

Your vi­sion,

our ex­per­tise

Book a con­sul­ta­tion