Tuesday, April 4, 2017

Lies, Damn Lies, and Traceability Matrices

Beware the Traceability Matrix. It deceives and misleads.

Let's take some requirements for a calculator that works out how much it will cost to park at an airport carpark:

Parking Calculator Application

I've written some test cases for each of these requirements. I've written these test cases in good faith following what proponents of test cases consider to be good practice. Each test case is written as unambiguously as possible; each step has an expected result with clear pass/fail criteria:

Test Cases for Parking Calculator

These test cases, as written, will pass (for a slightly more complex observation of what will happen when these test cases are executed, see this article, but in theory, and in intention, they pass).

As shown above, I've numbered the requirements, and you will see that there's at least one test case for each requirement.  Here is a traceability matrix:

According to the traceability matrix, I have "100% test coverage" and a "100% pass rate".  According to plenty of exit criteria and test plans I've read in my career, I have met the criteria needed to declare that the product is tested and ready to exit testing.

But wait

What about this?

...or these?

A famous game we play in the context-driven testing community is to try and have the calculator calculate the largest cost we can.

The point is though, is that there are problems with the calculator. Problems that people who matter might be interested in.  Problems invisible and masked by a seductively green traceability matrix.

Even worse, many test management tools automatically generate the traceablity matrix as a primary story-telling tool, creating bewitching tables with hypnotising statistics, with no warnings.

You might look at my test cases and, even though I wrote them with honesty and good faith, find flaws in them. Surprise surprise, just like real life. But unlike real life, I'm not giving you a traceability matrix with no context.

Traceability Matrices are like the sirens of mythology: Enchanting, but very dangerous. Never take a traceability matrix at face value.


  1. Nice post Aaron. As much as I dislike almost all testing metrics I do have one use for traceability matrices. Not enough for me to suggest that people use them unless they need to because they are in a regulated environment - but a use nonetheless.
    If I have completed my tested (scripted or exploratory) and I have spent the time to fill in the traceability fields. I might get useful information: if I am expecting 100% coverage and I get anything less. In other words, the requirements traceability coverage (or code coverage) metric can tell me if I have completely missed something. It cannot tell me that I have adequate coverage for any requirement but it can tell me I have no coverage.
    Once the 100% value is reached then the usefulness drops to zero and it is replaced with a dangerous false sense of security.

    1. Yes! A traceability matrix is merely a signpost towards other information; it gives no information itself apart from "someone at some point has linked one model to another another model". It can raise interesting questions, and be used to direct inquiry, but it's not a source of truth unto itself.

    2. IEEE Final Year projects Project Centers in India are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation. For experts, it's an alternate ball game through and through. Smaller than expected IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble. Final Year Projects for CSE It gives you tips and rules that is progressively critical to consider while choosing any final year project point.

      JavaScript Online Training in India

      JavaScript Training in India

      The Angular Training covers a wide range of topics including Components, Angular Directives, Angular Services, Pipes, security fundamentals, Routing, and Angular programmability. The new Angular TRaining will lay the foundation you need to specialise in Single Page Application developer. Angular Training

  2. This comment has been removed by a blog administrator.

  3. I personally like your post, you have shared good article. Research papers writing service It will help me in great deal.

  4. My friend recommended this blog and he was totally right Android Application Development Service keep up the good work

  5. In the event that you have faith in others and give them a positive notoriety to maintain, you can enable them to wind up noticeably superior to anything they to think they are, Helpful links for Homework We need to welcome that dealing with the outcomes of conduct is the most ideal approach to transform it. I'm unpleasant at posting routinely; I don't merit the blog achievement! be that as it may, i saw your blog, its truly astonishing, i like it,

  6. Its a stunning post, I truly like it, Essay Experts For everything you do, for your identity, I will be everlastingly appreciative you, Words can't express my emotions, nor my much obliged for all your assistance, If I see something questionable, say on a blog or a Web website, and I don't see it anyplace else, I'll simply go ideal to the source and look at it. pleasant blog, keep it up.

  7. On our website we provide a wide range of glasses of choice, quality and price, come please click here! !

  8. look at our website, we provide information about favorite bands, Come please click here !! visit !!!

  9. Thank you for that information you article great article, I was very impressed about it, wish you would have stayed next share

  10. It is very nice that you mention this in the article. It's good to read such articles.

  11. There is a value services provide the best internet services with the fiber cable
    it provides the best services in Australia the value services are the Australian-based organization which has provided an upgraded version of the internet and it is the most beneficial and fastest internet connection.