Wednesday, November 5, 2014

Shallow KPIs: A Tale of Two Testers

Once upon a time there was a thread on linked in on KPIs for software testers.  A Test Manager shared the KPIs she uses for her team:

1. Amount of bugs created.
2. Amount of bugs verified
3. Amount of assigned work completed.
4. Confirm to schedule. 

(At the risk of accusing anyone on Linkedin of being sloppy with their language, I will assume by 'Amount of bugs created' she means "number of bugs logged in some bug tracking tool").

When challenged, she provided the following 'real life' scenario as if it the sheer power of this example would dazzle us all into submission:

"Tester1 – found 30 defects, verified all assigned issues by deadline.
Tester2- found 0 defects, verified 10% of issues assigned by deadline.
Who performed better Tester1 or Tester2?"

So who performed better?

Tester 2 of course. She didn't log any defects because she had established a strong working relationship with the development team, and as she found an issue, she wrote it on a sticky note, and gave it to the developer. The developer then would rapidly fix and redeploy. The tester would retest, and verify the fix. Because of this, she was able to reduce a lot of administrative overhead, and help the developers produce a high quality product.
Tester 2 was unable to verify all the issues assigned to her by the deadline because she was very thorough, and felt that meeting an arbitrary deadline didn't contribute to the overall health of the project. Instead, she focused on doing great work.

Meanwhile, Tester1 logged many defects. They were poorly written, and many of them were just different symptoms of the same underlying issue. The developers had to spend a lot of time trying to decipher them, and would often spend many hours chasing down bugs that turned out to be merely configuration errors. Once, he logged 10 'defects' that were immediately 'fixed' when someone came over and updated his java environment. A lot of time is spend administering the defects in the bug tracking tool, and trying to work out if Tester 1's defects are legitimate or not.
Tester 1 works very hard to meet the deadline when verifying issues. To do so, he performs a very shallow confirmatory check. His vulnerability to confirmation bias has led him to verify many fixes as "complete" when there were regressive side-effects he didn't pick up on.

Tester 1 meets his KPIs and is up for promotion. In two years he'll be sharing his wisdom in Linkedin.

Tester 2 has been told she isn't performing as necessary. She is going home tonight to update her resume. In a year she'll be working at a company that assesses her performance by watching her work and regularly catching up for peer review. In two years she'll be sharing her wisdom at a peer conference.


  1. Wow, amazing blog layout! How long have you been blogging for?
    you make blogging look easy. visit here The overall look of
    your site is wonderful, let alone the content!

  2. I was very impressed by this post, this site has always been pleasant news. Thank you very much for such an interesting post. Keep working, great job! In my free time, I like play game: What about you?

  3. Very good post. I certainly love this website. Keep writing!

  4. I often get emotionally involved. Like when I read your post, I can cry, laugh, sad, funny, depending on your written words. I think you are a sensitive person.

  5. You’ve written nice post, I am gonna bookmark this page, thanks for info. I actually appreciate your own position and I will be sure to come back here.
    instagram viewer images

  6. Such an interesting article, I can totally relate!

  7. See more at the link: Power clean apk
    See more at the link: og youtube
    See more at the link: bobby movie
    See more at the link: vivavideo pro
    See more at the link: Fildo apk
    See more at the link: gangstar vegas mod apk
    See more at the link: sms bomber

  8. This blog shows very important things.
    Blonde girls

  9. the tutuapp apk download is the one of the best ios,android App tutuapp store to get the tons of free app and game. Pokego++ Here the latest version of TutuApp of free.


  10. the WhatsApp Plus APK is one of the best mod app for official WhatsApp Plus Download for free, get unlimited features of whatsapp plus for free. Here Click to download the latest version whatsapp plus apk.

  11. Hi This is SRK i'm suggesting you this is app Titanium Backup Mod Apk

  12. The Tutuapp is one of the best app stores where you” ll get all the things for the best mode app. There are many Spotify++ iOS multiple apps available in this field but this is the head of PokeGo++ all the apps. In the world of apps if you download only one app which gives you all the things if you open it then it’s the best option.

  13. The Whatsapp plus Today we all are using many social apps like WhatsApp, Instagram, Gb whatsapp Facebook, and many more. WhatsApp.This is the best time to download the latest version of Yowhatsapp and you’ll get the best version of WhatsApp on your devices.Fmwhatsapp is the mod version of WhatsApp, FMWhatsApp Apk which offers you a Mod version.