Inattentional Blindness in action - Anatomy of a testing session

 Inattentional blindness...the phenomenon of not being able to see things in plain sight.  This is how it just happened to me:



I'm testing an application that lets you create "shows" with a given resolution, but when you come to render the show as a movie file, you can override the show's resolution and choose another from a dropdown.  As I was testing away, I noticed that sometimes the output video didn't match what I had chosen from the dropdown.  After running some informal tests and not seeing the pattern, I drew a table to try to make sense of the situation.  I suspected it had something to do with overriding standard definition (SD) formats with High Definition (HD) formats

Since Blogger doesn't let me add tables the lines will be in the format:
Show spec ; Override value ; Result


First test:
720p ; No override ; 1280x720
So far so good.  A control test just to establish things when the show has an HD resolution

Second test:  Override HD format with SD format
720p ; override to 576i ; 720x576
Yup, that passed.  The output was set to the override value

Third test: establish control test for a show with SD resolution
576i ; No override ; 720x576
cool...so far so good

Fourth test: override SD format with HD format
576i ; override to 1080i ; 720x576
ah ha!  I might have this.  Hypothesis: "overriding SD format with HD format does not change the show's default resolution"

Now, let's be careful about confirmation bias here... so before we get carried away, let's run a couple more tests:
Fifth test: establish a control test for 1080i resolution
1080i ; No override ; 1920 x 1080

Sixth test: Let's go from one HD resolution to another..maybe when you change to a lower resolution, not just from HD to SD it'll keep the show's original resolution
1080i ; override to 720p ; 720x576
err wait...what?

Seventh test: (Starting to look up for patterns.  hmmm all the failing tests had an original video specification what was interlaced.  Let's try starting off with a progressive resolution)
576p ; override to 1080i ; 720x576
Nope not that.... now all failing tests are when I go from a progressive to an interlaced or vice versa.  Let's go from progressive to progressive

Eighth test:
576p ; override to 720p ; 720x576
Nope, that's not it...ok let's look at all my failing tests....

And as my eye glances up the table I see the pattern: All failing tests are outputting 720x576

Ninth test:
720p ; 480p ; 720x576

Yup, confident enough to stop testing now and declare: "If you override the show's original specification, no matter what resolution you pick, it will output in 720x576"

So simple, yet I was blind to it for an hour or so.

So what happened?

Look up at test two.  I was led down the garden path by how I interpreted the results of test two. I interpreted it as "The output was the show's original resolution" instead of "it outputs 720x576 regardless of what the original resolution is"

One false positive.

Look up at test four.  Again, I interpreted that as "The output was the show's original resolution" instead of "it outputs 720x576 regardless of what the original resolution is".

One false negative.

Such a simple bug confounded by the resolutions I had chosen to test with (576i which is our standard SD television resolution in this part of the world) combined with how I was interpreting my observations.

Lesson: Always be vigilant for bugs in the software you're testing, and for bugs in your testing.