tag:blogger.com,1999:blog-72504473065748016432024-03-18T16:02:28.119+13:00The Adventures of TestKiwiA blog by Aaron Hodder (@AWGHodder)Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.comBlogger23125tag:blogger.com,1999:blog-7250447306574801643.post-39081846027877430792017-07-26T00:01:00.002+12:002017-07-26T09:32:56.938+12:00Talking vs CommunicatingIt's easy to conflate <i>talking </i>with <i>communication</i>. Sure, most of the time I would suggest that when people are talking, people are communicating. But the opposite isn't necessarily true. A lack of talking does not necessarily mean there's a lack of communication, or a lack of collaboration. We should be aware of this when making observations about people and the teams they work in.<br />
<br />
<a name='more'></a><br />
<br />
Last week, I had the privilege of experiencing the Silent Experiment at the Inclusive Collaboration summit (more on that in the future).<br />
<br />
I was sat with two people whom I'd never met before, and not spoken to before, and assembled a <a href="https://odysseyteams.com/helping-hands/" target="_blank">prosthetic hand</a> for an amputee in a developing country.<br />
<br />
The twist: It was in complete silence, without so much as a 'hello'.<br />
<br />
I expected the experience to be awkward.<br />
<br />
I was wrong.<br />
<br />
While we couldn't communicate with words, we found ourselves communicating with our movements, and with our faces. Because we had to communicate intent with our actions, our movements were deliberate and graceful. As we reached the limit of our understanding of what to do next, the task flowed from hands to hands as we passed the partially-completed construct along. We stepped in when we could help, and stepped out when we couldn't.<br />
<br />
It was like a dance, with the three of us moving in and out as we established our rhythm and syncronised out movements. We were methodical and calm ... at least on the outside.<br />
<br />
Inside, there were times I didn't understand the written instructions, and I wanted to express my bewilderment. Instead, I was forced to watch what the others were doing, and relate it back to the instructions. I would eventually understand, and perhaps see what needed to happen next, in which case, I'd step in to perform the next step. I was never left behind; we were continually watching out for one another, communicating with our eyes and with our actions.<br />
<br />
On reflection, I realised that had I been allowed to talk, I would have expressed my frustration, and we would have stopped as we discussed what was going on. It would have broken our dance. Reading my team-mate <a href="https://lizkeogh.com/2017/07/18/a-helping-hand/" target="_blank">Liz's experiences</a> I learnt that she too had an inner storm raging. If we had let these out, we would have injected anxiety and frustration into the team. I sincerely believe it would have slowed us down. Or if not, the process would certainly have been more stressful. Instead, we calmly proceeded, took care of one another, and got on with it.<br />
<br />
As we performed the experiment we wrote down on post-its what we wanted to say, and at the end we divided the post-its into "Things we wanted to say, but didn't need to..." and "Things we wanted to say...and still do."<br />
<br />
A big takeaway was that most of the post-its went into the "didn't need to" pile. It turned out that a lot of the talking we wanted to do was ultimately unnecessary. And the things people still wanted to talk about were mostly expressions of gratitude and congratulations like "we did it!" and "well done!" We got around this by giving each other silent claps.<br />
<br />
The biggest takeaway though, was that it made stark the gulf of difference between "talking" and "communicating". Without saying a single word, we communicated, and we collaborated, and we built a prosthetic hand.<br />
<br />
How often do we find ourselves saying "The problem with this team is that they don't <i>talk </i>with one another" or "You need to <i>talk</i> to people".<br />
<br />
I propose that when we hear the word 'talk', we swap it with the word 'communicate'.<br />
<br />
"The problem with this team is that they don't <i>communicate </i>with one another" or "You need to <i>communicate </i>with people".<br />
<br />
Because you can certainly communicate without talking, and you can certainly talk without communicating. <br />
<br />
Talking isn't the goal; it's a means to an end. The goal is communication, and communication can come in many forms. It's tempting to walk into a quiet space and assume that people aren't communicating. And it's tempting to try and "fix" it by getting people to talk more. But talking isn't everyone's preferred or even effective communication style.<br />
<br />
Not only that, but talking produces a lot of noisy spaces. Technology is a creative field, and there is a lot of literature <span id="goog_569345131"></span><a href="http://www.motivateamazebegreat.com/2016/12/how-silence-can-be-the-foundation-of-creativity.html" target="_blank">linking</a> <span id="goog_569345132"></span><a href="https://blogs.wsj.com/speakeasy/2013/11/07/do-silence-and-creativity-go-hand-in-hand/" target="_blank">quietness</a> <a href="http://discoverthought.com/Education/References_files/2%20Reflectivity%20creativity%20silence%20Dawson.pdf" target="_blank">with</a> <a href="https://en.wikipedia.org/wiki/Incubation_(psychology)" target="_blank">creativity</a>.<br />
<br />
Similarly, it's tempting to walk into a space full of chatter and marvel at all the communication, whereas, there may not be much signal to the noise. Again, I suggest just using the mental heuristic of swapping "talking" with "communicating" just to check yourself. There's a lot of talking, yes, but is there a lot of <i>communication</i>? Is there a lot of <i>collaboration</i>?<br />
<br />
I'm not saying that talking is bad, and silence is good. I just want to point out that talking and communication aren't necessarily the same thing, and by being conscious of that, we can prevent trying to 'fix' problems that aren't there, and prevent being blind to problems drowned out by chatter. I also encourage teams to explore collaboration styles that involve silence, to see what emerges.<br />
<br />
Please read more about Inclusive Collaboration and the Silence Experiment here: https://www.infoq.com/articles/inclusive-collaboration-silence-experimentAaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com33tag:blogger.com,1999:blog-7250447306574801643.post-19431414635984393482017-05-03T15:00:00.003+12:002017-05-03T15:03:23.415+12:00Phased vs Threaded Testing<div dir="ltr" id="docs-internal-guid-0494127f-cc3e-9b8d-032a-e51768cae378" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">There have been many attempts to model different approaches to software testing. You'll be familiar with many of them: exploratory vs scripted; traditional vs agile; testing vs checking; standards-driven vs context-driven; and many more.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">There’s another spectrum I’d like to introduce that I’ve derived value from using: Phased testing vs. threaded testing.</span></div>
<a name='more'></a><br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">As the name implies, it describes the testing lifecycle as either a series of phases or as intertwining threads.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">I've found this useful in encouraging debate or conversation about testing practice in a way that is agnostic when it comes to SDLC and testing politics, and therefore is a model that can encourage useful introspection without too much baggage.</span></div>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 20pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Phased Testing</span></h1>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Testing has traditionally been modelled and organised as a series of stages that occur more or less sequentially, symptomatic of the waterfall and enterprise contexts in which this tradition was established. When we talk of “traditional” testing, this is usually what is meant, and it often looks like this:</span></div>
<br />
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Requirements-gathering phase</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Read requirements documents</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Collect useful artefacts</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Establish acceptance criteria</span></div>
</li>
</ul>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Scripting Phase</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Codify test ideas into artefacts such as ‘Test Cases’</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Codify test ideas into artefacts such as automated test scripts</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Codify test “coverage” into a traceability matrix</span></div>
</li>
</ul>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Execution Phase</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Testers perform testing as described in their test cases</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Machines execute automated test scripts</span></div>
</li>
</ul>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Exit / Reporting Phase</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Test exit reports are written</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: circle; text-decoration: none; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Political arguments are had about the coverage and the information found</span></div>
</li>
</ul>
</ul>
<br />
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">This sounds fair enough, and the </span><a href="https://en.wikipedia.org/wiki/Software_test_documentation" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline;">IEEE-829 standard for software test documentation</span></a><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"> defines a whole document structure to support this model of testing. This phased approach to testing closely mirrors the phase-based waterfall model of software development, but can also be used within agile models of software development as well. </span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">For example, I’ve seen situations where “agile testing” has been defined as little more than writing automated acceptance tests for the acceptance criteria attached to a story; the only difference between this and “traditional” testing being that a machine is executing the test scripts, and not a human. Granted, in agile environments, a tester typically has the opportunity to add value above and beyond the actual ‘testing’ of the product, and the lifecycle may be hours or days, rather than weeks or months, but when it comes to the testing activities per se, you can describe this more accurately as a linear series of phases with little to no iteration. You may argue that’s not ‘really agile testing’, and I’d agree with you, so it’s useful to have a label to describe this pattern independently of the wider SDLC it occurs in.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Therefore, I would like to call this model of software testing the ‘phased’ approach, as it views testing as a series of phases that, for the most part, progresses linearly from one phase to the next. Having to go back to a previous phase is seen as some kind of regression, as some kind of failure to adequately plan, control or execute.</span></div>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 20pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Phased Testing reframed</span></h1>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">It strikes me that if we look at the actual </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">intent</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"> or </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">purpose</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"> of the activities that signify these phases, that we can actually reframe these phases as follows:</span></div>
<br />
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;">Requirements-gathering phase</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;"> --> </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Learning Phase </span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;">Planning Phase</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;"> </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;">--> </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;"></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Modelling Phase</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;">Execution Phase</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;"> </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;">--> </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;"></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;"></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Performing Phase</span></div>
</li>
<li dir="ltr" style="background-color: transparent; color: black; font-family: Arial; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;">Exit / Reporting Phase </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;">--> </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; vertical-align: baseline;"></span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Communication Phase</span></div>
</li>
</ul>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">With this in mind, let’s look at these phases again.</span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 16pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Learning Phase</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Why do we gather and read requirements documents, specifications? To learn. This is the phase where we are learning about the project, the problem it is trying to solve, and what has been designed (so far). We collect and read any and all accumulated artefacts, and talk to relevant stakeholders to gain an understanding of what’s going on, and how we are going to test. This phase is not a success simply </span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">because</span><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"> we have read documents and accumulated artefacts, it is successful if and only if we have developed sufficient understanding of the project and our contexts to begin productively engaging in testing work.</span></div>
<br />
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 16pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Modelling Phase</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">A model is, essentially, something that represents something else in some way. We use a model to help us to understand or manipulate the thing the model represents.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">There are many kinds of models. User stories, architecture diagrams, even source code, are all models for the actual software that will be compiled into the machine code that becomes what the user will ultimately interact with.</span></div>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Test cases are models. They are a model for how a tester or machine will interact with the software and what they will look out for. Therefore, I would like to reframe the “Planning” phase to the more general “modelling” phase, because the success of this phase is not that x number of test cases have been created, but that the tester has developed useful models to facilitate interaction with and evaluation of the product.</span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 16pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Performance</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<a href="https://www.testingcircus.com/documents/TestingTrapeze-2014-February.pdf#page=31" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline;">Test Cases are not testing</span></a><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">. Testing is not an artifact; it is not something that can be ‘written’.Testing is an event; an activity; a performance. Testing is the evaluation of a product by learning about it through experimentation. Part of the experimentation process and a potential outcome for our learning may be the execution of confirmatory test cases and scripts, but at some point, a performance has taken place.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">The success of this phase is not simply that the pre-defined scripts have been fully executed, it is that the product has been interrogated and evaluated to a satisfactory degree.</span></div>
<h2 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 18pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 16pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Communication</span></h2>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Since this is the phase where we produce the ‘test exit reports’ and the pass/fail results, and we receive feedback on these results, let’s call this the reporting/feedback phase.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">As a phase, it is often the case that testing occurs in a black hole until this final phase, at which point communication is (reluctantly) engaged in.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">This phase though shouldn’t be simply about “exiting” testing, its success should be whether the test group have successfully communicated valuable and useful information about the state of the product to the wider project team and stakeholders.</span></div>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 20pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">The apparent absurdity</span></h1>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">When we reframe these phases, it starts to appear a little absurd. The idea that we constrain our learning to a phase, and once we’re done learning, we then move onto codifying that learning into test cases doesn’t appear to be a process that fosters continuous learning and innovation. Test cases are opaque. They are hard to understand. Hard to change. Which means that even if we decide to iterate our models after performing some hands-on testing, it is only with great difficulty. And because going ‘backwards’ through the phases is so expensive and painful, we invent structures to help us mitigate change. We have ‘entry criteria’ and ‘exit criteria’ and ‘stage gates’ and ‘sign-offs’.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">The phased approach to testing ignores that testing is all about ‘change’.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">We change one idea about how something works for another.. We learn more about what people want when we build models, and solicit feedback. We then change our models to incorporate that feedback. Designs change when we learn new things about the problem we want to solve, or the limitations of our proposed solution. </span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Let’s look to an alternative model of testing. One that embraces and acknowledges change.</span></div>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 20pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Threaded Testing</span></h1>
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">As an alternative to viewing testing as a series of discrete phases that only move forward (or back only with great pain and difficulty), we can instead consider each of these phases as threads that intertwine and influence one another throughout the entire testing lifecycle.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">There is also an element of strategy which, like a cable jacket around wires, guides and directs how those activities occur.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"><img alt="Threads.png" height="237" src="https://lh3.googleusercontent.com/jEpiXtscrT89yRO0rs7kIwJY1z35oUmDHk1VNl_1MRBLyxOr0Aa_d0nhrQI-LG_QgpVW8E0RwREAlQPtwAEewUGN-8-pMT1HEFUMP1klHolWa-mdmbDlPI1_Zgmseme0BLNelQF8" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="602" /></span></div>
<br />
<br />
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Therefore, we can talk about testing as being ‘phased’ or ‘threaded’ in accordance to the degree that the different threads influence one another, and how iterative the approach is.</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">In this model, change is embraced, and we let new information influence our behaviour. We let our models adapt and evolve, and we perform testing to learn, and to get feedback on what we’ve learnt. Of course, with a model of testing so malleable, we need structures and techniques that lend themselves to rapid adaptation, which is why you’ll find me often advocating for </span><a href="https://testerkiwi.blogspot.co.nz/2011/04/building-exploratory-test-plan-redux.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline;">visual</span></a><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"> </span><a href="https://testerkiwi.blogspot.co.nz/2013/12/thoughts-around-test-estimation-irony.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline;">test </span></a><a href="https://www.testingcircus.com/documents/TestingTrapeze-2015-June.pdf#page=15" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline;">management </span></a><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">tools such as mindmapping software, and approaches to test performance closer to the </span><a href="https://testerkiwi.blogspot.co.nz/2011/03/exploratory-testing-is-pleonasm.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline;">exploratory</span></a><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"> end of the </span><a href="http://www.developsense.com/images/ExploratoryScriptedContinuum.png" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline;">spectrum</span></a><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">.</span></div>
<br />
<br />
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">In the past, I’ve talked about </span><a href="https://testerkiwi.blogspot.co.nz/2016/02/lean-testing-in-theory-and-practice.html" style="text-decoration: none;"><span style="background-color: transparent; color: #1155cc; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: underline; vertical-align: baseline;">‘lean testing</span></a><span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">’, and I now consider a threaded testing approach to be a key tactic in achieving a lean testing approach, almost to the point of being synonymous. </span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="background-color: transparent; color: black; font-family: "arial"; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">The intention to describe testing as either phased or threaded is to be agnostic when it comes to SDLC and testing politics, and to encourage useful discussion without too much of the baggage sometimes associated with other terms. I have used it when I’ve wanted to talk about changing testing paradigms without getting derailed or sidetracked by other terms which can be politically loaded in the environments I sometimes operate in.</span></div>
<br />
<i>Thanks to Adam Howard (<a href="https://twitter.com/adammhoward" target="_blank">@adammhoward</a>) for his patience in reviewing the writing and rewriting of this post</i><br />
<br />Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com52tag:blogger.com,1999:blog-7250447306574801643.post-82421747282463347802017-04-04T12:26:00.001+12:002017-04-04T12:47:29.199+12:00Lies, Damn Lies, and Traceability MatricesBeware the Traceability Matrix. It deceives and misleads.<br />
<br />
<br />
<a name='more'></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a><a href="https://www.blogger.com/blogger.g?blogID=7250447306574801643" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"></a>Let's take some requirements for a calculator that works out how much it will cost to park at an airport carpark:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh18_5ZqoKPmfAwdoIJ8s2fWCH_TFo_iTw80yvVhmHpPsDSfPzdlxAN-0zYQqp1Akt-Qvs9LcyBEclzlKzgnEO7pRvG6MVDBFI14ZYnXuqTNTuFYFuI2c-r9qqqfEkWgQSL75eYMUHZX5k/s1600/Capture.JPG" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" height="170" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh18_5ZqoKPmfAwdoIJ8s2fWCH_TFo_iTw80yvVhmHpPsDSfPzdlxAN-0zYQqp1Akt-Qvs9LcyBEclzlKzgnEO7pRvG6MVDBFI14ZYnXuqTNTuFYFuI2c-r9qqqfEkWgQSL75eYMUHZX5k/s400/Capture.JPG" width="400" /></a><br />
<br />
<a href="http://adam.goucher.ca/parkcalc/" target="_blank">Parking Calculator Application</a><br />
<br />
I've written some test cases for each of these requirements. I've written these test cases in good faith following what proponents of test cases consider to be good practice. Each test case is written as unambiguously as possible; each step has an expected result with clear pass/fail criteria:<br />
<br />
<a href="http://bit.ly/2nD1bWE" target="_blank">Test Cases for Parking Calculator</a><br />
<br />
These test cases, as written, will pass (for a slightly more complex observation of what will happen when these test cases are executed, see <a href="https://solavirtusinvicta.wordpress.com/2013/12/20/insanity-and-illusion/" target="_blank">this article</a>, but in theory, and in intention, they pass).<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
As shown above, I've numbered the requirements, and you will see that there's at least one test case for each requirement. Here is a traceability matrix:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjb44oKohU_MJrX7Wnkh25Vnvox0EPNKS6hNh3Bcz6Qbfm0ekh-bPVxZYsbU555uELlkKP-xnPf_CIpkQCcdVXr9i1RG89DlUbd5onF1-a3bZRKb2ZfCLRR0i8hd_GhVEHoxIsV-uW_-N4/s1600/Capture.JPG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="227" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjb44oKohU_MJrX7Wnkh25Vnvox0EPNKS6hNh3Bcz6Qbfm0ekh-bPVxZYsbU555uELlkKP-xnPf_CIpkQCcdVXr9i1RG89DlUbd5onF1-a3bZRKb2ZfCLRR0i8hd_GhVEHoxIsV-uW_-N4/s400/Capture.JPG" width="400" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
According to the traceability matrix, I have "100% test coverage" and a "100% pass rate". According to plenty of exit criteria and test plans I've read in my career, I have met the criteria needed to declare that the product is tested and ready to exit testing.<br />
<br />
<h3>
But wait</h3>
What about this?<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4H7R3Sf-Xq4uPcF20_1FCUHVF3NP21ynwN00V_rg8eYrTM_pL7fzOJ4J6T-Y1QKz4nITQlBTkqPJAJeb1U1Za3_KIqFnOtoLyXEXSXWHg5U2YmI7rIyIQB8L3Sa6ex5vvJx1XpGypdtc/s1600/Capture.JPG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="166" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4H7R3Sf-Xq4uPcF20_1FCUHVF3NP21ynwN00V_rg8eYrTM_pL7fzOJ4J6T-Y1QKz4nITQlBTkqPJAJeb1U1Za3_KIqFnOtoLyXEXSXWHg5U2YmI7rIyIQB8L3Sa6ex5vvJx1XpGypdtc/s400/Capture.JPG" width="400" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
...or these?<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvry3NmtOpVoXRCfEaS7qv2OVRqOd_Yy0ePT7Tb3eG5m1uJxKHK-vqvcODYTqIiXiW9p6yv6md5COeIv7ok046_JwwstgcVulkyP0tssnuA_tIB-ZTqUqaR-BjyjWIsHpBjHO574elAiQ/s1600/Capture.JPG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="163" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgvry3NmtOpVoXRCfEaS7qv2OVRqOd_Yy0ePT7Tb3eG5m1uJxKHK-vqvcODYTqIiXiW9p6yv6md5COeIv7ok046_JwwstgcVulkyP0tssnuA_tIB-ZTqUqaR-BjyjWIsHpBjHO574elAiQ/s400/Capture.JPG" width="400" /></a></div>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiojmrPh2tDc91Uk7bJ-Kvai4qWD73_Zg58qVFjYvzguS6aO0jhiLMQTcoZCNrOKZpPsiKzSTdz9n4rzqfUeZu98kBFimpPXQDkmWmTajDy-_ExJgtTbpVmLf7DHNPuaGKHVwvZu1VS1-o/s1600/Capture.JPG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="163" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiojmrPh2tDc91Uk7bJ-Kvai4qWD73_Zg58qVFjYvzguS6aO0jhiLMQTcoZCNrOKZpPsiKzSTdz9n4rzqfUeZu98kBFimpPXQDkmWmTajDy-_ExJgtTbpVmLf7DHNPuaGKHVwvZu1VS1-o/s400/Capture.JPG" width="400" /></a></div>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span>
<span style="text-align: center;"><br /></span><br />
<span style="text-align: center;">A famous game we play in the context-driven testing community is to try and have the calculator calculate the largest cost we can.</span><br />
<br />
The point is though, is that there are problems with the calculator. Problems that people who matter might be interested in. Problems invisible and masked by a seductively green traceability matrix.<br />
<br />
Even worse, many test management tools automatically generate the traceablity matrix as a primary story-telling tool, creating bewitching tables with hypnotising statistics, with no warnings.<br />
<br />
You might look at my test cases and, even though I wrote them with honesty and good faith, find flaws in them. Surprise surprise, just like real life. But unlike real life, I'm not giving you a traceability matrix with no context.<br />
<br />
Traceability Matrices are like the <a href="https://en.wikipedia.org/wiki/Siren_(mythology)" target="_blank">sirens</a> of mythology: Enchanting, but very dangerous. Never take a traceability matrix at face value.<br />
<br />Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com36tag:blogger.com,1999:blog-7250447306574801643.post-7837652769853672032016-10-02T14:57:00.000+13:002016-10-02T18:27:25.261+13:00An Anecdote and a (small) Editorial - The man who opened the door to testing for meI worry about posting this. I'm worried I'm going to be told to 'check my privilege'. I'm worried I'm going to be told my perspective isn't important. But to add a positive experience isn't to deny that bad experiences happen. I also don't like getting involved in Twitter dramas, but as an opinionated observer, I find their pull irresistible at times. Especially when they involve people I consider to be friends.<br />
<br />
There are anecdotes of people leaving the testing field because of James Bach. I'm not denying this has happened. But if we're collecting anecdotes, I'd like to add mine. <br />
<br />
<h3>
An Anecdote</h3>
Towards the end of 2009, I had been testing for about two years and I had no public profile. The kind of testing I was doing I would describe now as 'exploratory testing in an agile environment.' I didn't know those words back then though. Describing what I did with the words I knew back I would say "I was doing ad hoc testing in an unstructured environment". I had done ISTQB, gone to local meetups, and interacted with other testers who worked for big corporations. They talked about things like "requirements" and "test scripts" and "traceability matrices". I'd lie in bed at night worrying about what a fraud I was being. You see, I was doing things like "talking to the developers" and "asking them what the most important things to test were." When I was done, I'd report back to them. Not in a test case management tool; I'd walk to their desk and talk to them. Sometimes I'd be as formal as writing an email, or a two-page summary document. I tried testing 'properly' once, but it didn't make much sense to me. I was obviously just not 'getting it' which added to my anxiety. <br />
<br />
The STANZ (Software Testing Australia New Zealand) Conference 2009 came around and I begged my employer to go as I was desperate to learn how to 'do testing properly'. One of the speakers was James Bach. Oh he's an expert in the field! His talk was called "Becoming a Software Testing Expert".<br />
<br />
Perfect.<br />
<br />
In his talk, he talked about "The Tester's Mindset". There was lots of talk about thinking critically, on how to identify threats to value, establishing context. It was immensely useful, and really helped give structure to what I was doing, and let me think critically about my testing and how to improve it.<br />
<br />
But it still didn't scratch the itch I had.<br />
<br />
So I plucked up the courage to ask him a question after his talk.<br />
<br />
"I really enjoyed your talk, and it talked a lot about the kind of testing I'm doing, but, umm.... what if I want to learn how to do testing by the book?"<br />
<br />
"<b>Why would you want to do testing by the book?" </b>he boomed back in his American accent. <b>"The book is wrong!"</b><br />
<br />
<b>"I wrote a book about testing, and noone ever checked to see if I knew what I was talking about."</b><br />
<br />
He continued talking to me, right through the break time. The next session was about to start, so he started rummaging through his bag. He said "It looks like we've got to go..." He found what it was he was looking for. "But here, you should read this book. You can have it."<br />
<br />
<span id="goog_1487908938"></span><span id="goog_1487908939"></span><br />
<br />
<div style="text-align: left;">
</div>
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZE4PgwUS-RAw3kKbZkND2s-nxt98uvfHmVyFomEDHkYr5TNyXgUmC6D5DZDuUf63Du8X7kEULU62DP67nz3pf83POdfCh-V46Ij3nAK99jki61NimPE9croEilDt9TL4n5M6pr2aYpwU/s1600/IMG_20161002_144534.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZE4PgwUS-RAw3kKbZkND2s-nxt98uvfHmVyFomEDHkYr5TNyXgUmC6D5DZDuUf63Du8X7kEULU62DP67nz3pf83POdfCh-V46Ij3nAK99jki61NimPE9croEilDt9TL4n5M6pr2aYpwU/s320/IMG_20161002_144534.jpg" width="180" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">"Exploring Science" by David Klahr. The book James gave me as a faceless attendee at a conference in a remote part of the world</td></tr>
</tbody></table>
<br />
The words <i>"The book is wrong" </i>was the key that opened the software testing door to me. Being gifted a book as a nobody by a gargantuan in the industry was what pushed me through it.<br />
<br />
Since then, I've interacted with James countless times:<br />
<br />
I've engaged in coaching sessions with him, which he volunteers his time to give (http://www.satisfice.com/blog/archives/393).<br />
<br />
The first time I spent one-on-one time in person with him was when he volunteered his time to help Brian Osman set up and be content owner of the first Kiwi Workshop on Software Testing.<br />
<br />
The first time I collaborated with James was when he volunteered to co-write an article for<a href="http://www.testingtrapezemagazine.com/" target="_blank"> Testing Trapeze magazine</a>, and I offered to co-write with him.<br />
<br />
<br />
These were all 'nice' things for him to do. They came from a place of authenticity.<br />
<br />
James can be difficult to interact with, because he doesn't <i>pretend </i>to
be nice. I find him intimidating. I have to think carefully about what I
say around him. I have to be careful with my ideas. If I say something,
I have to defend and justify it. But that's ok. That's who James is, and that's the deal I make when I engage with him. Through this all, I have learnt courage. I have
learnt how to be my own biggest critic. Yes, I also suffered from baby
tiger syndrome, but I also learnt and grew from that too.<br />
<br />
In short, interacting with James is difficult, but worthwhile and I owe much of my career and success so far to James Bach.<br />
<br />
That's the anecdote to add to the pile.<br />
<br />
<h3>
A (small) Editorial</h3>
I don't think James is a bad person. I don't think James is a malicious person. I do think James is <i>unusual </i>in that he values integrity over manners; honesty over niceness.<br />
<br />
I said in the anecdote that I owe much of my career and success so far to James Bach. I think most of the people who would read this have benefited from the work James has done directly, or indirectly, consciously, or not. In fact, I can't imagine what the testing profession would look like if not for his influence on it. <br />
<br />
But James <b>is </b>double edged sword, and it frustrates me when I read about people being hurt by his blade, when they don't seem to deserve it. I don't know what to do about that - his sharpness has helped so many; his contribution to the industry immeasurable. But his sharpness has undeniably hurt some people that probably didn't deserve to get hurt. Dulling the blade or sheathing the sword means losing a lot.<br />
<br />
Embracing diversity means challenging what is 'usual' and working out how to work with '<i>unusual</i>'. If we strive to embrace diversity, then we should acknowledge diversity of personality, and diversity of temperaments, and work out how to work with that. Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com4tag:blogger.com,1999:blog-7250447306574801643.post-20080941801151320042016-02-05T11:21:00.001+13:002016-02-05T11:26:55.276+13:00Lean Testing in theory and practice<span id="docs-internal-guid-6609d7ca-ae4b-3a8b-b213-228d73c5f840"></span><br />
<div dir="ltr" style="margin-bottom: 2pt; margin-top: 2pt; text-align: left;">
<span id="docs-internal-guid-6609d7ca-ae4b-3a8b-b213-228d73c5f840"><span style="color: #243045; font-family: "georgia";"><span style="line-height: 38.4px; white-space: pre-wrap;"><i>This article was originally published in an earlier form on the <a href="http://bit.ly/1nK8WJS" target="_blank">Assurity Consulting website</a></i></span></span></span></div>
<span id="docs-internal-guid-6609d7ca-ae4b-3a8b-b213-228d73c5f840">
</span>
<div dir="ltr" style="line-height: 1.2; margin-bottom: 2pt; margin-top: 2pt; text-align: left;">
<span id="docs-internal-guid-6609d7ca-ae4b-3a8b-b213-228d73c5f840"><br /></span></div>
<span id="docs-internal-guid-6609d7ca-ae4b-3a8b-b213-228d73c5f840">
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">There are many different definitions of software testing, and many views on what responsible testing looks like in our industry. How you view the role of a tester informs what practices and artifacts you believe are valuable.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;"></span><br />
<a name='more'></a><span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">My preferred definition of software testing is from Cem Kaner: </span><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">“Testing is an empirical investigation conducted to provide stakeholders with information about the quality of the product or service under test.”</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Basically, I see testing as belonging in the information-gathering and delivery business.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Furthermore, I identify as a context-driven tester. These are the seven principles of the context-driven school and this is my interpretation of them and what they mean to me:</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">“The value of any practice depends on its context”</span><span style="vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">“There are good practices in context, but there are no best practices.”</span><span style="vertical-align: baseline; white-space: pre-wrap;"> Every testing problem is unique. How we approach each testing problem is therefore unique. So learning as much as possible and communicating your understanding of what you think the problem is, is vital to performing effective testing. My choice in tools reflects this belief. I like tools that are adaptable and give me freedom. The more adaptable and less rigid the tool or process is, the more contexts I can use it in.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">“People working together are the most important part of any project's context.”</span><span style="vertical-align: baseline; white-space: pre-wrap;">Knowledge exists in people's heads. It doesn't exist in the world in specifications or emails. Those are merely representations of knowledge. On a project, everyone has little pieces of the puzzle and no one person knows everything. Any practice that enhances social interaction – and any tool that helps me to communicate clearly what I think is going on – is a Good Thing™.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">“Projects unfold over time in ways that are often not predictable.”</span><span style="vertical-align: baseline; white-space: pre-wrap;"> Unfortunately, we can't schedule all our insights at the beginning of the project. Things change. At any given point in time, we must be willing to throw away everything we believed to be true and change tack. If we acknowledge that early on, then we can be vigilant about keeping the cost of changing and discarding to a minimum. </span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">“The product is a solution. If the problem isn't solved, the product doesn't work.”</span><span style="vertical-align: baseline; white-space: pre-wrap;"> This, to me, is all about really understanding the purpose of what we're testing and about seeing the whole. Software systems as a whole are greater than the sum of their parts. Every individual written requirement may be met, every user story may have passed a set of automated checks but, taken as a whole system, the software may completely fail to achieve what it was designed to do – or do it in less than ideal ways. Other emergent defects can occur when the software system is taken as a whole. Workflow inefficiencies, for example, often only become apparent when the entire system is viewed as whole.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">“Good software testing is a challenging intellectual process”</span><span style="vertical-align: baseline; white-space: pre-wrap;"> and </span><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">“Only through judgement and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.”</span><span style="vertical-align: baseline; white-space: pre-wrap;"> The last two principles state that good testing requires skill and judgement to adapt and learn. It sums up that, since things change and since each situation is unique, we can't just take a canned procedure, apply it to interchangeable human cogs on a factory line and expect good results.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">In conclusion, central to the context-driven mindset is the idea that software development projects are complex and that it is better to accept that, rather than try to control it away. When we're developing software, we're solving novel and unique problems. When things change, it means someone has learned something. Learning leads us to better outcomes, so we should embrace that.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">We should try to mold our practices to our environment, rather than try to mold our environment to our practices.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">So if the principles of Context-driven Testing describe the world in which we find ourselves and how we ought to think about that world, are there any principles that describe how we should operate in those environments?</span></div>
<h2 dir="ltr" style="line-height: 1.44; margin-bottom: 12pt; margin-top: 4pt; text-align: justify;">
<span style="font-family: "arial"; font-size: 21.3333px; vertical-align: baseline; white-space: pre-wrap;">Lean testing</span></h2>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Three or four years ago, I encountered the principles of lean software development by Mary and Tom Poppendieck based on the principles of lean manufacturing – and they really resonated with me. I've adapted them and applied them to testing and what they mean to me. If the principles of Context-driven Testing describe the world and how we view it, then I believe these principles of lean testing describe how we may respond to that world:</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Eliminate waste:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> I consider waste to be anything that doesn't add value or something that the cost of creation is hugely asymmetrical to the benefit derived. Software projects have a limited amount of time in which we have nearly an infinite number of things to test. It should be our goal to squeeze as much as we can from every minute we have. </span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Amplify learning:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> As mentioned, software projects are volatile environments where change is constant. Information comes to us non-linearly and different people on the team are exposed to different information at different times. When we learn or discover something new, we need to make this new information as visible as possible.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Regularly revise:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> We can't schedule our moments of inspiration or schedule when new ideas occur to us. We are constantly learning, as are others on the project. We need to be able to adjust our outputs and actions with the changing landscape and, as the desires of the stakeholders change, so too should our test strategy. Be willing to throw out everything I believed thus far.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Rapidly respond:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> Our role is to provide information to stakeholders to allow them to make quality-related decisions. Giving them information sooner allows them to make decisions sooner. Finding important bugs quickly allows them to be fixed in a timely manner. It aids with our credibility too. If someone taps us on the shoulder and asks "How did the testing go on the search module last week?", we want to be able to give them a satisfactory answer quickly.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Collaborate and communicate:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> We don't operate in a vacuum and I try to communicate what I'm doing and how I did it with anyone who's interested. This is essential for maintaining transparency and trust. Speaking of which...</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Maintain transparency and trust:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> Our greatest asset for getting things done and being able to do our jobs to the best of our ability, is trust. A lot of testing theatre and wasteful activities are undertaken in environments with little trust to avoid penalties or criticism. I also believe in meaningful reporting. A lot of the time, when someone is asking you for metrics or status reports in particular formats, it's the only way they really know how to ask "What's going on?". I try to keep everyone in the loop as much as possible. It also means that rather than giving a ‘pass / fail’, I try to pass on as much information as possible about how I tested it and why I think a bug may exist or not. This means that any errors in my thinking or reasoning can be picked up by others. An environment of trust allows me to be transparent and being transparent builds trust.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">See the whole:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> This is about staying mindful of the problem the software is trying to solve and for constantly questioning the context in which the product you're testing fits in with the values of the customers and other stakeholders.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">To sum up, I want to be able to give and receive as much information as I can in the limited amount of time I have and communicate it in a way that is respectful of others' time and resources. These are my values and what I think constitutes responsible testing. However, the two biggest threats to these values as I see it, are the prevalence of premature test scripting and the overuse of IEEE829-style documentation. I have seen the overuse of these two practices undermine good testing and good communication.</span></div>
<h2 dir="ltr" style="line-height: 1.44; margin-bottom: 12pt; margin-top: 4pt; text-align: justify;">
<span style="font-family: "arial"; font-size: 21.3333px; vertical-align: baseline; white-space: pre-wrap;">IEEE829-style documentation and premature test scripting</span></h2>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">When I talk about IEEE829-style documentation, I mean any style of documentation that is overly wordy, template-driven or ceremonial in nature.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">I'm not saying there isn't a time and place for these. Sometimes this style of documentation is necessary and useful, but the very beginning of a project when we know less than we will ever know again, when we have so much learning to do, is rarely the time or place. This style of documentation is a poor medium to communicate evolving understandings.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">At worst, this style of documentation is often template-based and acts as a ceremonial deliverable to satisfy some checkbox. </span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">But even at best, developing them requires a lot of time and effort and reading them and assimilating the information requires a lot of time and cognitive energy because any useful information is obfuscated within multi-thousand-word documents. It is difficult to extract the meaningful from the insignificant. How often do we see documents like this get written, signed off and then forgotten about on a shared drive, collecting virtual cobwebs?</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">To me, useful documentation is documentation that reflects my current understanding of the project in a format that makes it easy for me and others to retrieve that knowledge.</span><span style="vertical-align: baseline; white-space: pre-wrap;"><br class="kix-line-break" /></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">All in all, to crystallise our ideas so early when the cost of creation, editing and retrieval is so high and we still have so much to learn, makes little sense.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Below I offer an approach. This isn't necessarily the best approach for your context, or even a good approach for your context. It is just an example of an alternative approach to overly formalised and prescriptive approached that I have used.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">With regards to premature test scripting, I certainly favour a more exploratory approach to testing. My argument against scripting too early is:</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">1) At the beginning of a project, we know least about the project</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">2) Skilled exploratory testers perform hundreds of what could be considered discrete tests in a session. Few are worth repeating</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">3) Most tests performed are informed by the results of the previous test</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">4) The usefulness of a test is often not known until it has been performed. You don’t know if a rock is worth lifting until you’ve lifted it to see what’s underneath</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">5) Testers, when following a script, will deviate from the script</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">6) Different testers interpret scripted instructions differently resulting in differences between testers, even when following the same script</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Therefore:</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">7) Test scripts don’t tell you what you may think they are telling you</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Let's revisit the principles of lean testing I proposed and see how the two practices of test scripting and traditional heavy documentation creation stack up:</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Eliminate waste:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> Creating test scripts and large testing documents is a very labour-intensive job. Time spent writing documentation is time not spent testing. Investing large amounts of time and resources upfront on artifacts that are likely to become obsolete is wasteful.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Amplify learning:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> Because information comes to us non-linearly and at unscheduled times throughout a project's lifetime, it doesn't make sense to crystallise information early on in a format that is difficult to edit and difficult to retrieve information from. </span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Regularly revise:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> As mentioned, it's important to always be willing and able to change our course and throw away what we previously believed to be true. Test scripts and traditional test documents are very difficult to edit and, based on the time spent creating them, it can be very painful to throw them away and start again based on new information. The linear format of a document makes revision difficult and test scripts by their nature do not allow flexibility in their execution.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Rapidly respond:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> When someone asks us how our testing is going, what we intend to test and what we've done so far, we want to give a quick response. Retrieval of information should be rapid. Also, if an issue arises or a fix for a defect is delivered, it is important that we have the ability to retest quickly and report our results efficiently. When testing and we discover that the map and the territory differ, our ability to be flexible and respond to that is important.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Collaborate and communicate and maintain transparency and trust: Lengthy word documents and detailed test scripts obfuscate valuable information. Reading a document requires an investment of time and energy. This often means that our day-to-day activities are hidden and our mental model of the testing problem is hidden too. Traditional test scripting and reporting also hinders communication and transparency. If a manager asks me “What was the greatest issue you discovered in the last week?”, it is less valuable to say “Well, our pass to fail ratio is 89.5%” than to be able to quickly tell a story about the system under test, how you tested it and what you discovered about it. If I am to collaborate and communicate on a daily basis in a way that is rich and honest, I need a way to do so that requires little effort on the part of my audience.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">So where to next?</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">If it’s time to move on from premature test scripting and large cumbersome test documents, then how do we fill that gap? What do we do instead?</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Instead, let’s delve a little deeper into what’s being asked of us when others request test scripts and test documentation. Could it be that that’s the only artifact they know of that will deliver what it is they want? What do people want? People want to know what you did, what your coverage model is and what you intend to do next.</span></span></div>
</span><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">This approach is comprised of three elements: The Heuristic Test Strategy Model, aspects of Session-based Test Management, presented in a visual model using mind-mapping software. This ensures that testing is performed systematically and intelligently and be auditable, accountable and traceable.</span></div>
<h2 dir="ltr" style="line-height: 1.44; margin-bottom: 12pt; margin-top: 4pt; text-align: justify;">
<span style="font-family: "arial"; font-size: 21.3333px; vertical-align: baseline; white-space: pre-wrap;">Heuristic Test Strategy Model</span></h2>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">The Heuristic Test Strategy Model (HTSM) by James Bach is a set of patterns and heuristics designed to spark ideas around developing a test strategy. It is available to download for free from </span><a href="http://www.satisfice.com/tools/htsm.pdf" style="text-decoration: none;"><span style="font-weight: 700; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">James’ site</span></a><span style="vertical-align: baseline; white-space: pre-wrap;">. I was fortunate to discover the HTSM early on in my testing career and I can say without any doubt that it is the most useful tool I have ever used. I even carry a copy with me to client sites.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">For the purposes of creating a Visual Testing Coverage Model (VTCM), I want to investigate and map out all the different aspects of the product, as well as all the ways in which a stakeholder may value the product. For this reason, I use the </span><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Product Elements </span><span style="vertical-align: baseline; white-space: pre-wrap;">and </span><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">Quality Criteria </span><span style="vertical-align: baseline; white-space: pre-wrap;">categories for the VTCM.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Session-based Test Management</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Session-based Test Management (SBTM) is a way to structure exploratory testing and offers a framework for reporting and measuring testing activities.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">The important elements of SBTM are:</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Charter:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> This briefly describes the agenda or mission of the testing session.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Time-boxed session:</span><span style="vertical-align: baseline; white-space: pre-wrap;"> This is an uninterrupted period of time spent testing or pursuing the Charter. Session lengths are typically 60 to 120 minutes, but you can timebox in half or full-days at the expense of granularity of measurement.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Session Report: </span><span style="vertical-align: baseline; white-space: pre-wrap;">This reports what occurred during the test session and can include:</span></span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Charter or mission</span></div>
</li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Environment</span></div>
</li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Start and end times</span></div>
</li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Tester’s name</span></div>
</li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Notes on how testing was conducted</span></div>
</li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">List of any bugs found</span></div>
</li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">List of any issues encountered</span></div>
</li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Time spent off charter (called ‘opportunity’ in SBTM)</span></div>
</li>
<li dir="ltr" style="list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">If necessary, percentage of time spent testing, time spent investigating bugs, time spent reporting bugs and time spent setting up data and test environments.</span></div>
</li>
</ul>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">The kind of information recorded can, and should be, tailored for the project. If a standardised reporting format is used, the reports can be parsed to aggregate data that is deemed useful. There are also many tools that support SBTM.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<a href="http://www.satisfice.com/sbtm/sample_session.htm" style="text-decoration: none;"><span style="color: black; font-family: "arial" , "helvetica" , sans-serif; font-weight: 700; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">See here for an example of a test session report</span></a></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="vertical-align: baseline; white-space: pre-wrap;">The advantage of SBTM is that a tester has the freedom to apply their skills to actively investigate the product, while still being accountable and their work traceable. The advantage over pre-scripted testing is that the tester is reporting on what they </span><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">actually did</span><span style="vertical-align: baseline; white-space: pre-wrap;">, as opposed to describing what they </span><span style="font-style: italic; vertical-align: baseline; white-space: pre-wrap;">intended to do </span><span style="vertical-align: baseline; white-space: pre-wrap;">at some point in the past.</span></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">In practice, scripts are often obsolete by the time the software is delivered and, even under scripted conditions, testers will deviate from the script and two different testers may interpret the script two different ways. If important information is discovered under these deviations, this information is usually lost. SBTM allows rapid execution of tests and gives the tester the freedom to perform the next best action based on the information they have just learned.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Find out more about SBTM here:</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<a href="http://www.satisfice.com/sbtm/" style="text-decoration: none;"><span style="color: black; font-family: "arial" , "helvetica" , sans-serif; font-weight: 700; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://www.satisfice.com/sbtm/</span></a></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<a href="http://www.satisfice.com/articles/sbtm.pdf" style="text-decoration: none;"><span style="color: black; font-family: "arial" , "helvetica" , sans-serif; font-weight: 700; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://www.satisfice.com/articles/sbtm.pdf</span></a></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<a href="http://www.satisfice.com/articles/et-article.pdf" style="text-decoration: none;"><span style="color: black; font-family: "arial" , "helvetica" , sans-serif; font-weight: 700; text-decoration: underline; vertical-align: baseline; white-space: pre-wrap;">http://www.satisfice.com/articles/et-article.pdf</span></a></div>
<h2 dir="ltr" style="line-height: 1.44; margin-bottom: 12pt; margin-top: 4pt; text-align: justify;">
<span style="font-family: "arial"; font-size: 21.3333px; vertical-align: baseline; white-space: pre-wrap;">Mind-mapping software</span></h2>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">The testing process is not a nice, neat, linear process. Information comes to us non-linearly and often spontaneously. I need a way to capture this information as it arrives without being concerned about how to adjust it to a linear model (such as a test plan document written in MS Word). I also need a way to communicate non-linear, complex information in a way that is easy to understand and process. So far, I have found that mind-mapping software is the best-suited tool for these goals. Along the way, I have discovered a wonderful side effect of using mind-mapping software: unsolicited feedback... which I will go into later.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">So, how do we begin?</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">1. Start planning the framework</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">I like to begin by arranging the elements of the Structure and Quality criteria around a central node in the mind-mapping software. I then identify the properties I’m interested in, or may be interested in (after all, in these early days, who knows what may be important and the point is to stimulate your thinking in ways you may not have considered) and begin the framework for your map:</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;"><img alt="TPAH2 Test Project" height="470" src="https://lh4.googleusercontent.com/wZOOBXr1GMo6mL-dr4iQjV9z9DkSfxXRhbiPUjFsIEOdkozLrq-Zz7GhmHHW2DqrTQXmxCwO4Zrbw_hnaEuaKrkvutlUtvYKKG5oRIa9nxr6EOBxOaHaxVetg48VF6A1WMwF2cJymFOcpX0j" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="500" /></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">2. Start learning and collecting information</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Before setting sail for uncharted lands, we should gather as much information as we can. After all, we are in a state of ignorance about what we will encounter. There may be tales of great mountains, of bountiful oceans and of savage beasts. There may be scrolls written by soothsayers and there will be accounts written by previous travellers. </span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">In other words, there may be specification documents, responses to RFP’s, wiki pages, high-level descriptions, requirements documents. There may be project managers who can tell us great tales and business analysts with promises of bounty. We should begin mapping now so we have a frame of reference for our journey, but also fully expect that we will change it, add details, remove wrong details etc when we get in there. </span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">The act of mapping sparks new questions allowing you to ask more focused questions of the people around you. The map becomes the tool to make the map better. This is the beauty of an organic system. Give it some initial conditions (The Heuristic Test Strategy Model, for example) and it almost builds itself. Here is the same map after a conversation with a developer:</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;"><img alt="TPAH2 Test Project detailed" height="222" src="https://lh4.googleusercontent.com/feRbyX2kLyftzisAb6ZI4AfKSdW7fFIP9Tm0OyEjnOJd9HC7WDNm9VCyi8SvSdLLzFrPFABkrWV8dInpMv1ZrToWzuc0w1LizffuYgFHMh28Yhvcc0jt1LJkywkYuHKuo3AcsEFtARCF95cU" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="400" /></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">New parts have been added, we've learned a bit about the structure and the platforms and what is required to install it all. We were able to ask good questions because we could see where the silhouettes were that needed to be illuminated.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">3. Begin touring</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">We feel we have a useful framework to begin the actual work of touring the product. We know where the general inlets are and where the major landmarks are. We have enough to make landfall. Now we get the lay of the land. "Oh, here's a river we didn't know about. Better map that so we can take a closer look later." "This mountain is a surprise. I wonder if anyone knows about it".</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">As an example, as I installed something, it asked me for a port number [3306]. “Interesting,” I thought. “I’d better map that as a test idea”: </span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;"><img alt="TPAH2 cut out2" height="40" src="https://lh3.googleusercontent.com/V1YHBm5pmwUS2TK5t16na-fdybhLnwJmmLWdVoEJ2-vMeOPI_JKmIey4DxWOhRU9TwpJNjbkQts9TYkSFt42qTpUSLVL7CPkbWIWuakdCsGZwUtXv8LWf-SDG5TG9JhIqbcnpHa5ouMQ-7hc" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="400" /></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Eventually your map will be fleshed out and you’ll be confident that it is a pretty good and useful model of the software under test.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">4. Create test charters</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">With a little tweaking, you've almost automatically generated your test charters from your visual model and you can attach your test session reports directly to the model.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;"><img alt="TPAH2 Digg for desktop" height="419" src="https://lh3.googleusercontent.com/UN9psKmN6MIvSK2gX79Y911ndZKKzo3Mv6je2sj5p3bXzIEMNFoAbB_0vrDegMV-M5OzYz4zfdee8gdoyosh0uo5joNTcsGtUTsi68J8XWb2qACvpaijyISiWNV3L0etSS1Pnmzjwyqb3d1-" style="-webkit-transform: rotate(0.00rad); border: none; transform: rotate(0.00rad);" width="600" /></span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">5. Keep adjusting the map</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">Your map is a model. As your model changes, so will your map. Parts that you thought were important will be deleted and parts you didn't know existed will rise to prominence. The map is a reflection of YOUR understanding of the product at a given time.</span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">It also has other uses. You can use it to manage the testing process by assigning testers to a branch, letting them take ownership of all sub-branches. This lets you quickly assign work without having to go through requirements and specifications one-by-one and manually assigning individual tasks. You could colour branches that you're currently working on and shade in branches that have been completed. At a glance, you can see what has been done, what is being worked on and what is left to do. </span></div>
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial" , "helvetica" , sans-serif; vertical-align: baseline; white-space: pre-wrap;">"Auditable", "traceable", "reportable" need not be synonyms for "prescriptive" and "cumbersome". I have offered one way of making the test management process a bit leaner, and there will be others that may suit your particular context. </span></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 12px; vertical-align: baseline; white-space: pre-wrap;"> </span></div>
<br />
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial"; font-size: 12px; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span></div>
<br />
<br />
<br />
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial"; font-size: 12px; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span></div>
<br />
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial"; font-size: 12px; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span></div>
<br />
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<span style="font-family: "arial"; font-size: 12px; vertical-align: baseline; white-space: pre-wrap;"><span class="Apple-tab-span" style="white-space: pre;"> </span></span></div>
<br />
<div dir="ltr" style="line-height: 1.44; margin-bottom: 6pt; margin-top: 2pt; text-align: justify;">
<br /></div>
<div dir="ltr" style="margin-left: 0pt;">
<table style="border-collapse: collapse; border: none;"><colgroup><col width="170"></col><col width="170"></col><col width="170"></col><col width="180"></col></colgroup><tbody>
<tr style="height: 157px;"><td style="border-bottom: solid #000000 0px; border-left: solid #000000 0px; border-right: solid #000000 0px; border-top: solid #000000 0px; padding: 0px 0px 0px 0px; vertical-align: top;"><br /></td><td style="border-bottom: solid #000000 0px; border-left: solid #000000 0px; border-right: solid #000000 0px; border-top: solid #000000 0px; padding: 0px 0px 0px 0px; vertical-align: top;"><br /></td><td style="border-bottom: solid #000000 0px; border-left: solid #000000 0px; border-right: solid #000000 0px; border-top: solid #000000 0px; padding: 0px 0px 0px 0px; vertical-align: top;"><br /></td><td style="border-bottom: solid #000000 0px; border-left: solid #000000 0px; border-right: solid #000000 0px; border-top: solid #000000 0px; padding: 0px 8px 0px 8px; vertical-align: top;"><br /></td></tr>
</tbody></table>
</div>
</div>
Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com24tag:blogger.com,1999:blog-7250447306574801643.post-76901329725714363062015-06-06T16:43:00.001+12:002015-06-18T10:29:49.813+12:00Resources relating to "That's not the map I had in mind"<br />
<h2>
Resources for "That's not the map I had in mind"</h2>
I expect that these diagrams will evolve and grow over time which is why I have included them here for comment. This list will grow over time.<br />
<h4>
Xmind file for taxonomic hierarchy: <a href="https://drive.google.com/folderview?id=0B3U5XiI6J110fnNTcm5fTGNRZF9wQjE5R1lXQVJDaUNmTHpYQWszajQ4TDJQVGJ3YVcwTUU&usp=sharing" target="_blank">XMind file</a></h4>
<h4>
JPG file for taxonomic hierarchy: <a href="https://drive.google.com/file/d/0B3U5XiI6J110Zl9ZSGdnU3ZGUVk/view?usp=sharing" target="_blank">Image</a></h4>
<h4>
Downloadable examples of various testing models coming soon </h4>
Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com25tag:blogger.com,1999:blog-7250447306574801643.post-36639514809784205852014-12-02T16:29:00.000+13:002014-12-02T20:37:22.436+13:00WeTest Weekend Workshops 2014 theme: Evolve<br />
<br />
This last weekend (29/11/2014), we had our second WeTest Weekend Workshops 2014. The theme was "Evolve".<br />
<br />
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2V6bZTFxqmcgLdeRawzXUbvmw49ZaVbCxNDcVDEFvQtG2R5YmBuCIMqkusqDi3Z8UCPOn72msjApCgpLgAG3jEbJjPTv4_Qr3I8f0_QZ96jh-5j7oQxS8NgwtX6DhdqzeUPKX1woyWME/s1600/600_404619522.jpeg"><img border="0" height="233" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2V6bZTFxqmcgLdeRawzXUbvmw49ZaVbCxNDcVDEFvQtG2R5YmBuCIMqkusqDi3Z8UCPOn72msjApCgpLgAG3jEbJjPTv4_Qr3I8f0_QZ96jh-5j7oQxS8NgwtX6DhdqzeUPKX1woyWME/s1600/600_404619522.jpeg" width="400" /></a><br />
<br />
<br />
<br />
<a name='more'></a><br />
<br />
<br />
On reflection, it seems particularly apt with the resurgence in controversy over ISO29119 that occurred this year; a standard being, by definition, an attempt to prevent evolution; to hold a process static.<br />
<br />
Three ingredients are needed for evolution to occur: mutation, replication, and natural selection.<br />
<br />
A small mutation that gives an organism a survival advantage over those without the variation means those changes have more of a chance of being replicated, and over successive generations, more of the population will carry the new variation.<br />
<br />
Mutations in biological systems are those little errors in copying, and spontaneous changes that happen within the strands of DNA that define life.<br />
<br />
Mutations in our field are those little experiments we do; when we tinker with a process, when we try something new, and discover a different way of thinking or of doing.<br />
<br />
We knew these mutations were happening in organisations, because we’d hear about them when we talked with people, we observed people in their organisations, and we see all these little innovations or the opportunities for them. We met enthusiastic, smart people with fresh approaches, and alternative views.<br />
<br />
And we even started seeing similar variations created independently in multiple locations. People performing exploratory testing in one organisation without knowing what was happening in another. People encountering the same issues and problems, experiencing the same insecurities, and thinking they were the only ones. The problem was isolation.<br />
<br />
The next ingredient needed for evolution to really happen is replication.<br />
<br />
I'm sure you're all familiar with how replication occurs in nature! Replication of ideas in our field occurs when you have a conversation, or read a book, or attend a conference, and take those ideas away and act on them. The good ideas survive and continue to be mutated, and replicated, and the bad ones die out.<br />
<br />
In theory.<br />
<br />
The replication aspect was - and is - broken. And it is broken because our main replication channels - conferences, text books, education courses - are usually monopolised by the same old voices; espousing the same old models, and repeating the same old folk-lore. Talking about maturity models, and best practices, and metrics, and test cases. The actually interesting stories from the front lines weren't being heard; weren't being replicated. And that was one of the main motivations for Katrina and me to start the Wellington Testers Workshop (WeTest) .<br />
<br />
As an aside, natural selection is also broken in many organisations. Broken by the artificial selection of mandated procedures in both waterfall and Agile methodologies. By insistence on unquestioningly following The Process, you are suffocating a natural robust process of evolution by preventing mutation, and preventing the natural selection of practices by the practitioners themselves!<br />
<br />
Best practices, maturity models, standard templates, ISO29119, test cases, test scripts, all that old testing folk-lore; all those dinosaurs can be made obsolete by evolution.<br />
<br />
So everyone, let's do our part for evolution right now, and let’s start replicating those stories, adapting them, altering them, keeping the ones that work, and continuing to spread the stories of survival far and wide.<br />
----------------------------------------------------<br />
<br />
ps: <i>Once you start looking, the evolution of ideas is everywhere. Take the WeTest conference format, itself (apologies for any facts I get wrong. Please let me know of any historical inaccuracies):<br /><br />It's lineage can be directly traced back to Los Altos, California, February 1997, where the first Los Altos Workshop on Software Testing (LAWST) occurred. Of course, the LAWST format had its own origins, and I'd love to dive into that more, but as far as I can tell, Cem Kaner and Jerry Weinberg had a lot of influence on the initial format.</i><br />
<br />
<i>October 1998: Division between clarifying question stage and open season stage instituted after fiery LAWST 5, when Doug Hoffman could not get through his ER due to constant objections by James Bach. It was also recognized after LAWST 5 that ER's were better than normal idea talks at preventing terrible arguments.</i><br />
<br />
<i>On October 2, 2003 the first WOPR conference began (Workshop on Performance and Reliability) which had a format that looks to be directly influenced by the LAWST format (Focus on experience reports, followed by facilitated discussion. Participation is expected etc). James Bach facilitated WOPRs 1 & 2.<br /><br />September 8, 2004, Paul Holland facilitates WOPR 3. Ross Collard asks Paul to facilitate WOPR 4.<br /><br />September 8, 2005, at WOPR 5, K-Cards made their first appearance to replace hand signals to communicate with the facilitator. (fascinating story about the evolution of K-Cards here: http://testingthoughts.com/blog/26)<br /><br />June, 2010, <a href="https://twitter.com/bjosman">Brian Osman</a> asks James Bach how to build community in New Zealand....<br /><br />June 24th, 2011, the LAWST-style format hits New Zealand soil for the first time with the first Kiwi Workshop on Software Testing (KWST 1) which I attended.<br /><br />June 2012, Both <a href="https://twitter.com/katrina_tester" target="_blank">Katrina Clokie</a> and I attend KWST 2.<br /><br />Sometime later, in 2012, Katrina, Brian, and I meet in the foodcourt at the bottom of the BNZ centre in Wellington and work out a way to make the KWST format accessible to a wider group, and happen more often. WeTest is born with the values of openness, community, diversity, and accessibility at its core. The focus was on hearing stories by practitioners about what did, and didn't work for them. All presenters must have 'skin in the game'.</i><br />
<i><br /><b>October 25th, 2012: Katrina and I host the very first WeTest Workshop in a back meeting room at St Andrews on the Terrace, Wellington.</b></i><br />
<br />
<i><b>September 26, 2013: First Auckland WeTest Workshop is held </b></i><br />
<br />
<i><b>November 30th, 2013: First WeTest Weekend Workshop is held with 50 registered attendees.</b></i> <i>Includes a mix of LAWST-style facilitated ERs, and hands-on practical workshops</i>.<br />
<br />
<i><b>November 29th, 2014: Second WeTest Weekend Workshop is held with 85 registered attendees.</b></i><br />
<br />
<i><b>2015: </b>Watch this space...especially if you live in Christchurch.</i><br />
<i><b> </b></i><br />
<i>pps: The DNA of LAWST can also be found at CAST conferences, Let's Test conferences, PEST in Estonia, SWET in Sweden, as well as WOPR, WHET, WTST & DEWT.<b><br /></b></i><br />
<h1 itemprop="name" style="background-color: white; margin: 0px; padding: 0px 0px 9px;">
<div style="border-bottom-color: windowtext; border-bottom-width: 1pt; border-style: none none solid; padding: 0cm 0cm 1pt;">
<div class="MsoNormal" style="border: none; margin-bottom: 0.0001pt; padding: 0cm;">
<br /></div>
</div>
<div style="color: rgba(0, 0, 0, 0.952941); font-family: Whitney, helvetica, arial, sans-serif; font-size: 38px; letter-spacing: -0.75px; line-height: 1.1;">
<!--[endif]--></div>
</h1>
<!-- Blogger automated replacement: "https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2V6bZTFxqmcgLdeRawzXUbvmw49ZaVbCxNDcVDEFvQtG2R5YmBuCIMqkusqDi3Z8UCPOn72msjApCgpLgAG3jEbJjPTv4_Qr3I8f0_QZ96jh-5j7oQxS8NgwtX6DhdqzeUPKX1woyWME/s1600/600_404619522.jpeg" with "https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2V6bZTFxqmcgLdeRawzXUbvmw49ZaVbCxNDcVDEFvQtG2R5YmBuCIMqkusqDi3Z8UCPOn72msjApCgpLgAG3jEbJjPTv4_Qr3I8f0_QZ96jh-5j7oQxS8NgwtX6DhdqzeUPKX1woyWME/s1600/600_404619522.jpeg" --><!-- Blogger automated replacement: "https://images-blogger-opensocial.googleusercontent.com/gadgets/proxy?url=http%3A%2F%2F4.bp.blogspot.com%2F--Q25TwFcm-8%2FVH0jfPzQcaI%2FAAAAAAAAAkk%2FZ_jND6wMBR0%2Fs1600%2F600_404619522.jpeg&container=blogger&gadget=a&rewriteMime=image%2F*" with "https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2V6bZTFxqmcgLdeRawzXUbvmw49ZaVbCxNDcVDEFvQtG2R5YmBuCIMqkusqDi3Z8UCPOn72msjApCgpLgAG3jEbJjPTv4_Qr3I8f0_QZ96jh-5j7oQxS8NgwtX6DhdqzeUPKX1woyWME/s1600/600_404619522.jpeg" -->Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com9tag:blogger.com,1999:blog-7250447306574801643.post-4622738296692791572014-11-05T21:34:00.003+13:002014-11-06T09:53:40.766+13:00Shallow KPIs: A Tale of Two TestersOnce upon a time there was a thread on linked in on KPIs for software testers. A Test Manager shared the KPIs she uses for her team:<br />
<br />
<i>1. Amount of bugs created.
<br />
2. Amount of bugs verified
<br />
3. Amount of assigned work completed.
<br />
4. Confirm to schedule. </i><br />
<br />
(At the risk of accusing anyone on Linkedin of being sloppy with their language, I will assume by 'Amount of bugs created' she means "number of bugs logged in some bug tracking tool").<br />
<br />
When challenged, she provided the following 'real life' scenario as if it the sheer power of this example would dazzle us all into submission:<br />
<br />
<blockquote class="tr_bq">
<i>"Tester1 – found 30 defects, verified all assigned issues by deadline.
<br />
Tester2- found 0 defects, verified 10% of issues assigned by deadline.
<br />
Who performed better Tester1 or Tester2?"</i></blockquote>
<br />
So who performed better?<br />
<br />
Tester 2 of course. She didn't log any defects because she had established a
strong working relationship with the development team, and as she found
an issue, she wrote it on a sticky note, and gave it to the developer.
The developer then would rapidly fix and redeploy. The tester would
retest, and verify the fix. Because of this, she was able to reduce a
lot of administrative overhead, and help the developers produce a high
quality product. <br />
Tester 2 was unable to verify all the issues assigned to her by the
deadline because she was very thorough, and felt that meeting an
arbitrary deadline didn't contribute to the overall health of the
project. Instead, she focused on doing great work. <br />
<br />
Meanwhile, Tester1 logged many defects. They were poorly written, and
many of them were just different symptoms of the same underlying issue.
The developers had to spend a lot of time trying to decipher them, and
would often spend many hours chasing down bugs that turned out to be
merely configuration errors. Once, he logged 10 'defects' that were
immediately 'fixed' when someone came over and updated his java
environment. A lot of time is spend administering the defects in the bug
tracking tool, and trying to work out if Tester 1's defects are
legitimate or not. <br />
Tester 1 works very hard to meet the deadline when verifying issues. To
do so, he performs a very shallow confirmatory check. His vulnerability
to confirmation bias has led him to verify many fixes as "complete" when
there were regressive side-effects he didn't pick up on. <br />
<br />
Tester 1 meets his KPIs and is up for promotion. In two years he'll be sharing his wisdom in Linkedin.<br />
<br />
Tester 2 has been told she isn't performing as
necessary. She is going home tonight to update her resume. In a year she'll be working at a company that assesses her performance by watching her work and regularly catching up for peer review. In two years she'll be sharing her wisdom at a peer conference.<br />
Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com10tag:blogger.com,1999:blog-7250447306574801643.post-47496271769245417502014-09-19T15:10:00.000+12:002014-09-19T15:10:50.871+12:00The Responsibilities of a Conference FacilitatorI have just returned from Let's Test Oz 2014, and like the CAST conferences, operates on a <a href="http://testingthoughts.com/blog/26" target="_blank">K-Card style facilitation format.</a><br />
<br />
During the three day conference I saw the power a great facilitator can have. I got to experience first-hand the influence a good facilitator can have on the success of a talk, so I would like to offer my perspectives on what makes a good faciliator.<br />
<br />
<a name='more'></a><h2>
</h2>
<h2>
Serves the Speaker</h2>
The facilitator serves the speaker. A lot of speakers are nervous or otherwise distracted just before they are due to deliver a presentation. The facilitator can aid the speaker by checking that the room has been set up appropriately, the speaker has water, and that the A/V equipment is set up to the speaker's liking. Yes, most speakers can take of this themselves, and I would expect them to, but it eases the burden to know there is someone to support you if you need a runner to find a technician while you're booting up your laptop, for example. A speaker, especially an inexperienced one, can have so much running through their minds pre-presentation that it is a great comfort to know that someone else is taking care of the details allowing you to focus your mind on what's coming up.<br />
<br />
Some things a facilitator could do to help a non-experienced speaker:<br />
<ul>
<li>Ensure they have water, notepads, pens etc on their podium or table if they want it</li>
<li>Ensure the room is arranged how the presenter would like it, including equipment such as flipcharts, whiteboards, etc</li>
<li>Plug the A/V equipment in, and check the projector is working</li>
<li>Just be there as a support person to help with the nerves!</li>
</ul>
<h2>
</h2>
<h2>
Sets the Tone</h2>
The first words of a presentation don't come from the speaker, but from the facilitator when they introduce the speaker. YES, this is part of the presentation, therefore, you have a responsibility to your speaker to make it count. Interview your speaker beforehand to ask them how they would like to be introduced. Your introduction is a mini-speech! It should be short, sharp and to the point. But it should also set the tone for the speaker and get the audience warmed up. It is much easier for a speaker to begin with a warm room, than a cold one.<br />
<h2>
</h2>
<h2>
Is a Time-Keeper</h2>
The facilitator is a time keeper. I recommend asking the speaker at what time stamps they wish to be notified and whether they want to be alerted to "Time-elapsed" or "Time-to-go". Have a clear protocol on how you will communicate time to your speaker.<br />
<h2>
</h2>
<h2>
Leads the applause</h2>
After the introduction, lead the applause. The audience will take your cue and follow. They will. Trust me. If you do not lead the applause, the audience may kind of awkwardly look around, not sure if they should clap or not. Don't leave it ambiguous.<br />
<br />
At the conclusion of the talk, lead the applause again. <br />
<h2>
</h2>
<h2>
Conducts Open Season </h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="http://kennysilva.net/wp-content/uploads/2010/12/orchestra-conductor.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://kennysilva.net/wp-content/uploads/2010/12/orchestra-conductor.jpg" height="266" width="320" /></a></div>
The facilitator is the conductor of open season, and, therefore, controls the rhythm and energy of open season. Don't leave the speaker hanging. Be responsive at the end of the talk and take control of the room. I have seen the speaker stop, then the facilitator, with his head down in his tablet say "hold on...". Please don't do that. Your speaker has probably worked hard to create a buzz, and you are now the custodian of that energy. Don't drop it.<br />
<br />
A good conductor knows how to control the dynamics of a room. Knows when to crescendo, when to ease back. There are <a href="http://markwaite.blogspot.co.nz/2009/07/cast-2009-red-yellow-green-facilitation.html" target="_blank">great</a> <a href="http://testingthoughts.com/blog/28" target="_blank">posts</a> on how to handle the mechanics of facilitation. Remember, you are facilitating your speaker too. I had great rapport with my facilitator at Let's Test Oz, and picked up on a few subtle cues on when to stop addressing a question and let him move onto another yellow card.<br />
<br />
<h2>
In Conclusion</h2>
As a facilitator, you contribute to the success of your speaker by helping them set up, handing them an energised audience, and keeping the energy alive during the open season. This is a responsible, but fun and rewarding role to have.<br />
<h2>
</h2>
Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com2tag:blogger.com,1999:blog-7250447306574801643.post-84682391083173501242014-08-21T16:45:00.003+12:002014-08-21T16:45:45.011+12:00Very Short Blog Post: A date with test cases.<div class="kl" dir="ltr" id=":1pa">
Here's a test case problem:<br />
<br />
<h4>
The requirement:</h4>
<i>"Formatting is automatically applied to all date fields (dd/mm/yy formatted)"</i></div>
<div class="kl" dir="ltr" id=":1ma">
</div>
<div class="kl" dir="ltr" id=":1ma">
Here are my findings after a 15 minute test session:</div>
<div class="kl" dir="ltr" id=":1m9">
<ul>
<li>Formatting is automatically applied when entering dates as </li>
</ul>
<ul style="margin-left: 40px;">
<li>12.12.2014 </li>
<li>12th Dec 2014</li>
<li>12 Dec 2014</li>
<li>12 December 2014</li>
<li>12th December 2014</li>
<li>12-12-2014</li>
<li>12-DEC-2014</li>
</ul>
<ul>
<li> Formatting is not applied to:</li>
</ul>
<ul style="margin-left: 40px;">
<li>12.12.14</li>
<li>12122014</li>
<li>12th Dec 14</li>
<li>12th December 14</li>
<li>12/12/14</li>
</ul>
</div>
<div class="km" role="chatMessage">
<div class="kk">
<span dir="ltr" id=":1nm"><br /></span></div>
</div>
<div class="km" role="chatMessage">
<div class="kk">
<span dir="ltr" id=":1o2">a) Did the requirement 'pass'?</span></div>
<div class="kl" dir="ltr" id=":1nh">
b) According to some claims, it is best practice to write one positive test case and one negative test case per requirement. What would I have learned by writing and executing two test cases?</div>
<div class="kl" dir="ltr" id=":1nd">
c) Some test management tools would report 100% coverage with 1 test case and if it passed, it would say that the requirement passed.</div>
<div class="kl" dir="ltr" id=":1nd">
</div>
<div class="kl" dir="ltr" id=":1nd">
Maybe talking about testing in terms of test cases and of passes and fails isn't useful.</div>
<div class="kl" dir="ltr" id=":1nd">
<br /></div>
<div class="kl" dir="ltr" id=":1nd">
</div>
<div class="kl" dir="ltr" id=":1nd">
</div>
<div class="kl" dir="ltr" id=":1nd">
</div>
</div>
Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com9tag:blogger.com,1999:blog-7250447306574801643.post-89728207715417938942014-04-06T14:42:00.000+12:002014-04-06T17:30:48.521+12:00“Anyone Can Be a Tester” - ResponseI was pointed to this article on twitter: http://www.morethanfunctional.co.uk/1/post/2014/04/anyone-can-be-a-tester.html by Jari Laakso who has already written a response <a href="http://jarilaakso.blogspot.ro/2014/04/anyone-can-be-bad-tester.html" target="_blank">here</a>.<br />
<br />
There are some things I agree with, some things I disagree with, and some things I think are just ugly.<br />
<br />
<br />
<a name='more'></a><br />
<h3>
Stuff I agree with</h3>
<blockquote class="tr_bq">
"<i>I recommend you get as many non-testers to test your software as possible</i>."</blockquote>
I completely agree with the sentiment here. Every person who interacts with your product brings their own history, their own quality criteria, and their own biases. As trained professional testers we try to be conscious of our biases but they cannot be eliminated completely. And over time, we become familiar with the interfaces, consciously or unconsciously work around quirky behaviour, and become blind to the familiar. Getting different perspectives can be a very Good Thing, and will augment your testing effort.<br />
Augment.<br />
Rarely replace.<br />
And bear in mind that somebody still needs to design the testing sessions, collate that information, and present it in a meaningful way.<br />
<br />
<blockquote class="tr_bq">
"<i>You can use as many testing techniques and tools as you want but if it isn't very usable, the software's crap.</i>" </blockquote>
Again, I agree completely...in principle. Bearing in mind that what is useful for one user may be completely unusable for another. So, assuming you mean "...if it isn't very usable for the people who will be using it, the software's crap." Collecting usability and user experience data from an audience of "developers, product managers, directors and whoever else we could find" may be of limited use if the target user group are power plant engineers, the elderly, or toddlers, for example. <br />
<br />
<blockquote class="tr_bq">
"<i>I was shocked at how usability and user experience weren't mentioned once</i>."</blockquote>
I guess your 'thing' can be to bring usability and user experience into common testing discourse, then. That's a positive opportunity.<br />
<br />
<h3>
Stuff I disagree with:</h3>
<blockquote class="tr_bq">
"<i>anyone can test software</i>"</blockquote>
Whenever someone scratches their head and tries to work out how to indent their bullet list in MS Word without indenting the paragraphs around it, they are testing. Whenever someone is trying to work out what a menu option does, they are testing. When someone downloads an app and is assessing whether it meets their needs and desires, they are testing. So yes, to that extent, anyone can test software.<br />
<br />
However...<br />
<br />
...I don't think that's what you meant. While anyone can interact with software and make some meaningful observations about it, a professional tester does it methodically, precisely, and within the overall mission of testing. That means making judgements about what to test and how to test it based based on knowledge of the stakeholders and what they value, <a href="http://www.satisfice.com/tools/htsm.pdf" target="_blank">and a whole bunch of other context-related criteria</a> while being mindful about responsible use of time and money.<br />
<br />
Yes, anyone can wander around a product and bump into the odd bug, some of which may be valuable. But not anyone can actively investigate a product in a methodical way and be able to justify what they will test and how, and what they will not test, to discover threats to value and present it in a meaningful way to stakeholders, while working within project constraints.<br />
<br />
Most people probably have the capacity to learn testing in this way, but not everybody has learnt testing in this way.<br />
<br />
<h3>
The ugly</h3>
<blockquote class="tr_bq">
"<i>when I've heard them say it, there are normally other testers nodding
their head or concurring in a "yeah, everyone who isn't a tester is a
dick" kind of way.</i>"</blockquote>
OK, that's your <a href="http://en.wikipedia.org/wiki/Psychological_projection" target="_blank">projection</a>, not mine. <br />
<blockquote class="tr_bq">
"<i>After filtering out non-issues and duplicates (one developer logged a
bug that had been assigned to him for over a month) there were over 100
bugs logged. </i></blockquote>
<blockquote class="tr_bq">
<i>Comfortingly for me and my boss, there were no massive issues logged.</i>"</blockquote>
I consider a professional software tester to be able to critically analyse those statements, and be able to demonstrate how the statements are meaningless out of context, and may even be used against your argument. Not everyone can do that.<br />
<br />
I kindly suggest that if you don't want to alienate others, and you want people to take your message seriously, perhaps you should rethink statements such as these:<br />
<blockquote class="tr_bq">
"<i>you might as well take your test sessions, your mindmaps and your
automation scripts then insert them promptly and firmly up your arse.</i>"</blockquote>
This makes you sound like a dick. Maybe you want to sound like a dick, I don't know, but in case you don't, I'm letting you know.<br />
<blockquote class="tr_bq">
"<i>...and using pretentious terms like 'test charter' (that means a test
objective or mission for anyone who doesn't attend the church of James
Bach)."</i></blockquote>
<blockquote class="tr_bq">
<i>"However, software testing recently seems to be all about using the
fashionable terms and techniques instead of doing what a tester is
supposed to do</i>" </blockquote>
I'm sorry you haven't heard of a term that's been around for over <a href="http://www.satisfice.com/articles/sbtm.pdf" target="_blank">thirteen years</a>. But that doesn't make it pretentious. But you're not alone in not hearing about these 'fashionable' terms. Which is exactly why the community appears to harp on about them. Because those of us who use them have <a href="http://katrinatester.blogspot.co.nz/2014/03/reporting-session-based-testing.html" target="_blank">discovered they work really well, and we want to share what we've learnt.</a><br />
<h3>
In Conclusion</h3>
I like your message that "we should be asking as many people as possible to test the software we're working on and listening to their feedback." I agree that there can be too much emphasis on the 'functional' aspect of software testing. But that message is completely obliterated by all the other stuff in there.<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com2tag:blogger.com,1999:blog-7250447306574801643.post-45109231581886184212013-12-21T21:15:00.001+13:002013-12-21T21:16:55.464+13:00Thoughts around Test estimation<br />
The irony of being asked to provide an estimate as to how long testing is going to take is that, not only is this like asking us how long it's going to take us to find all the easter eggs at easter, but often, it's not really up to us to decide when to stop.<br />
<a name='more'></a>Testing is the empirical technical investigation performed to allow stakeholders to make important quality-related decisions (Cem Kaner). Therefore, the question of "how long will testing take?" is really the question, "How long is it going to take for a decision-maker to have enough information to inform a risk-based decision?"<br />
<br />
Therefore, the first estimation question is: "How much information do we need to gather to inform a risk-based decision within the context of this project?"<br />
<br />
The second estimation question is: "How long will it take to gather this information and communicate it?" This answer of course depends on the product, the complexity of the product, the tester's understanding of the product, other stakeholders' understanding of the product, access to oracles, presence of obstacles to testing. So a lot of the key factors that impact the time it takes to test are completely outside the control of the tester. And the amount of information required to gather is more of a function of the decision-maker, than of the tester. Yet testers are asked to provide this information.<br />
<h3>
So how have I handled the testimation problem?</h3>
Here is one way I've done estimation in the past:<br />
Considering the multiple estimates involved, and the multiple factors outside of my control when it comes to how long the testing task is going to take, I make clear that I'm giving a 'ball-park' estimate, and should be treated as a heuristic for determining potential resources required which should be continually analysed and adjusted. Therefore, it's important for me to be as transparent as possible lest a manager is mislead by my giving him a number and he assumes it's a scientifically-derived answer.<br />
<h4>
Key Factors:</h4>
<ul>
<li>Visual Testing Coverage Map (VTCM): A visual representation of our model of test coverage (more info <a href="http://testerkiwi.blogspot.co.nz/2011/04/building-exploratory-test-plan-redux.html" target="_blank">here</a> & <a href="http://www.assurity.co.nz/community/our-thoughts/part-1-aaron-hodder-on-using-mind-mapping-software-as-a-visual-test-management-tool/" target="_blank">here</a> & <a href="http://www.inspiredtester.com/1/post/2013/04/visual-test-models.html" target="_blank">here.)</a> </li>
<li>Test Coverage Areas: Functions or features of the product, or test activities we could dedicate testing time to cover, based on our model of test coverage.</li>
<li>Test Session: A time-boxed uninterrupted test session dedicated to covering test coverage areas.</li>
</ul>
Assuming that I use test sessions of 90 minutes:<br />
<ol>
<li><b>The VTCM is divided into test sessions;</b> that is, the test coverage areas that I believe could be covered in an ideal 90 minute uninterrupted testing session.</li>
<li><b>I will assume that a tester can complete three test sessions per day.</b> Time taken to investigate difficult-to-reproduce bugs, bug reporting, and setting up environments are not included as these are very variable and impossible to predict. Emails, meetings, learning activities, and other interruptions are also not included, hence the estimate of three test sessions per day.</li>
</ol>
Of course, these are the kinds of implicit assumptions (I try to make them explicit in any testimation document) that we make:<br />
<ul>
<li>The product is well understood by the people around us and we have access to their knowledge in some form (documents, conversations etc)</li>
<li>The test environment meaningfully reflects the production environment</li>
<li>Login details, account names, passwords, IP addresses, URLs, FTP details, Database details and any other information needed to access and test the product are available to the tester.</li>
<li>The tester has sufficient permissions within their environment and within the application so as to be able to perform testing to a satisfactory level. (this usually means admin access to the application and to your machine)</li>
<li>Where good testing relies on production or production-like data to be present, then that data will be present, or the ability to create and populate that data will be available (at a level I can perform).</li>
<li>Communication between testers, PMs, developers, BAs and other project team members will be unimpeded by geographical, political, or other constraints.</li>
</ul>
I'm keen to hear what other tacit assumptions we make that I haven't thought of.<br />
<h4>
Example:</h4>
This is a VTCM for Digg for Desktop which I created in a recent <a href="http://weekendtesting.com/archives/3114" target="_blank">Weekend Testers ANZ exercise:</a><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQIftjCHKoUr4ZhM6VKgQLpAsMS-KTzQkQEHKyzVncRLZ0jmdHqiwn40Q65tUtXjcDVuyy9pI0DJjYlYmIXs-PH2efBjXg947_MyHgVJvYr7GtcqwpaaCHrSE2GlIrSvsJCDnbkiZo7xE/s1600/Digg+VTCM.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="278" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQIftjCHKoUr4ZhM6VKgQLpAsMS-KTzQkQEHKyzVncRLZ0jmdHqiwn40Q65tUtXjcDVuyy9pI0DJjYlYmIXs-PH2efBjXg947_MyHgVJvYr7GtcqwpaaCHrSE2GlIrSvsJCDnbkiZo7xE/s400/Digg+VTCM.JPG" width="400" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
(view full size here: http://i.imgur.com/6zW6RZp.jpg)<br />
Each shaded area represents the Test Coverage Areas of a 90 minute session. Therefore, according to this model, each shaded area is an <b>equivalent </b>amount of work. There are 12 sessions. 12/3 = 4 therefore I estimate it will take four days to cover this model....<br />
<br />
<br />
...which we should immediately distrust.<br />
<br />
However, we now have a benchmark that we can use to monitor how our testing is tracking based on the assumptions we have made. Where reality and our estimate differ, we can adjust our estimate on the fly.<br />
<br />
I'm making no claim that this method is any more accurate (if there can be such a possibility) than any other method. But what it does provide is <b>visibility.</b> My experience with using this method is that I can show someone that VTCM and they can say "you're dreaming if you think it's going to take only 90 minutes to cover X" My estimate, <i>and the model I used to derive that estimate </i>is all contained together.<br />
<br />
"How did you come up with four days?" "Let me show you" I can reply. People are then able to critique my estimate because the testing coverage areas, AND my model of the product are available and visible for anyone to see. And when we start testing, we should hopefully begin seeing right away whether there's something systemically wrong with the estimate. Turns out there's more bugs than anticipated and I'm only getting through two sesions a day? We can adjust. I have a map that stakeholders can use to select which areas they may want to descope, since my model contains areas I intend to test <i>and areas I don't intend to test. </i>It's kind of like a menu, with prices next to them. A stakeholder can then select what items on the menu they'd like, and if they have a fixed budget, be able to select the best course for them.<br />
<br />Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com19tag:blogger.com,1999:blog-7250447306574801643.post-38897312597650707572011-07-03T22:19:00.000+12:002014-08-21T10:58:59.740+12:00ISTQB: Possum CertificationIn my <a href="http://testerkiwi.blogspot.com/2011/06/kwst-possum-testing.html">last post</a>, I talked about the concept of possum testing: Doing testing-related activities that the tester does not value, motivated on some level by fear. I'd like to extend this concept out, and talk about the fundamental problem I have with ISTQB certification: It's a possum certification.<br />
<br />
<a name='more'></a>If possum testing is testing that the tester does not value, motivated on some level by fear, then possum certification is the acquisition of certification that the receiver does not value, and the attainment of that certification is motivated at some level by fear.<br />
<br />
ISTQB rely on deceiving their customers that what they will be getting is a valid qualification. They have successfully created a vicious cycle where employers believe that ISTQB certification is somehow some kind of valid measure of a tester's skill so they ask for it in their job ads. Prospective employees see it in the job ads, and therefore think it must be a valid qualification, after all, look at all these companies asking for it, so they go out and get it. Employers see employees with it on their CV, and thus confirm in their minds that ISTQB certification is a valid certification, after all, look at all these applicants with it on their CVs. And on the cycle goes. Meanwhile, ISTQB do nothing to correct the situation. But why would they?<br />
<br />
I can only conclude that the ISTQB is deliberately exploiting the fearful, at the point in their careers when they are the most vulnerable. Instead of helping possums cross the road safely, they are creating a deception in the industry. The name itself "International Software Testing Qualifications Board" has been deliberately constructed to dazzle employers into thinking it is somehow an official industry board. They are taking our craft, and diluting it into a three day dictionary definition course, and passing it off as a legitimate qualification. They are stifling innovation and critical thinking by indoctrinating new recruits into the field with "best practices" and definitions that don't even hold up to a moment's critical analysis.<br />
<br />
They are making money off the fear of new testers, who fear that they aren't employable without certification, and the ignorance of employers who don't understand the field. They perpetuate the ignorance by giving themselves an official sounding name that implies universal acceptance and authority.<br />
<br />
To me, it is wrong for ISTQB to be intentionally misleading employers, and exploiting the fear of newbie possums. If you feel the same way then speak out, and stop letting this ruining our profession. Stop the spread of folklore and myth that the ISTQB syllabus teaches, and perhaps we can take back our profession from those who seek to only profit from it, rather than study it.<br />
<br />
<u><b>EDIT 21 August 2014</b></u><br />
Karen Johnson has written an eloquent piece about certification <a href="http://karennicolejohnson.com/2014/08/my-thoughts-on-testing-certifications/" target="_blank">here</a>. If you feel the same way, then I urge you to sign the Professional Tester's Manifesto is available here: <a href="http://www.professionaltestersmanifesto.org/">http://www.professionaltestersmanifesto.org/</a>Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com30tag:blogger.com,1999:blog-7250447306574801643.post-72346478756672550122011-06-28T22:09:00.003+12:002011-06-30T10:51:49.502+12:00KWST & Possum Testing<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="http://irishchicklette.files.wordpress.com/2009/02/possum-on-the-road.jpg?w=300&h=269" imageanchor="1" style="clear: right; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="http://irishchicklette.files.wordpress.com/2009/02/possum-on-the-road.jpg?w=300&h=269" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">(from http://irishchicklette.wordpress.com/2009/02/24/)</td></tr>
</tbody></table>This last weekend, I had the pleasure and privilege of attending the first ever KWST (Kiwi Workshop on Software Testing). Seventeen leaders in the software community came together in Wellington to discuss the topic of test professionalism. You can read about the workshop <a href="http://bjosman.wordpress.com/2011/06/28/kwst-kiwi-workshop-on-software-testing/">here</a> and <a href="http://softwareeducation.wordpress.com/2011/06/28/kiwi-software-testers-unite/">here</a>. Two very interesting terms came out that <a href="http://bjosman.wordpress.com/">Brian Osman</a> had coined to described his observations in the testing profession. One was stealth testing, which I'll talk about in another post, but the first was "Possum Testing".<br />
<br />
<a name='more'></a>The analogy is of a possum frozen in the headlights in the middle of the road. It should be noted that after much discussion and exploration of the concept, everyone came away with their own definitions, but the definition I came away with was: "Testing that is motivated by fear." An alternative definition was "Testing that you don't value" and yet another put a bet on both ways and used the definition "Testing you don't value, motivated on some level by fear." I'm sure that over time as we think and discuss it more, we will refine the definition.<br />
<br />
<i>(**Edit 30/06/2011 On reflection, I realise it's important to distinguish the "testing you don't value" aspect, so I see now that "Testing that you don't value, motivated by fear at some level" is the better definition.) </i><br />
<br />
Possum testing manifests itself when a tester is undertaking any test activity, such as writing test scripts, creating test documents, and doing "bad" testing because of the fear of being wrong, fear of appearing foolish, fear of being exposed as ignorant, fear of being challenged, or even fear of losing your job. They are simply going through the motions because, like a possum in the headlights, they are frozen from fear to do anything else.<br />
<br />
<br />
I can remember being a newbie tester, unsure of myself, and being told by people that wear ties that I need to script what I do so it can be repeatable. It seemed wrong to me, but I complied (for a day before I really decided that it was a complete waste of time) out of fear of being accused of not knowing what I was doing. Fortunately the road I crossed to become the tester I am today was mostly empty of speeding vehicles.<br />
<br />
Our industry is full of possums, and it's no wonder. Testers are being confronted constantly by the blinding headlights of folklore and myth, attached to heavy process-vehicles driven by managers and even other testers who don't actually understand the complexities and subtleties of testing.<br />
<br />
We need to take back control of our profession, and refuse to let the traffic freeze us. It can be scary. I recently was in a situation where I was asked to produce a document that I didn't value. Producing this document would have propagated deception in the name of process, but I feared the potential conflict that would arise if I refused. Realising that complying would be "possum testing" helped me to have the courage to take control of the situation, and unfreeze myself. I offered an alternative document that I did value, and I was able to step out of the headlights.<br />
<br />
For those of us who have made it safely across the road, I think we should try to help our possum brothers and sisters who are still frozen in the middle of the road, and show them the true light.<br />
<span style="color: black; font-size: 10pt;"><br />
</span>Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com19tag:blogger.com,1999:blog-7250447306574801643.post-51924055361865378562011-06-22T22:34:00.000+12:002011-06-22T22:34:33.136+12:00"Learn from others' mistakes"We're always told to learn from others' mistakes. I read a lot of testing blogs, and take the lessons that other people have encountered and learnt the hard way, and go "phew, glad I get to learn this the easy way." But I'm discovering that it's not really good enough, especially when it comes to personal credibility. I can cite anecdotes from other people and explain why I accept their reasoning and points of view for certain things, but until those stories are mine, and until those lessons are mine, I can't credibly pass those lessons on to others. <br />
<br />
Traditionally, when faced with a situation, I would throw my hands up and go "Nooo, I read about this in Michael Bolton's blog, it's a bad idea cos X, Y, Z!" or "Oh James Bach did a talk on this and why it's bad, let me find it!" Learn from others' mistakes, right? Don't repeat them, right?<br />
<br />
Well, it may be to the detriment of my own experience. I'm not seeing first hand the lessons others have learnt the hard way. So I am now going to consciously try and do things I'm told to do, even if I've read it's bad to do, just to experience it first hand. <br />
<br />
Perhaps I can go into work tomorrow and declare that all our testing should be automated. Perhaps not. ;)Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com2tag:blogger.com,1999:blog-7250447306574801643.post-51891181465776482112011-06-14T18:03:00.001+12:002011-06-14T18:11:54.097+12:00Inattentional Blindness in action - Anatomy of a testing session Inattentional blindness...the phenomenon of not being able to see things in plain sight. This is how it just happened to me:<br />
<br />
<a name='more'></a><br />
<br />
I'm testing an application that lets you create "shows" with a given resolution, but when you come to render the show as a movie file, you can override the show's resolution and choose another from a dropdown. As I was testing away, I noticed that sometimes the output video didn't match what I had chosen from the dropdown. After running some informal tests and not seeing the pattern, I drew a table to try to make sense of the situation. I suspected it had something to do with overriding standard definition (SD) formats with High Definition (HD) formats<br />
<br />
Since Blogger doesn't let me add tables the lines will be in the format:<br />
<b>Show spec ; Override value ; Result</b><br />
<b><br />
</b><br />
First test:<br />
720p ; No override ; 1280x720<br />
So far so good. A control test just to establish things when the show has an HD resolution<br />
<br />
Second test: Override HD format with SD format<br />
720p ; override to 576i ; 720x576<br />
Yup, that passed. The output was set to the override value<br />
<br />
Third test: establish control test for a show with SD resolution<br />
576i ; No override ; 720x576<br />
cool...so far so good<br />
<br />
Fourth test: override SD format with HD format<br />
576i ; override to 1080i ; 720x576<br />
ah ha! I might have this. Hypothesis: "overriding SD format with HD format does not change the show's default resolution"<br />
<br />
Now, let's be careful about confirmation bias here... so before we get carried away, let's run a couple more tests:<br />
Fifth test: establish a control test for 1080i resolution<br />
1080i ; No override ; 1920 x 1080<br />
<br />
Sixth test: Let's go from one HD resolution to another..maybe when you change to a lower resolution, not just from HD to SD it'll keep the show's original resolution <br />
1080i ; override to 720p ; 720x576<br />
err wait...what?<br />
<br />
Seventh test: (Starting to look up for patterns. hmmm all the failing tests had an original video specification what was interlaced. Let's try starting off with a progressive resolution)<br />
576p ; override to 1080i ; 720x576<br />
Nope not that.... now all failing tests are when I go from a progressive to an interlaced or vice versa. Let's go from progressive to progressive<br />
<br />
Eighth test:<br />
576p ; override to 720p ; 720x576<br />
Nope, that's not it...ok let's look at all my failing tests....<br />
<br />
And as my eye glances up the table I see the pattern: All failing tests are outputting 720x576<br />
<br />
Ninth test:<br />
720p ; 480p ; 720x576<br />
<br />
Yup, confident enough to stop testing now and declare: "If you override the show's original specification, no matter what resolution you pick, it will output in 720x576"<br />
<br />
So simple, yet I was blind to it for an hour or so.<br />
<br />
So what happened?<br />
<br />
Look up at test two. I was led down the garden path by how I interpreted the results of test two. I interpreted it as "The output was the show's original resolution" instead of "it outputs 720x576 regardless of what the original resolution is"<br />
<br />
One false positive.<br />
<br />
Look up at test four. Again, I interpreted that as "The output was the show's original resolution" instead of "it outputs 720x576 regardless of what the original resolution is".<br />
<br />
One false negative.<br />
<br />
Such a simple bug confounded by the resolutions I had chosen to test with (576i which is our standard SD television resolution in this part of the world) combined with how I was interpreting my observations.<br />
<br />
Lesson: Always be vigilant for bugs in the software you're testing, and for bugs in <i>your </i>testing.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmS4ePh0m6jQ3OK9W2gdmrxU6R_XKj0N2N_8UNHXmGwKfXwiNTPofHBKZeOU0hdyGvClpHevwa9Vy2hk-Pc4a9pS97yk6LBFHZpvXe306Y3mhhlUfyTGnqPxcBl6N4blaYLtRloEdPRuM/s1600/table.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmS4ePh0m6jQ3OK9W2gdmrxU6R_XKj0N2N_8UNHXmGwKfXwiNTPofHBKZeOU0hdyGvClpHevwa9Vy2hk-Pc4a9pS97yk6LBFHZpvXe306Y3mhhlUfyTGnqPxcBl6N4blaYLtRloEdPRuM/s320/table.JPG" width="226" /></a></div>Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com2tag:blogger.com,1999:blog-7250447306574801643.post-69364702584091904562011-04-14T12:22:00.000+12:002011-04-14T12:22:59.161+12:00Building an exploratory test plan redux: Software cartography<i>A <b>map</b> is a visual representation of an area—a symbolic depiction highlighting relationships between elements of that space such as objects, regions, and themes. </i>- http://en.wikipedia.org/wiki/Map<br />
<br />
Testing software is like exploring new lands. The same principles apply: There is an unknown something out there, and we want to turn it into a known something.<br />
<br />
<a name='more'></a>When exploring new lands, there are certain properties we are interested in. We may be interested in physical features such as coastlines, mountains, and rivers, or we may be interested in political borders, we may be interested in underwater features, we may be interested in rock distribution and faultlines, or we may be interested in all these things. And how do we report on what we find, and what our understanding of the land is, based on the properties we're interested in? We map them.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://www.nzetc.org/etexts/Gov13_07Rail/Gov13_07Rail017a.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="316" src="http://www.nzetc.org/etexts/Gov13_07Rail/Gov13_07Rail017a.jpg" width="320" /></a></div><br />
<br />
<b>1. Identify Properties you're interested in</b><br />
James Bach's Heuristic Test Strategy model is a great resource for stimulating test ideas. Identify the properties we're interested in, or <i>may </i>be interested in (after all, in these early days, who knows what may be important. We may not set out being interested in rock formations, but that probably would change should we find a large coal seam) and begin the framework for your map:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpf6nhZd87o5oj0d635UrNMsk340QhcSSNctxsQyU7PRUb1SrGW8uiHy8zCL6usDhv9U5rIQBB8hQc6TJYqr7r7pTisJY6tHn8J0fPDktwJvnfztShkw-QF06t9CiNiIOvQC4IwrY6kD4/s1600/mindmap1.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpf6nhZd87o5oj0d635UrNMsk340QhcSSNctxsQyU7PRUb1SrGW8uiHy8zCL6usDhv9U5rIQBB8hQc6TJYqr7r7pTisJY6tHn8J0fPDktwJvnfztShkw-QF06t9CiNiIOvQC4IwrY6kD4/s320/mindmap1.JPG" width="320" /></a></div><br />
<b>2. Start collecting intel</b><br />
Before setting sail for uncharted lands, we should gather as much information as we can, after all, we are in a state of ignorance about what we will encounter. There may be tales of great mountains, of bountiful oceans, and of savage beasts. There may be scrolls written by soothsayers, and there will be accounts written by previous travelers. We should sketch out our map based on this so we have a frame of reference for our journey, but fully expecting that we will change it, add details, take away wrong details etc when we get in there. A scripted traveler would take this heresay and folklore, and immediately chisel their map in stone, and describe in detail their path through these unknown lands.<br />
<br />
When we begin a testing project, we are in a state of ignorance. We should gather as much information as possible. There may be specification documents, Responses to RFPs, wiki pages, High level descriptions, requirements documents. There may be developers who can tell us great tales, and BAs with promises of bounty. We should begin mapping now so we have a frame of reference for our journey, but fully expecting that we will change it, add details, take away wrong details etc when we get in there. A scripted tester would take this heresay and folklore, and immediately chisel their map in stone, and describe in detail their path through these unknown lands.<br />
<br />
Map taking form as more information is discovered.<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipgjNPjzGw8bb2NrUd6mTDogbX3as5gsm0c7RsRbyfgczzxAPyatTw-yKLwjBxmursHTmoTLfuyFQfVMAPaDxnCLGKGt2VmL1G7OIH1kwm68O33wtStf5EemCtkebzEHvH8RK_3UP583E/s1600/mindmap2.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="118" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipgjNPjzGw8bb2NrUd6mTDogbX3as5gsm0c7RsRbyfgczzxAPyatTw-yKLwjBxmursHTmoTLfuyFQfVMAPaDxnCLGKGt2VmL1G7OIH1kwm68O33wtStf5EemCtkebzEHvH8RK_3UP583E/s320/mindmap2.JPG" width="320" /></a></div><br />
The act of mapping sparks new questions allowing you ask more focussed questions of the people around you. The map becomes the tool to make the map better. This is the beauty of an organic system. Give it some initial conditions (the heuristic test strategy model) and it almost builds itself. Here is the same map after a conversation with a developer:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1-Aw0MjYRcIjLyhisaqbvBGloHXPWreEzEBtpNoiSyP_4l8Q-PWA38llXsf1Fs3CXa0dwyah8QOQUVIMLJrTXqFN0MUOqG9MeR4yG2IOSe7eTVJ__YBHyR1OLR5EpUf3MtCEy7l8dSH8/s1600/mindmap3.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="178" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1-Aw0MjYRcIjLyhisaqbvBGloHXPWreEzEBtpNoiSyP_4l8Q-PWA38llXsf1Fs3CXa0dwyah8QOQUVIMLJrTXqFN0MUOqG9MeR4yG2IOSe7eTVJ__YBHyR1OLR5EpUf3MtCEy7l8dSH8/s320/mindmap3.JPG" width="320" /></a></div><br />
New parts have been added, We've learnt a bit about the structure, and about the platforms, and what is required to install it all. We were able to ask good questions because we could see where the silhouettes were that needed to be illuminated.<br />
<br />
<b>3. Begin touring</b><br />
We feel we have a useful framework to begin the actual work of touring the product. We know where the general inlets are, where the major landmarks are; we have enough to make landfall. Now we get the lay of the land. "Oh here's a river we didn't know about, better map that so we can take a closer look later." "This mountain is a surprise, I wonder if anyone knows about it".<br />
As an example, as I installed something, it asked me for a port number [3306]. Interesting, I better map that as a test idea:<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh812A6ZHRxyA90_TeFYz4RcyVS8IkQv4W1Q7btSRE5rLajmQaO8VoCwmlrQbD5N687DLE64FPbJ4xaf8b8FKscFWU9Myi3OlKxn2F70y0mUt0nU3vsDtFyblvh_g_QoVQEz8QH8RH_T_w/s1600/mindmap4.JPG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="31" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh812A6ZHRxyA90_TeFYz4RcyVS8IkQv4W1Q7btSRE5rLajmQaO8VoCwmlrQbD5N687DLE64FPbJ4xaf8b8FKscFWU9Myi3OlKxn2F70y0mUt0nU3vsDtFyblvh_g_QoVQEz8QH8RH_T_w/s320/mindmap4.JPG" width="320" /></a></div><br />
<br />
<br />
Eventually your map will be fleshed out that you're pretty confident that it is a pretty good and useful model of the software under test.<br />
<br />
<b>4. Create Test Charters</b><br />
<div style="font-family: inherit;"><span style="font-size: small;">If you're unfamiliar with test charters, or session-based test management, read these:</span></div><div style="font-family: inherit;"><span style="font-size: small;"><a href="http://www.satisfice.com/sbtm/">http://www.satisfice.com/sbtm/</a></span></div><div style="font-family: inherit;"><span style="font-size: small;"><a href="http://www.satisfice.com/articles/sbtm.pdf">http://www.satisfice.com/articles/sbtm.pdf</a></span></div><div style="font-family: inherit;"><span style="font-size: small;"><a href="http://www.satisfice.com/articles/et-article.pdf">http://www.satisfice.com/articles/et-article.pdf</a></span></div><div style="font-family: inherit;"><span style="font-size: small;"><br />
</span></div><div style="font-family: inherit;"><span style="font-size: small;">With a little tweaking, you've almost automatically generated your test charters from your mind map. As you run along the lowest levels of hierarchy, create your test charters from them.</span></div><div style="font-family: inherit;"><span style="font-size: small;"><br />
</span></div><div style="font-family: inherit;"><span style="font-size: small;">For example, one of my test charters will be "Install the Schedule Database with ports other than 3306".</span></div><br />
<b>5. Keep adjusting the map</b><br />
Your<b> </b>map is a model. As your model changes, so will your map. Parts that you thought were important will be deleted, and parts you didn't know existed will rise to prominence. The map is a reflection of YOUR understanding of the product at a given time.<br />
<br />
It also has other uses. <span style="font-size: small;">You can use it to manage the testing process by assigning testers to a branch, letting them take ownership of all sub branches. This lets you quickly assign work without having to go one by one through requirements and specifications and manually assigning individual tasks. You could colour branches that you're currently working on, and shade in branches that have been completed. At a glance, you can see what has been done, what is being worked on, and what is left to do. </span><br />
<br />
I hope this illustrates a good way to structure, manage, and report on exploratory testing. The main advantage is that it is quick to do: It is extremely lightweight and can be easily added to, and parts moved around with ease. It also doesn't need to be "read". A multipage document requires sitting down to read, and it is left to the reader to try and interpret what you mean. A map can be glanced at and almost immediately the structure, connections, functions, what needs testing, what hasn't been thought of are apparent. A map is worth a thousand words.<br />
<br />
<br />
The software I used in this post is the free version of XMind, but there are lots of different tools you can use. Freemind is also a good one.Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com7tag:blogger.com,1999:blog-7250447306574801643.post-64557123528918175502011-03-31T16:20:00.004+13:002011-04-01T09:52:25.351+13:00Putting systems into System Testing<i>System - A group of interacting, interrelated, or interdependent elements forming a complex whole.</i> - http://www.thefreedictionary.com/system<br />
<br />
People mean different things when they call themselves testers, and people think different things when they think about what it means to test something. People generally distinguish between unit testing, integration testing, system testing, and user acceptance testing. But what does system testing actually mean? Why is it, when we're "taught" about system testing, noone ever explains what a system is?<br />
<br />
<a name='more'></a><br />
I was introduced to the idea of general systems thinking a couple of years ago, and fell in love with it, even though I didn't quite understand what it was I was falling in love with. I still don't quite understand it, but I'd like to have a go putting the general systems thinking approach into system testing. I think we can take ownership of the term so that if you say "I'm a system tester", people won't confuse you with an automater, or a scripter.<br />
<br />
Gerald Weinberg offers this diagram to illustrate different types of systems with respect to methods of thinking:<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWEiD3K0uBj_YsgKvdXqlYhHmRPaJwGslUKCOx4HLCVbtq-MaWRKdCI8J5yG3S6zmlcawuuVbBBceovehubDqSDH5-v6-hq4nE3rFM2KqFKybm4JXe-EMjdQvMY7QnoRoinaxbhnihlyg/s1600/systems.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="288" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWEiD3K0uBj_YsgKvdXqlYhHmRPaJwGslUKCOx4HLCVbtq-MaWRKdCI8J5yG3S6zmlcawuuVbBBceovehubDqSDH5-v6-hq4nE3rFM2KqFKybm4JXe-EMjdQvMY7QnoRoinaxbhnihlyg/s400/systems.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">from Figure 1.9 'An Introduction to General Systems Thinking' Gerald M Weinberg</td></tr>
</tbody></table><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
We can use statistics for solving problems in region II, reductionism for solving problems in region I, but region III is a tricky region. It's too complex for reductionism, and too organised for statistics. This is where general systems thinking comes in. <br />
<br />
<br />
While some testers may focus on unit testing, or integration testing, and most testing literature ignores everything else but the software what is the role of a <i>system tester</i> when taking an expanded look at the meaning of a system? How can we apply systems thinking to this?<br />
<br />
<br />
Well, first thing we can do is try to define software out of region III and into one of the other regions, where there are already tools we can use. The Factory and analytical school already try to do this by reducing software to a set of stated requirements, and then reducing testing to the task of verifying those requirements. They attempt to push software testing down into region I, reducing testing to a set of components that can be individually tested while the agile school try and push software up into region II by dividing software up into a structureless mass of equal interchangeable parts, ie stories. The goal of agile testing is to determine when programming of a particular story is complete. The whole is considered done when the constituent parts are done, and each part is considered as equal as any other part.<br />
<br />
Here is ISTQB trying to define itself out of region III and into region I:<br />
<b>System Testing: </b>The process of testing an integrated system to verify that it meets specified requirements. - Standard glossary of terms used in Software Testing Version 1.3 (dd. May, 31st 2007)<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwjOq0rx9wkzYfKvIov2Cfdg1AjOEa2emDWi5WAgb6x9ljQnxxrv504v6pEid5yqtrd8Uz3TVeOD6D3peaQ992pxZJRDq8EDzvy4R_UYaWm5n2fQA-2szIkho3ISuY1izLNc8RRtcmIuw/s1600/Diagram3+.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="287" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwjOq0rx9wkzYfKvIov2Cfdg1AjOEa2emDWi5WAgb6x9ljQnxxrv504v6pEid5yqtrd8Uz3TVeOD6D3peaQ992pxZJRDq8EDzvy4R_UYaWm5n2fQA-2szIkho3ISuY1izLNc8RRtcmIuw/s400/Diagram3+.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Adapted from Figure 1.9 'An Introduction to General Systems Thinking' Gerald M Weinberg</td></tr>
</tbody></table><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Another issue is that testing text books, and courses such as ISTQB, give examples of problems that reside in region I. They then show reductive methods that work beautifully with region I problems, and then expect you to use those techniques in region III.<br />
<br />
But we're just deceiving ourselves, and those around us, by trying to sweep software into regions I and II. We're lulling ourselves into a false sense of security that we say we have everything under control because "100% of our stories' automated tests pass" or "We can trace every test back to a requirement in the requirements document. We have 100% traceability!"<br />
<br />
It also means we have a scapegoat when things go wrong: "Noone could have foreseen that" or "it wasn't in the requirements". This isn't good enough.<br />
<br />
So, if we venture out into the frontier of region III, what do things begin to look like? <br />
Software is not only a system in itself, but software is the product of a system (the dev team) for a system (the client) to be a component in yet a larger system (the client's world).<br />
<br />
Therefore software development: "A group of people who come together and help solve a problem for another group of people via the medium of computer software." <br />
<br />
Requirements: The gap between what the customer desires, and what they have. An emergent property of the relationship between the customers and their world. This means, to understand the requirements (vs what's written in a requirements document) we must understand the customers, AND the world in which they live in. This also implies that as the customer's world changes, so do their requirements.<br />
<br />
Requirements Document: One person's attempt to capture the above property of the customer's world via the medium of written language.<br />
<br />
Bug: The gap between what the user desires, and what the user gets. To understand if something's a bug, you must understand the customer.<br />
<br />
<br />
A systems tester understands the whole is greater than the sum of its parts. We cannot understand the whole by analysing the parts. All parts can individually pass, but the system as a whole can fail. The system can "pass" today, but "fail" tomorrow as the world changes.<br />
<br />
A systems tester understands that source code has no intrinsic value. If there's something 'wrong' with the source code, it's only "wrong" in so far as a user experiences something they shouldn't. Supposing the source code were hard to understand or maintain? Then there is the potential for programmers (who are also users of the code) to experience headaches (which they shouldn't) and it increases the potential for code to be written or omitted which could cause users to experience things they shouldn't.<br />
<br />
A systems tester understands that, just as the perfect transister is useless if the solder holding it to the board is defective, it doesn't matter how strong each team member is individually, if the connections between members is weak, the organism as a whole is weak.<br />
<br />
OK, that's my first pass at trying to get my head into applying general systems theory to software testing.<br />
<br />
It would be nice to be able to say "I'm a system tester" and have people understand what it means to think in systems, and what life is like in region III.<br />
<br />
In a future post I'll try to illustrate how other schools of software testing are using region I and region II techniques to solve region III problems, and then, hopefully, show how the context-driven school uses region III techniques to solve region III problems.Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com12tag:blogger.com,1999:blog-7250447306574801643.post-170525289543789232011-03-07T19:46:00.001+13:002011-03-08T13:05:20.013+13:00"Exploratory Testing" is a pleonasmIt's time to stop talking about Exploratory Testing as a subset of testing. There is no "exploratory testing" and "other testing". All testing is exploratory. If it's not exploratory, it's not testing.<br />
<br />
<a name='more'></a>Let's go back to Cem Kaner's definition of software testing:<br />
<i>Testing is an empirical investigation conducted to provide stakeholders with information about the quality of the product or service under test.</i><br />
<br />
Let's take the dictionary definition of "empirical" investigation: <br />
<i>Based on, concerned with, or verifiable by observation or experience" (Oxford English Dictionary)</i><br />
<br />
Combine:<br />
<i>Testing is an investigation </i><i>based on, concerned with, or verifiable by observation or experience</i><i> conducted to provide stakeholders with information about the quality of the product or service under test.</i><br />
<br />
What is exploratory testing?<br />
<i>"a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project." - Cem Kaner</i><br />
<br />
Or to put it a bit simpler: the simultaneous learning, design, and execution of tests. The design of the next test is based on the observations of the last test, and the experience the tester has accrued during the testing process. <br />
<br />
Exploratory testing is the opposite of scripted testing. The design of scripted tests is <i>not </i>based on the outcome of the previous test. The design of scripted tests is not based on, or concerned with observation or experience. Writing and executing test scripts is not testing.<br />
<br />
You could say: "oh, but while we are scripting, we have to explore the application so we know what to script." Correct. You are testing, and then scripting. Two separate activities.<br />
<br />
"Oh, but we often deviate from the script if we notice something strange, or want to check something else." Correct. You are following a script, and then sometimes you're testing. Two separate activities.<br />
<br />
Sometimes, the test script will be created using exploratory testing, and then will be executed with exploratory testing, and will be called scripted testing. This just highlights the fact that there is no scripted testing and exploratory testing. If you're testing, it's exploratory. It's time to drop the "exploratory" from testing and time to drop the "testing" from scripted.<br />
<br />
There is testing, and scripting. Dropping exploratory from testing means we don't need to justify what we're talking about when we say "exploratory" testing, since most people, even other testers, are ignorant as to what that actually means. And it should highlight the fact that when you go to an ISTQB course, or even many conferences, there isn't actually much talk about how to test. And it should immediately nullify any attempt by ignorant testers who say "well, exploratory testing has good and bad sides" or "we don't do exploratory testing". It sounds much more obviously silly to say "well, testing has good and bad sides" or "we don't do testing."<br />
<br />
Seriously, let's get down to what testing really means and begin teaching and talking about it, without the distraction of having to explain ET every time.Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com313tag:blogger.com,1999:blog-7250447306574801643.post-18874154297341103522011-02-17T11:29:00.007+13:002011-05-03T16:50:55.076+12:00Building a test plan from Woe to Go using Heuristics, Mind Maps, and Test ChartersCreating a test plan can be a daunting task, especially when under time pressure. "We need you to test this application and, oh, we go live next Monday." How do you create a plan to test that is accountable, transparent, and structured, yet lightweight enough that can quickly be understood by developers, project managers, other stakeholders, and new testers that may be brought on to help out? And not only that, but be created quickly?<br />
<br />
<a name='more'></a><br />
<br />
I'd like to propose a solution by combining knowledge and techniques from James Bach and others in the context driven community to create a solution which <i>may </i>help, depending on the application under test, and your workplace culture. It <i>sometimes</i> works for me, and I hope that it may work someone else.<br />
<br />
<span style="font-size: large;">1. James Bach's Heuristic Test Strategy Model</span><br />
Available here: <a href="http://www.satisfice.com/tools/satisfice-tsm-4p.pdf">http://www.satisfice.com/tools/satisfice-tsm-4p.pdf</a><br />
As it says on the front page: "The immediate purpose of this model is to remind testers of what to think about when they are creating tests."<br />
<br />
<i></i><br />
I like to build from the Quality Criteria categories, although I think you could begin building from any of the categories, depending on the type of model you want to build.<br />
<br />
<span style="font-size: large;">2. Mind Mapping</span><br />
There's a lot of talk in the context driven testing community right now about mind mapping, and I have to admit I was originally dismissive of the idea. I had always associated mind mapping with high school social studies teachers for some reason, or, at best, highly paid advertising executives, and wondered if they really had a practical purpose in the 'real world'. I've since learned, of course, there is no such as the 'real world' so I started looking into it. There are a few free mind mapping tools out there: <a href="http://en.wikipedia.org/wiki/List_of_mind_mapping_software">http://en.wikipedia.org/wiki/List_of_mind_mapping_software</a><br />
In these examples I'll be using <a href="http://freemind.sourceforge.net/wiki/index.php/Download">FreeMind</a><br />
You can read more about using mind mapping in testing here:<a href="http://www.bettertesting.co.uk/content/?p=956"> http://www.bettertesting.co.uk/content/?p=956</a><br />
<br />
<span style="font-size: large;">3. Combine</span><br />
Select the appropriate criteria for your project and map it out (I use the word 'Functionality' instead of 'Capability'):<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQTfw5T81NS6T6fvDodvwtoTFLsX34It3ssZNFTAh6Msj7kp2kgOZr1OheRne40DjeZGAkO1IsQ-AhRMoAD5YRgkkEZEzuwRZJhIgMELyxNfXaTNUTq868E0Z3uNIxM0W9LTnRD7zQ2FI/s1600/Capture.JPG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="135" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQTfw5T81NS6T6fvDodvwtoTFLsX34It3ssZNFTAh6Msj7kp2kgOZr1OheRne40DjeZGAkO1IsQ-AhRMoAD5YRgkkEZEzuwRZJhIgMELyxNfXaTNUTq868E0Z3uNIxM0W9LTnRD7zQ2FI/s400/Capture.JPG" width="400" /></a></div><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<span style="font-size: large;"> 4. Flesh Out</span><br />
Start fleshing out using the other categories to generate ideas.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia6bwVUrCcWlu8J_JDF29YTLXClUHgGKtYLzL79UiOq4G1xuNekfJFT2-UqaIbmJ1bRfSri913nbZbcPDhPCv2mhCZYhsWn6dO4hQl7O2xIwS07dhKF9UJ_LHzrFWWOs5gjh1mMy8lreQ/s1600/Capture.JPG" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="167" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEia6bwVUrCcWlu8J_JDF29YTLXClUHgGKtYLzL79UiOq4G1xuNekfJFT2-UqaIbmJ1bRfSri913nbZbcPDhPCv2mhCZYhsWn6dO4hQl7O2xIwS07dhKF9UJ_LHzrFWWOs5gjh1mMy8lreQ/s400/Capture.JPG" width="400" /></a></div><br />
<div class="separator" style="clear: both; text-align: left;"></div><span style="font-size: large;">5. Post up in a public area</span><br />
<span style="font-size: large;"><span style="font-size: small;">When you think you've finished, or you've run out of ideas, print out, and post up somewhere visible. This lets PMs and developers see what your model of the application is, and they can point out areas that you may have missed.</span></span><br />
<br />
<span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;">6. Create Test Charters</span></span></span><br />
<span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">If you're unfamiliar with test charters, or session-based test management, read these:</span></span></span></span><br />
<a href="http://www.satisfice.com/sbtm/"><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">http://www.satisfice.com/sbtm/</span></span></span></span></a><br />
<a href="http://www.satisfice.com/articles/sbtm.pdf"><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">http://www.satisfice.com/articles/sbtm.pdf</span></span></span></span></a><br />
<a href="http://www.satisfice.com/articles/et-article.pdf"><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">http://www.satisfice.com/articles/et-article.pdf</span></span></span></span></a><br />
<br />
<span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">With a little tweaking, you've almost automatically generated your test charters from your mind map. As you run along the lowest levels of hierarchy, create your test charters from them. I find that usually each lowest level generates 1 to 3 test charters. In the example above, some example test charters may be:</span></span></span></span><br />
<ul><li><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">Create admin user; ensure has admin rights as described in document X, and can create, edit, and delete users</span></span></span></span></li>
<li><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">Check the claims made by the sales team are present in the application</span></span></span></span></li>
<li><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">Test the Proin at ligula libero</span></span></span></span></li>
<li><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">Explore the Quisque quis libero urna using Internet explorer 6</span></span></span></span></li>
</ul><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;">7. Managing the testing process</span></span></span></span></span><br />
<span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">The mind map lets you visualise the testing process and can be used to report how testing is going. You can use it to manage the testing process by assigning testers to a branch, letting them take ownership of all sub branches. This lets you quickly assign work without having to go one by one through requirements and specifications and manually assigning individual tasks. You could colour branches that you're currently working on, and shade in branches that have been completed. At a glance, you can see what has been done, what is being worked on, and what is left to do. </span></span></span></span></span></span><br />
<br />
<span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;">I hope this illustrates a good way to structure, manage, and report on exploratory testing. The main advantages are that it is quick to do: no lengthy documents with headings and paragraphs of text; and has many emergent properties which also aid the management of the testing process.</span></span></span></span></span></span><br />
<span style="font-size: large;"><span style="font-size: small;"><span style="font-size: large;"><span style="font-size: small;"><br />
</span></span></span></span>Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com71tag:blogger.com,1999:blog-7250447306574801643.post-27605110510740461742011-01-26T17:17:00.002+13:002011-05-03T16:51:24.423+12:00We should be aiming to automate all our testingYou've heard it. I've heard it. This is my response:<br />
<br />
<br />
<a name='more'></a><br />
<br />
<div class="MsoNormal"><span style="color: #1f497d;">You need to differentiate between ‘testing’ and ‘checking’. A check is when you verify something to confirm existing beliefs. You check when you want to verify something still works the way it did before. You aren’t learning anything new, you’re verifying old knowledge.</span></div><div class="MsoNormal"><br />
</div><div class="MsoNormal"><span style="color: #1f497d;">‘Testing’ is exploratory. You ‘test’ when you want to find out new information. Testing is investigating the application to discover information the help people make quality and risk-based decisions about the application under test.</span></div><div class="MsoNormal"><br />
</div><div class="MsoNormal"><span style="color: #1f497d;">Checks can be automated. Tests cannot. Tests require a skilled tester to engage their brain and question and evaluate the application. Checks can be done by a machine.</span></div><div class="MsoNormal"><br />
</div><div class="MsoNormal"><span style="color: #1f497d;">Why the fetish with automation? Why would you want to rerun an old test when you can execute a brand new test? Why would you want to keep feeding the same data in, when you can be trying to invent new ways to test your system? Why would you spend weeks writing scripts that could be designed, and executed by a human in minutes (or seconds?). </span><br />
<br />
<span style="color: #1f497d;">Yes, in terms of regression, good checks are important to have, and are good to automate, but why would your goal to try and be to restrict your testers from finding out new and interesting things, and only getting a machine to tread the same paths again and again?</span></div><div class="MsoNormal"><br />
</div><div class="MsoNormal"><span style="color: #1f497d;">Not only that, but checks can only find things that are specified. But a tester can find potential problems no one has even thought to look for just by the simultaneous learning, design and execution of tests. Serendipity is a powerful ally.</span></div><div class="MsoNormal"><br />
</div><div class="MsoNormal"><b><i><span style="color: navy; font-family: "Arial","sans-serif"; font-size: 10pt;"></span></i></b><span style="color: navy; font-family: "Arial","sans-serif"; font-size: 10pt;"></span></div><div class="MsoNormal"><span style="color: #1f497d;">Would you rather write about a test, or actually perform a test and learn something about the application which may lead you to design and perform more interesting tests?</span><br />
<br />
<span style="color: #1f497d;">The goal of automating as many tests as possible is mistake, and if this is how you feel you really need to question why you feel that way. </span></div>Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com9tag:blogger.com,1999:blog-7250447306574801643.post-22197494301753236322011-01-06T16:09:00.002+13:002011-05-03T16:51:56.900+12:00Don't (explicitly) charge your clients for testingSoftware development houses who write bespoke software will usually provide a quote to develop or customise an application. Often, it's broken down into "Project management, requirements gathering, development, testing/QA, etc etc". In my experience, it is a mistake to break testing off as a separate quote to give to clients.<br />
<br />
Why?<br />
<br />
<a name='more'></a><br />
<br />
Because the client will look at the quote and decide "we'll do our own testing" and strike it from the quote. Your company, not wanting to lose the contract, will agree. I have seen it happen many times, and it is a mistake.<br />
<br />
Software testing is used to discover information so that stakeholders can make informed decisions. Software testers shine the light on the project. We hold the torch, and point it in directions we think need illuminating. We look in the corners, and say "hey, umm, there's this spider over here, is that ok?" Whether you go after the spider or not is your decision, but at least you know it is there, and won't be surprised by it in the night. We provide information to the project team to that they can make quality decisions.<br />
<br />
<br />
It's tempting to think "yes I know this, but it's not really relevant who does the testing, just that it gets done, right?"<br />
<br />
Your in-house testers work for your software company, not for the client. Your testers protect your company's interests, not the client's.To reject your inhouse testing on the basis that the client will do their own testing is misguided. Client test teams don't work your for your company, and aren't there to protect your interests. Client testers often<b> </b>don't provide any useful information which will help the project make good decisions. By the time they see the product, most of the risky decisions will have already been made. Clients aren't specialised testers. They often test in a rote fashion, adhering strictly to necessarily limited test scripts; have a limited understanding of the application; sometimes they outsource to cheaper testing companies which are sometimes paid by the bug, which leads to unimportant issues being logged - which the client will then want fixed.<br />
<br />
You are letting the client decide whether to allow you to make decisions in the dark, or to be able to make informed decisions. This makes no sense. The client is given the power to decide how much risk <b>you</b> should take on, and not only that, but given an incentive to decide you should take on more risk.<br />
<br />
Anyway, should a client be asked to pay for a service that helps you with your risky decisions? Do you ask them to pay for your insurance? You don't invoice clients for rent or power, even if those costs are an inherent part of your invoice.<br />
<br />
Either accept that the Test Team should be part of these inherent costs, too, whose cost will be recovered in quicker and more efficient development cycles, and in bugs being found <i>before</i> your clients find them, or "hide" them in the general development quote. You don't break development down into "setting up dev environments, writing code, writing unit tests" so why break testing away from that too? <br />
<br />
<br />
Just don't decide that it's easier to let your clients be your testers. Your software will suffer, and it will cost you real money and reputation. A penny you won't lose is a penny earned. Let your testers do what they do best, and that's look out for you.Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com2tag:blogger.com,1999:blog-7250447306574801643.post-76594566119078225082010-12-16T13:13:00.004+13:002011-05-03T16:52:22.532+12:00Before we begin<i>"Many arguments appear to be about conclusions, when they're almost always about premises."</i><br />
<br />
Before I begin, I think it's important to lay out exactly what I mean when I use certain words and phrases. Then if you disagree with me, you can decide if we're arguing about conclusions or premises. :)<br />
<br />
<a name='more'></a><br />
<br />
<b>Testing</b> - I use Cem Kaner's definition of testing:<br />
<i>Testing is an empirical investigation conducted to provide stakeholders with information about the quality of the product or service under test.</i><br />
<br />
To me, testing is more of a social science than a computer science. So what does that definition above actually mean? Let's break it down:<br />
<br />
<u><b>Empirical Investigation:</b></u> <i>"Based on, concerned with, or verifiable by observation or experience" (Oxford English Dictionary)</i><br />
<br />
Let's look at <a href="http://en.wikipedia.org/wiki/Adriaan_de_Groot">Adriaan de Groot's</a> empirical cycle: <br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: right; margin-left: 1em; text-align: right;"><tbody>
<tr><td style="text-align: center;"><a href="http://upload.wikimedia.org/wikipedia/commons/thumb/5/53/Empirical_Cycle.svg/250px-Empirical_Cycle.svg.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="http://upload.wikimedia.org/wikipedia/commons/thumb/5/53/Empirical_Cycle.svg/250px-Empirical_Cycle.svg.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">from Wikipedia: http://en.wikipedia.org/wiki/File:Empirical_Cycle.svg</td></tr>
</tbody></table> <br />
<b>Observation:</b> Observe some attribute of the application: "I observe a screen with two text input fields. One has "username" next to it, and the other has "password" next to it. There is a "login" button and there is a "register" hyperlink.<br />
<b>Induction: </b>Formulating hypothesis: In my experience, screens that have these fields in them are called "login screens" and allow registered users to access content and restrict non-registered from accessing the content. <br />
<b>Deduction: </b>Creating a testable prediction based on your hypothesis: I have not registered for this site, therefore, if I enter my login details, I should be denied access.<br />
<b>Testing:</b> Test the hypothesis: I enter "aaronhodder" in the username field, and "password" in the password field. "Access Denied" displays.<br />
<b>Evaluation</b>: Evaluate the outcome of the testing: It behaved how I expected it to behave. I have evidence that this screen <i>may </i>be working in a way that is consistent with other web applications.<br />
<br />
But we're not finished<br />
<br />
<b>Observation: </b>"Access denied" displays<br />
<b>Induction: </b>"Access denied" displays when any user, registered or not, enters their login details<br />
<b>Deduction: </b>If I enter the details of a registered user, then "Access denied" will display<br />
<b>Testing: </b>Obtain the details of a known registered user and enter them in the text fields. I am allowed access to the restricted part of the site<br />
<b>Evaluation: </b>Hypothesis was incorrect, and I have further evidence that the screen <i>may</i> be working in a way that is consistent with other web applications<br />
etc<br />
etc<br />
and so the cycle continues.<br />
<br />
Note that the definition I use puts the tester's skills at evaluation and judgement at the centre of the testing process, not tools, or scripts. Context over process. Requirements over requirements documents.<br />
<br />
<u><b>Stakeholders:</b></u><b> </b>Developers, project managers, customers, clients, the business. Anyone who holds a stake in the project. It is important to know who your stakeholders are, and what is important to them to ensure that the information you are providing is useful.<br />
<u><b>Information:</b></u> Examples of information: Does it do what we claim it does? Does it do what the customer wants it to do? What defects have we found?<br />
<u><b>Quality:</b></u> Here I use Gerald Weinberg's definition: <i>Quality is value to some person</i> with Cem Kaner's addendum: <i>...that matters.</i> Quality is value to some person that matters. Part of the tester's role is to discover who the people are that matter, and what they value.<br />
<br />
So that's the sum of its parts. The whole is of course much larger, and that is that testing is an active investigative process to discover and report important information about the product or service to the people that ought to know it.<br />
<br />
<br />
I hope this makes future posts easier to understand and debate by framing it in the above context.<br />
<br />
<u><b><br />
</b></u>Aaron Hodderhttp://www.blogger.com/profile/14103184994911374509noreply@blogger.com21