Monday, March 7, 2011

"Exploratory Testing" is a pleonasm

It's time to stop talking about Exploratory Testing as a subset of testing.  There is no "exploratory testing" and "other testing".  All testing is exploratory.  If it's not exploratory, it's not testing.

Let's go back to Cem Kaner's definition of software testing:
Testing is an empirical investigation conducted to provide stakeholders with information about the quality of the product or service under test.

Let's take the dictionary definition of "empirical" investigation:
Based on, concerned with, or verifiable by observation or experience" (Oxford English Dictionary)

Combine:
Testing is an investigation based on, concerned with, or verifiable by observation or experience conducted to provide stakeholders with information about the quality of the product or service under test.

What is exploratory testing?
"a style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project." - Cem Kaner

Or to put it a bit simpler: the simultaneous learning, design, and execution of tests.  The design of the next test is based on the observations of the last test, and the experience the tester has accrued during the testing process.

Exploratory testing is the opposite of scripted testing.  The design of scripted tests is not based on the outcome of the previous test.  The design of scripted tests is not based on, or concerned with observation or experience.  Writing and executing test scripts is not testing.

You could say: "oh, but while we are scripting, we have to explore the application so we know what to script."  Correct.  You are testing, and then scripting.  Two separate activities.

"Oh, but we often deviate from the script if we notice something strange, or want to check something else."  Correct.  You are following a script, and then sometimes you're testing.  Two separate activities.

Sometimes, the test script will be created using exploratory testing, and then will be executed with exploratory testing, and will be called scripted testing.  This just highlights the fact that there is no scripted testing and exploratory testing.  If you're testing, it's exploratory.  It's time to drop the "exploratory" from testing and time to drop the "testing" from scripted.

There is testing, and scripting.  Dropping exploratory from testing means we don't need to justify what we're talking about when we say "exploratory" testing, since most people, even other testers, are ignorant as to what that actually means.  And it should highlight the fact that when you go to an ISTQB course, or even many conferences, there isn't actually much talk about how to test.  And it should immediately nullify any attempt by ignorant testers who say "well, exploratory testing has good and bad sides" or "we don't do exploratory testing".  It sounds much more obviously silly to say "well, testing has good and bad sides" or "we don't do testing."

Seriously, let's get down to what testing really means and begin teaching and talking about it, without the distraction of having to explain ET every time.

39 comments:

  1. Hi Aaron,

    Firstly, fantastic post!

    Just to add to what you've already described fantastically in the difference between scripting and testing.

    The process of generating scripted tests without a system in place, e.g. up front via requirements specification, can also be exploratory testing testing, as long as they are not just copying and pasting rules from the requirements (I've seen this happen).

    Thanks for sharing, I really enjoyed reading this.

    Cheers,

    Darren.

    ReplyDelete
  2. I'd be interested to hear about other ways of generating scripts that doesn't include putting "verify that" in front of every bullet point in the requirements document.

    ReplyDelete
  3. You have some good points, but I can't stop thinking that we have a weak ET definition if it doesn't mean anything (more than "good".)
    Maybe the old one (technique/activity) was better?

    There is also one hole in your argument:
    What about unscripted tests that aren't based on the observations of the last test?

    Some things about other formats than "Verify that..." can be found at
    http://thetesteye.com/blog/2010/11/synthesizing-test-ideas/

    ReplyDelete
  4. >I can't stop thinking that we have a weak ET definition if it doesn't mean anything (more than "good".)

    Or...we now have a stronger "testing" definition?

    >What about unscripted tests that aren't based on the observations of the last test?

    Good point. The error there was saying in the first place that the next test is based on the outcome of the last test, as if that were always the case. I'll try and be more careful next time :)

    ReplyDelete
  5. Hi Aaron,

    You'll find if you write your test conditions from requirements specifications, you actually clarify the scope.

    Whenever I write up front tests for our developers to run prior to committing code, I spend probably half the time communicating with others, identifying gaps, risks and expanding on unclear requirements. I probably spend about 35% of the remaining time thinking and only 15% of my time actually writing any test conditions for the scripts.

    The big thing for us though is that we as testers only write these scripts, we don't execute them, they are essentially acceptance checks which the developer is required to check off before submitting a user story card. We write small use cases to act as regression tests with the view that most of these will later be automated.

    Remember being at the requirements stage the cost to fix is very low, as nothing has been built yet except for the specifications.

    If you haven't read it already I'd recommend having a read of my article on lean test case design (http://www.bettertesting.co.uk/content/?p=253) I go into more detail on the power of up-front test cases there.

    Cheers,

    Darren.

    ReplyDelete
  6. This comment has been removed by the author.

    ReplyDelete
  7. >>What about unscripted tests that aren't based on the observations of the last test?

    >Good point. The error there was saying in the first place that the next test is based on the outcome of the last test, as if that were always the case. I'll try and be more careful next time :)

    Actually, having thought about it some more, then, yes, your next test IS always decided by the outcome of the last test. If the outcome of the last test (and the one before that, and the one before that) yields no useful information, then you will use that outcome to decide to do something else, or apply a stopping heuristic and go to lunch.

    Apart from external factors (the building catching on fire, a phone call etc), and apart from scripting / checking, are there any times that the next test you execute is not based on observations of the last test? I'm trying to think of one...

    ReplyDelete
  8. "Apart from external factors (the building catching on fire, a phone call etc), and apart from scripting / checking, are there any times that the next test you execute is not based on observations of the last test?"

    Sure. Imagine that you wanted to act on ten test ideas. On the eighth, you observe something interesting. You consider investigating it right then, right there... but you want to act upon the last two test ideas, so you do.

    In your overall post, though, it seems to me as though you're treating exploratory and scripted as binary; a test is either exploratory or scripted. I don't think it's that clear. A test is scripted to the degree that a) the idea comes from someone else; and to the degree that b) the idea comes from some time in the past. A test is very scripted when the script, the other person, or the past determines the tester's next action. A test is exploratory to the degree that the tester him or her self has the authority to determine the next action, and to the extent that the test idea could come from the most recent instant.

    So, all testing that you do for a client is scripted to some degree, in that you're acting on a mission you've been given. All testing that a human performs is to some degree exploratory, since a human will alter his or her behaviour in ways that the script can't control or anticipate. People tend to work around problems with the script, rather than freezing and turning blue.

    I agree that all testing worthy of the name is exploratory, but I'm not convinced that it's helpful to say you're doing one or the other exclusively. Only machines can perform testing in a completely scripted way (I call that checking). I think it is important, though, to identify the degree to which you're doing one or the other; the degree to which you're free to do what you need to do in the moment; the degree to which you're responsible for it; and the degree to which you're being guided by other people and by the past.

    ReplyDelete
  9. Thanks Michael, that helps a lot in clarifying my thinking.

    ReplyDelete
  10. hahaha, I just happened upon this coincidentally: http://www.satisfice.com/blog/archives/496

    ReplyDelete
  11. From Virdesi: At P&R Bakery and Cafe, our slogan "Stuff as good as your Momma's" means exactly that. bersavto.ru With the new name came the addition of high scores.

    ReplyDelete
  12. By correctly matching the beat of bouncing Thumpies, the player progresses through a song. download apps You can Undo a move by pressing the undo button on top.

    ReplyDelete
  13. The clock as a time machine has become a decisive machine in our civilization. ilovesharewares.com Users may answer multiple concurrent chats and respond to Mobile Text Connect (MTC) inquiries from prospective customers.

    ReplyDelete
  14. Are you on a diet and you want to keep track of your progress. 4demotivator.ru It presents different stretching exercises for your wrists, elbows, shoulders, neck, body, legs, and arms.

    ReplyDelete
  15. ""Like the new voice in the newsest version and the Thai is now much better :) just what I needed, keep up the great work. Welcome to my site. NO REGISTRATION Use guest mode to play poker without registration.

    ReplyDelete
  16. Website is the premier method to introduce your business to the market. I find it good to see that you have done well in maintaining your site. Students not being able to write their academic coursework could opt for our Assignment Help and can get comprehensive homework written from us.

    ReplyDelete
  17. This is really a great stuff for sharing. Project Management MBA Capstone Project Help Thanks for sharing.

    ReplyDelete
  18. Really i appreciate the effort you made to share the knowledge. This is really a great stuff for sharing. harvard case solutions Keep it up . Thanks for sharing.

    ReplyDelete
  19. I appreciate your efforts in preparing this post. DO My C programming Assignment I really like your blog articles.

    ReplyDelete
  20. I loved the way you discuss the topic great work write essay for me thanks for the share.

    ReplyDelete
  21. I’m really impressed with your article, Do My Matlab Project such great & usefull knowledge you mentioned here

    ReplyDelete
  22. Hi buddy, your blog' s design is simple and clean and i like it. online BBA dissertation writing service Your blog posts about Online writing Help are superb. Please keep them coming. Greets!

    ReplyDelete
  23. The leading assignment help UK firm offers state of the art services to its clients Do My Medical Science Homework with a promise of delivering all the required work well within the deadline.

    ReplyDelete
  24. I genuinely appreciated understanding it. Do My Medical Science Homework Sitting tight for some more incredible articles like this from you in the nearing days

    ReplyDelete
  25. I’m really impressed with your article, OS Assignment Help such great & usefull knowledge you mentioned here

    ReplyDelete
  26. My friend recommended this blog and he was totally right PHP Programming Assignment Help keep up the fantastic work!

    ReplyDelete
  27. With these pictures Test Drive Assistant provides you a way to do a side-by-side comparison of the vehicles. checkout this. If you want even more colors (and patterns!) or just want to get to that next color faster, purchase the Bonus Pack.

    ReplyDelete
  28. - iCloud sync for iOS 8- Dropbox auto-backup (In-App purchase)- Sort added for Contacts and Products/Services pages. Click To Discover More! Pontos Fracos: Caractersticas que demonstram desvantagens em relao a outros.

    ReplyDelete
  29. Price: 1 star - even though it just came out, it's not worthy of such a price. This link If enough people had this App, perhaps the bad-guys might think twice before abducting someone.

    ReplyDelete
  30. only professional writers Programming Homework Help can make this kind of material, cheers

    ReplyDelete
  31. Thanks a lot for the post. It has helped me get some nice ideas. Buy R Programming Assignment I hope I will see some really good result soon.

    ReplyDelete
  32. I appreciate this work Stats Homework Help amazing post for us I like it.

    ReplyDelete
  33. This comment has been removed by the author.

    ReplyDelete
  34. Well thanks for posting such an outstanding idea. I like this blog & Digital Marketing Agency I like the topic and thinking of making it right.

    ReplyDelete
  35. Good way of telling, good post to take facts regarding my presentation subject matter, Medical Thesis Writing Service which i am going to deliver in my college

    ReplyDelete
  36. by visiting this site Marketing Research Homework Help I found cool stuff here keep it up.

    ReplyDelete