Wednesday, January 26, 2011

We should be aiming to automate all our testing

You've heard it.  I've heard it.  This is my response:




You need to differentiate between ‘testing’ and ‘checking’.  A check is when you verify something to confirm existing beliefs.  You check when you want to verify something still works the way it did before.  You aren’t learning anything new, you’re verifying old knowledge.

‘Testing’ is exploratory.  You ‘test’ when you want to find out new information.  Testing is investigating the application to discover information the help people make quality and risk-based decisions about the application under test.

Checks can be automated.  Tests cannot.  Tests require a skilled tester to engage their brain and question and evaluate the application.  Checks can be done by a machine.

Why the fetish with automation?  Why would you want to rerun an old test when you can execute a brand new test?  Why would you want to keep feeding the same data in, when you can be trying to invent new ways to test your system?  Why would you spend weeks writing scripts that could be designed, and executed by a human in minutes (or seconds?).  

Yes, in terms of regression, good checks are important to have, and are good to automate, but why would your goal to try and be to restrict your testers from finding out new and interesting things, and only getting a machine to tread the same paths again and again?

Not only that, but checks can only find things that are specified.  But a tester can find potential problems no one has even thought to look for just by the simultaneous learning, design and execution of tests.  Serendipity is a powerful ally.

Would you rather write about a test, or actually perform a test and learn something about the application which may lead you to design and perform more interesting tests?

The goal of automating as many tests as possible is mistake, and if this is how you feel you really need to question why you feel that way.

7 comments:

  1. Hi Aaron!

    I disagree that doing checking testers are not learning new things. In order to write checking code in some scripting language (or tool) you need to know how to do this. You are learning about system under test by reading requirements, wiki pages or by talking with developers.
    Doing the exploratory testing you are thinking about putting system under test in action that is product of your imagination but also based on knowledge that you had collected during the development of checking scripts.
    Also exploratory test can be automated, but only after you designed it (using your knowledge, imagination, ...).

    Regards, Karlo.

    ReplyDelete
  2. "You are learning about system under test by reading requirements, wiki pages or by talking with developers."

    If you are reading about them in requirements, wiki pages, or by talking to developers, then it is not new information. It is information already known.

    "Doing the exploratory testing you are thinking about putting system under test in action that is product of your imagination but also based on knowledge that you had collected during the development of checking scripts."

    I'd say it's the other way around. People develop scripts based on the knowledge they learnt during exploratory testing.

    "Also exploratory test can be automated, but only after you designed it (using your knowledge, imagination,"

    The execution can be. I can't really comment, but Michael Bolton wrote something that supports the statement "Exploratory testing can be automated". I think you'll find it depends on how you define the question though: http://www.developsense.com/blog/2010/09/can-exploratory-testing-be-automated/

    ReplyDelete
  3. checking vs testing is one perspective
    manual vs automated is another

    additional perspectives can be found in Crispins Vancouver quadrant.

    Full automation is about as possible as finding all bugs. There is a long tail of bugs and a long tail of testcases.

    http://www.eurostarconferences.com/blog-posts/2010/9/6/close-the-tool-and-start-exploring---jesper-ottosen.aspx

    ReplyDelete
  4. "If you are reading about them in requirements, wiki pages, or by talking to developers, then it is not new information. It is information already known."

    You are right about that this in not new information. But my point was that this is starting point. You need to gain that knowledge in order to revel new information using exploratory testing.

    ReplyDelete
  5. Jesper is right - I think you'd find the Testing Quadrants used in Agile really educational. It kind of highlights areas where automation works best and where it does not. And the best kinds of testing use a bit of both.

    I like you're article. You're kind of right, the kind of tests you do with automation are just a kind test you probably expect to pass. But automation removes the donkey work from testing - when you're running a test for the 6th time you tend to go "yeah yeah yeah I know this" a lot.

    That said, there's always a place for human intuitive "playing around". However the best place for that is after you've run the automated tests.

    ReplyDelete
  6. Picture of those quandrants ...

    http://onestepbacktwostepsforward.blogspot.com/2009/06/agile-testing-quadrants.html

    ReplyDelete
  7. I learned a lot from this article, thanks a lot for sharing. You need to gain that knowledge in order to revel new information using exploratory testing. Best regards, paper-writing.services

    ReplyDelete