Checking Vs Testing – How this applies to Test Analysis

There have been a few posts by the great and the good of the testing world over the last few days about Testing vs. Checking, which was further broken down in to Human Checking and Machine Checking.

Before we go any further I thought I would give a little background about me. I am not a great or good. I am just a career tester making my way in this world. I would like to think however that I have gained a bit of valuable experience in the last 15 years that I have been in this industry. I take my profession seriously, am passionate about it, and I have an opinion on most things. Therefore I thought what the hell, I might as well make my opinions/thoughts and views known on this particular topic for my first (and maybe last) blog post. As this is my first ever post, and I am not the most prolific writer in the world, I apologies in advance if this post is either poorly written or a little bit choppy in its direction.

OK so back to the point. I want to make it clear that I am in support of the refined definitions of testing vs. checking that as been put forward by James Bach and Michael Bolton in the post Testing and Checking Refined. It makes a lot of sense to me and I have seen both human checkers and testers during my career. However, the one thing that stood out to me when I read this post, and others in response to it from the Iain McCowatt (Human and Machine Checking) and Paul Holland (Reply to Human and Machine Checking), was how the focus was on what happens during the execution of a check. There was no discussion or thought given to the process of defining that check in the first place from what I could tell.

If I have interpreted what these guys are saying, then checking, whether human or machine comes from a pre-scripted testing approach, as opposed to an exploratory one and as such this would mean that someone had spent time analysing the system to be tested prior to the execution and will have applied some knowledge to that process. Therefore could it be possible that although at the time of execution the script is a check, due to the process of the defining of that check a wider element of testing has been applied and as such makes the check at the time of execution stronger (or weaker if the tester doing the analysis has  not done well in this task)? or more correctly, the set of checks be stronger?

In my experience the analysis phase of testing is as much of a minefield as execution when it comes to the test vs. check debate. Test analysts can fall into a similar categorisation where they either “check” by identifying the requirements and then define a test to prove/disprove that requirement or they “test” the analysis done by the designers by analysing the requirements and then by using there knowledge are able to read between those requirements to identify questions or hypothesizes about what has not been explicitly stated in the design/requirements. If they have done the latter then they are adding more value to the set of checks that they produce than if they just apply the former. In my opinion this will make the list of checks to be executed stronger than than if they have just “ticked” of the requirements that had been presented to them. This leads me to believe that although checks (in the execution phase term) are only part of testing, there strength will be based on the type of test anaysis / Design that was performed during the generation of those checks.  .

So in conclusion I believe that applying the testing vs checking view at the test analysis phase means that at the time of execution not all checks are equal. You can have a strong list of checks or a weak list of checks depending. If you then have a human executing these checks, then as has been mentioned you will add further value.

This is my views that I wanted to put out there. As I stated at the beginning I am not one of the greats or goods, but I have views. I thought long and hard about putting this first blog out, can be very daunting to a newbie, putting your thoughts out there to be analysed and pulled apart, but even if no one agrees I hope that people will explain constructively why they feel I am mistaken or talking rubbish so that I can learn more. …. all I ask is be gentle 🙂

 

 

6 thoughts on “Checking Vs Testing – How this applies to Test Analysis

  1. Hello, and welcome to blogging. I started less than a year and a half ago, so I can relate to emotions that you’ve just shared!

    Now, to checking: I’d like to ask a few questions, if I may?

    Does a check need to be contained within a script?
    Might a check be used in an exploratory manner?
    Can a requirement be proven, by a check, or by any other means?
    What purposes, other than confirming requirements, might a check serve?
    Does lack of discussion imply lack of thought? 😉

    BTW, your insight that some checks might be stronger than others is a good one IMO. Kaner talks about this, in reference to test cases in “What is a good test case?”

    -Iain

    • Hi Iain,

      Thank you for taking the time to read my blog and reply. As for asking me questions.. of course you can, and I hope I can answer them too.

      Does a check need to be contained within a script?

      No I guess it does not, checks can be done during exploratory testing but this was not my interpretation of the “Check” definition. Also would the process of defining that check not still be the same, just done “in flight” rather than prior in a scripted version, and therefore still fall into the analysis “check” or “Test”?

      Might a check be used in an exploratory manner?

      Not if I have understood the definitions discussed (which is open to debate). If I have then if we start to state that during exploratory testing we perform checks then are we not blurring the definitions again? One could argue that exploratory testing is a number of checks that spawn new checks based on the results of those checks previously executed.. but would we want to? 🙂

      Can a requirement be proven, by a check, or by any other means?

      Well yes I believe a requirement can be proven/disproved by a check (or collection of checks depending on the granularity of the requirement), but when I refer to a requirement in this context I am referring to a predefined explicit requirement, not an implicit one. As for by other means well it is late but no I cant think of anything that could, might have to come back to you on that one.

      What purposes, other than confirming requirements, might a check serve?

      Well a check could answer a question born from prior testing as an example, but this would then be using a different definition of check again than those put forward on the Blogposts mentioned, or at least my initial interpretation of them.

      Does lack of discussion imply lack of thought?

      Absolutely not lol 🙂

      Thanks again for your reply, already made me start to challenge some of my own initial interpretations/view so that’s all good 🙂

      • “checks can be done during exploratory testing but this was not my interpretation of the “Check” definition”
        Say you’ve observed some system behaviour. You form a mental model as to what the system is doing. Based on this model you deduce how the system might behave under slightly different conditions. You check whether this appears to be the case. When we’re testing, we dip in and out of checking in this manner all the time. What in the definitions suggests to you that this would not be a “check”?

        “Also would the process of defining that check not still be the same”
        A good question. What similarities / differences can you think of?

        “a requirement can be proven/disproved by a check”
        Let’s leave disproven for now. How many barren, lifeless worlds does it take to prove that Earth is the only one to support life? How many alien-infested worlds does it take to disprove this hypothesis?

        “Well a check could answer a question born from prior testing as an example, but this would then be using a different definition of check”
        How so? This is related to a lengthy conversation that I had with James, to a degree with Michael, and with others on Twitter. A check might disconfirm (depending on its results), might be intended to disconfirm (depending on how you plan on using it), or might simply be designed to provide an information point. I’ll concede that checks are most commonly used in a confirmatory manner.

        -Iain

        PS I suggest you include an About page, and use a real identify on Twitter 🙂 It’s nice to know who you’re talking to.

  2. Welcome to blogging.
    I would like to clarify my position on the first time a check is executed. As we don’t know how the system will behave then we are actually testing. On subsequent executions then we will be checking the resulting behaviour hasn’t changed.

    Keep blogging!

    -Paul

    • Paul

      Why is a first-run check not a check? It links and observation to a rule and outputs the result.

      What about a second-run check? Do you /actually/ know what the software is going to do? If you get a different result, is the check suddenly not a check?

      -Iain

  3. Strong or Weak tests / checks also depends on the related stage of the tests,
    In initial stage of the version – we might just want to verify that GUI for instance was made by design – A simple generic check list which can be derived automatically may be found very useful for that stage.
    After SW is a bit more mature, we might need to switch to other methods, as that method would be found weak.
    If we start right away using complex scenarios – we might find these weak up to not even executable, but these will serve as great tests for later stages and regression.
    So what may be strong and useful for one stage, might be weak and useless for another,
    Each serves it purpose in right time.

    @halperinko – Kobi Halperin

Leave a Reply

Your email address will not be published. Required fields are marked *