Acceptance tests are not enough!

Acceptance testing is a key method in Agile. One way of defining acceptance tests are Gojko Adzic‘s ”Specification by example” paradigm which has gained quite a bit of momentum lately. I personally found it to be both refreshing and nice when I heard him present it at Agile Testing Days 2009, and I also found his book Bridging the Communication Gap a nice read.

Photo of Gojko Adzic demonstrating communication gaps in his keynote presentation at Agile Testing Days 2009
Gojko Adzic demonstrating communication gaps at Agile Testing Days 2009

I’m sceptical of the concept of acceptance testing. Not because verification of agreed functionality is not a good thing, but because it tends to shift attention to verification instead of exploration.

This will shift attention from testing to problem prevention. Is that bad, you ask? Isn’t it better to prevent problems than to discover them?

Well, most people think ”why didn’t I prevent this from happening?” when problems do happen. Feelings of regret are natural in that situation and that feeling can lead you into thinking you should improve your problem prevention. And maybe you should, but more examples aren’t going to do it!

Real testing is still necessary.

To explain why, I’ll consult one of the great early 20’th century mathematical philosophers: Kurt Gödel. In particular his first incompleteness theorem. It says that no consistent system of axioms whose theorems can be listed by an “effective procedure” is capable of proving all facts about the natural numbers.

What does this mean to us?

It means that we will never be able to list all things that can be done with this particular set of data.

A specification is a kind of listing of ”valid things to do” with data, thus Gödel’s theorem teaches us that there are infinitely more things to a system than any long list of requirements. This also applies when the requirements are listed as examples.

If you’re in the business to deliver products of only ”agreed quality” to a customer, you can be all right only verifying things which are explicity agreed. If something goes wrong you can always claim: ”It wasn’t in the specification!”

But if you’re striving for quality in a broader sense, verifying that the system works according to specifications is never going to be enough.

Gojko has made a good contribution to agile. Examples can be useful and efficient communication tools, and if they are used correctly they can help making users and other interested parties better aware of what’s going on on the other side. His contribution can help bridge a communication gap. It can also produce excellent input for automated unit tests.

Just don’t let it consume your precious testing time. The real testing goes far beyond verification of documented requirements!

If you want to learn more about this, I recommend you sign up for one of the Rapid Software Testing courses offered by James Bach and Michael Bolton.

Photo from Rapid Software Testing course in London November 2010 as offered by Electromind
Michael Bolton with one of the many interesting and challenging test exercises at the RST course

5 thoughts on “Acceptance tests are not enough!

  1. Hi Anders, thank you for a great post! Couple of days ago I wrote about testing vs checking from the perspective of what programmers should be needed to do. And I like the way you explain the similar thing but with a somewhat different angle.

    Here is my post:


    1. andersdinsen 2011-04-03 — 14:43

      Hej Sigge

      Thanks for linking to your recent post. It’s an interesting read (- comments too!).


  2. Acceptance testing is a horrible concept. What happens of the users “accept” the software and some time later a major problem is found? Are they stuck with it?

    What if the users “reject” the software due to a defect or missing feature? Can they walk away from the software and find another solution?

    It’s not about acceptance and rejection. It’s about making the software do what the users need in a simple, reliable way.

    1. andersdinsen 2011-04-03 — 15:14

      Hi Vin,

      Thanks for your comment!

      I actually don’t agree about acceptance testing being horrible in general. It can certainly be misused, but I strongly believe in Quality Control of software, and acceptance testing is one way of controlling, e.g. as a QC on the customer side. Many (most?) software contracts specify some kind of “acceptance test” to be conducted before the customer accepts the delivery, usually as part of the legal handover of ownership of the system.

      But you have a very valid point in saying that it’s not about acceptance/rejection. There’s a lot more to it than that. The concept of “quality” exists only in the grey area between those two extremes.


Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close