Pitfalls of relying on/or looking only for "Expected" behavior in Testing

http://www.testingeducation.org/a/TheOngoingRevolution.pdf

Testers should specify the expected result of every test, in advance?
This is another guidance from Myers (1979) that has had lasting influence. See, for example, ISEB's current syllabus for test practitioner certification at www1.bcs.org.uk/DocsRepository/00900/913/docs/practsyll.pdf. One fundamental problem with this idea is that it is misguided. Every test has many results. No one specifies them all. An "expected result" points the tester at one or a few of these results, but away from the others. For example, suppose we test a program that adds numbers. Give it 2+3 (the intended inputs) and get back 5 (the monitored output). This looks like a passing test, but suppose that the test took 6 hours, or that there was a memory leak, or that it erased a file on the hard disk. If all we compare against is the expected result (5), we won't notice these other bugs.

The problem of an observer (such as a tester) missing obvious failures is not just theoretical. I've seen it at work, and psychologists have described it in research, naming it inattentional blindness (Mack & Rock, 2000). If you are focusing your attention on something else, then you have a significant probability of completely missing a surprising event that happens right in front of you. Some striking demos are available at http://viscog.beckman.uiuc.edu/djs_lab/demos.html.

Does this mean that it is a bad idea to develop expected results? No, of course not. It means that there are pluses and minuses in using, and relying on, expected results.

Expected results are valuable when an answer is difficult or time-consuming to compute. They are valuable when testing is done by junior staff who don't know the product (but realize that these people are especially likely to miss other failures). They are valuable for the test planner who is thinking through how the product works while she design the tests—working the tests through to their predicted conclusions is an important tool for learning the product. They are of course valuable for automated tests—but the ability to automatically derive an expected result is much more valuable for automated testing (because it is so much faster) than the ability of a human to write the result down.