Skip to main content

Pitfalls of relying on/or looking only for "Expected" behavior in Testing

Testers should specify the expected result of every test, in advance?
This is another guidance from Myers (1979) that has had lasting influence. See, for example, ISEB's current syllabus for test practitioner certification at One fundamental problem with this idea is that it is misguided. Every test has many results. No one specifies them all. An "expected result" points the tester at one or a few of these results, but away from the others. For example, suppose we test a program that adds numbers. Give it 2+3 (the intended inputs) and get back 5 (the monitored output). This looks like a passing test, but suppose that the test took 6 hours, or that there was a memory leak, or that it erased a file on the hard disk. If all we compare against is the expected result (5), we won't notice these other bugs.

The problem of an observer (such as a tester) missing obvious failures is not just theoretical. I've seen it at work, and psychologists have described it in research, naming it inattentional blindness (Mack & Rock, 2000). If you are focusing your attention on something else, then you have a significant probability of completely missing a surprising event that happens right in front of you. Some striking demos are available at

Does this mean that it is a bad idea to develop expected results? No, of course not. It means that there are pluses and minuses in using, and relying on, expected results.

Expected results are valuable when an answer is difficult or time-consuming to compute. They are valuable when testing is done by junior staff who don't know the product (but realize that these people are especially likely to miss other failures). They are valuable for the test planner who is thinking through how the product works while she design the tests—working the tests through to their predicted conclusions is an important tool for learning the product. They are of course valuable for automated tests—but the ability to automatically derive an expected result is much more valuable for automated testing (because it is so much faster) than the ability of a human to write the result down.
Post a Comment

Popular posts from this blog

Compact and Repair an Access Database. Add Ref. to : AdoDb, Jro

< ?xml version="1.0" encoding="utf-8" ?>

using ADODB;
using JRO;
using System.Configuration;
using System.Data.OleDb;
using System.IO;

public class CompactAndRepairAccessDb : System.Windows.Forms.Form
private System.ComponentModel.Container components = null;
private JRO.JetEngine jro;
private System.Windows.Forms.Button btnConfirm;
private System.Windows.Forms.TextBox tbxOriginalDbSize;
private System.Windows.Forms.TextBox tbxCompactedDbSize;
private OleDbConnection cnn;

public CompactAndRepairAccessDb() {

FileInfo fi = new FileInfo( ConfigurationSettings.AppSettings["PathOriginal"] );
int s = Convert.ToInt32( fi.Length/1000 );
this.tbxOriginalDbSize.Text = s.ToString() + " kb";

private void btnConfirm_Click(object sender, System.EventArgs e) {
// First close all instances of the database

VBScript to Automate login into gmail

Dim IE
Dim crtScreen
Set IE = CreateObject("InternetExplorer.Application")
USERNAME = "saudaziz"

With IE
.navigate ""
End With

'wait a while until IE as finished to load
Do while IE.busy
set WshShell = WScript.CreateObject("WScript.Shell")
Do While UCase(IE.Document.readyState) <> "COMPLETE"
WScript.Sleep 100
set WshShell=nothing
IE.document.all.Item("Email").value = USERNAME
IE.document.all.Item("pASSWD").value =pASSWORD
Set IE = Nothing