ABSTRACT
Software testing of any kind is a process of sampling. Each
test case for a piece of software must be chosen from a set
of possibilities, here called the test domain. (See the formal
definitions, to follow.) It is a basic fact of practical testing
that this domain is almost always hopelessly large, so that
the number of test samples possible during software devel-
opment is a minuscule fraction of the possibilities. Thus the
choice of test cases is the critical factor in any testing
activity. At the same time, once software has been released
its users proceed to sample the test domain, and may do so
far more thoroughly than the developer could. A trivial
example will illustrate this unfortunate situation. Imagine a
program that makes a numerical calculation using two
non-negative integer inputs. Further stipulate that the pro-
gram correctly checks that its inputs are integers in the
range [0, 232 – 1], issuing a proper message otherwise. This
confines the test domain to about 264 values. (Without the
stipulation, the test domain would be effectively unlimited,
since it would include unlimited strings for inputs,
e.g., "2B or not 2B, that is the ?". It is very often the case that programs fail to check for valid input.) Suppose
also that the developer has a collection of 104 integer test
cases for which the required results are known, and has time
to execute and check these. Discount any time required to
fix problems that are encountered. (Again, the situation is
idealized because typically no such collection may exist,
and weeks of test time may not be available; fixing pro-
blems could be a lengthy operation.) Finally let this soft-
ware be intended for the PC market with about 10 million
users, each of whom uses the program 100 times a week on
average. Altogether the idealized situation differs from
reality in giving development testing a huge favorable
bias. The test domain has about 4.3 billion points, of
which 104, about 0.00023%, are tested, while usage is
about 109 points/week, about 23% of the test domain. (It
is true that a great deal depends on exactly which elements
of the test domain are tried and which used, but that
important subject will be raised later in the “Functional
Subdomain testing” section.) Thus in this case some user
hits almost every point within a month of usage, while only
a negligible fraction have been tested. Thus if there are any
problems with the software, it is quite likely that they will
be seen by users but not by prerelease testers.
