ABSTRACT
When new developments occur in technology, it is common to say that they cannot be compared sensibly to the “old way” of doing things. For example, comparing word processing to typewriters was ultimately an unproductive enterprise because word processing provided many more functions than a simple typewriter, rendering any direct contrast of results partial and unconvincing. It is also true that technologists may themselves resist applying quality criteria to their new enterprises. For some, creating a proof of concept equals a proof of value. For example, in the 1980s, it was sufficient to demonstrate that an artificial intelligence (AI) system “ran” as opposed to its achieving high degrees of accuracy in its analysis. AI researchers were unwilling to consider evaluating the impact of their work in part because the process of making the system seemed as important as its potential outcomes. In fact, much of new technology has not been systematically evaluated by scientific methods, a process largely bypassed because of the speed of change and the expanding consumer market (Baker & Herman, in press).
