ABSTRACT
As we have gained experience evaluating PROUST, we have found that addressing the two evaluation questions leads to a somewhat different perspective on evaluation from that of traditional educational evaluation. Traditional educational evaluation consists of two main categories, formative and summative evaluation. Designers of educational technology use formative evaluation to define and refine their goals and methods during the design process. They use summative evaluation to determine whether a finished educational product is effective after it has been built. Because building ITSs is still somewhat an art, and because there are few ITSs that can be called "finished," designers of ITSs are currently more concerned with usefully guiding the development of their systems than with determining whether they are effective educational end products. At least for the time being, then, the idea of formative evaluation seems more appropriate for designers of ITSs than does the idea of summative evaluation. Hence, we have formulated Evaluation Question I and Evaluation Question 2 to be much more focused on the development of ITSs than on determining whether they are effective educational end products.
