ABSTRACT
As pervasive information systems weave their way into society, it is critical that these new systems are accepted and utilized. However, it is difficult to determine what makes for a good design and a successful interaction because evaluation methodologies and metrics are in their infancy for these types of systems. The complexity and diversity of these systems has made it difficult to establish common evaluation techniques and practices. However, the necessity for such a framework is overwhelmingly apparent. A framework will make it easier for researchers to learn from each other’s results, create effective discount evaluation techniques and design guidelines for pervasive computing, provide a mechanism for researchers to share what they have learned about the appropriateness of different evaluation techniques, and provide structure so that key areas of evaluation are not overlooked.
In this chapter, we present a framework of areas of evaluation for pervasive information systems. The framework includes nine evaluation areas that include elements of usability, interaction, and values (such as privacy and trust). We present sample metrics and measures and examples for each area from the literature. We review a number of methodologies that have been used in evaluation and provide a case study of an evaluation using a number of the evaluation areas in the framework. We conclude with a discussion of future needs to enable researchers to share evaluation results.
