ABSTRACT
Artificial Intelligence in Education research for STEM domains has largely been quantitative in nature, but qualitative research offers several advantages as part of a mixed-methods approach. In particular, qualitative research enables researchers to develop deeper phenomenological understanding of how learners represent their activity to themselves. However, qualitative research can be challenging to apply in classrooms: it is resource-intensive, does not scale well, and the phenomena of the greatest interest to AIEd researchers are often intermittent and occasional. For example, researchers may be interested in studying situations where a learning activity is known to be overly time-consuming or difficult, or in theoretical investigations of shifts in student affect such as transitions from confusion to frustration. However, given multiple potential learners to interview (e.g., a classroom of students), it can be difficult for a researcher embedded in the classroom to prioritize which learner to speak with next. Simple strategies, whether sequential or random, may miss (often fleeting) key moments in a participant's experience (e.g., affective transitions).
We address this problem with a new app that leverages user modeling techniques (e.g., behavior and affect-sensing) to direct interviewers to learners at critical, theory-driven moments as they learn with AIEd technologies in the classroom. This paper details the design and implementation of this research paradigm as an alternative method for studying learning and using existing STEM AIEd technologies in research. We examine the potential of this paradigm through the lens of two case studies where 99 students interacted with a computer-based learning environment as part of their regular classroom instruction. Unscripted interviews were triggered at or immediately after critical moments (such as peak frustration or shifts from confusion to boredom). The app facilitated 594 interviews, each averaging 1–2 minutes in length. Our findings indicate that by using machine learned models to optimize researcher time, we can gain a deeper insight into students’ behaviors and their motivations, thus furthering AIEd research. We discuss the potential broader applications of this app and the research it affords.
