ABSTRACT
This chapter articulates how the capability of digital learning systems to collect data as people use them, coupled with advances in data science, offers exciting opportunities to improve learning. The learning data available for research has expanded dramatically in terms of the amount of detail around individuals’ learning processes, the number of people learning with systems that generate such data, and the ability to run a series of rapid experiments online. Learning system data can be used in feedback loops that inform the design of learning technology products, students’ future learning approaches, instructor practices, and the knowledge base around human learning. Bringing together teams of collaborators with different kinds of expertise—teaching, subject-matter knowledge, instructional design, and data analysis—is a prerequisite for realizing the full potential of learning system data. This Collaborative Data-Intensive Improvement Research (CDIR) signals commitment to doing research with, rather than on, educators and to using data to help education systems improve. This chapter describes several CDIR collaborations, including the successes and the challenges encountered in their execution. These include the rush to premature impact evaluation, difficulty synchronizing research activities with product development cycles, and a lack of evaluation capacity within educational institutions. These challenges and possibilities for ameliorating them will be discussed from an organizational change perspective. Finally, this chapter makes a case for the importance of obtaining consensus around key learner outcomes and of having valid and reliable methods for assessing those outcomes.
