ABSTRACT
Determining the accuracy of question items in an adaptive learning system is often approached as more of an art than a science—especially when the early data from question item analysis comes from a small sample set. This study examines the use of methods for evaluating the effective measure of student learning in an adaptive system, but with a significantly smaller sample size. The goal is to determine the effectiveness of early analysis from small cohorts.
This chapter is a continuation of work on developing effective and efficient methods for measuring assessments within adaptive learning systems. What is presented here builds upon findings published at the Human–Computer Interaction conference in 2021. For that publication and conference, a guide was produced for determining the validity, reliability, and standardization of questions asked by an adaptive learning system. In that work, the approach of using psychometric principles to evaluate question items was found to be effective as an improvement process for the accuracy of adaptive learning content and assessments. This chapter will publish the results of a later study on the use of question item reliability data to estimate the conditional probability that a question item may need refinement to measure learning more accurately. It is our intent to convey these positive findings for more stakeholders in the adaptive learning community.
