ABSTRACT

"Automated scoring engines […] require a careful balancing of the contributions of technology, NLP, psychometrics, artificial intelligence, and the learning sciences. The present handbook is evidence that the theories, methodologies, and underlying technology that surround automated scoring have reached maturity, and that there is a growing acceptance of these technologies among experts and the public."

From the Foreword by Alina von Davier, ACTNext Senior Vice President

Handbook of Automated Scoring: Theory into Practice provides a scientifically grounded overview of the key research efforts required to move automated scoring systems into operational practice. It examines the field of automated scoring from the viewpoint of related scientific fields serving as its foundation, the latest developments of computational methodologies utilized in automated scoring, and several large-scale real-world applications of automated scoring for complex learning and assessment systems.  The book is organized into three parts that cover (1) theoretical foundations, (2) operational methodologies, and (3) practical illustrations, each with a commentary. In addition, the handbook includes an introduction and synthesis chapter as well as a cross-chapter glossary.

chapter 1|10 pages

The Past, Present, and Future of Automated Scoring

ByPeter W. Foltz, Duanli Yan, André A. Rupp

part Part I|158 pages

Part I: Theoretical Foundations

chapter 2|16 pages

Cognitive Foundations of Automated Scoring

ByMalcolm I. Bauer, Diego Zapata-Rivera

chapter 3|20 pages

Assessment Design with Automated Scoring in Mind

ByKristen DiCerbo, Emily Lai, Ventura Matthew

chapter 4|20 pages

Human Scoring with Automated Scoring in Mind

ByEdward W. Wolfe

chapter 5|24 pages

Natural Language Processing for Writing and Speaking

ByAoife Cahill, Keelan Evanini

chapter 6|20 pages

Multimodal Analytics for Automated Assessment

BySidney K. D’Mello

chapter 7|20 pages

International Applications of Automated Scoring

ByMark D. Shermis

chapter 8|18 pages

Public Perception and Communication around Automated Essay Scoring

ByScott W. Wood

part Part II|158 pages

Part II: Operational Methodologies

chapter 10|24 pages

Operational Human Scoring at Scale

ByKathryn L. Ricker-Pedley, Susan Hines, Carolyn Connelly

chapter 11|22 pages

System Architecture Design for Scoring and Delivery

BySue Lottridge, Nick Hoefer

chapter 12|24 pages

Design and Implementation for Automated Scoring Systems

ByChristina Schneider, Michelle Boyer

chapter 13|22 pages

Quality Control for Automated Scoring in Large-Scale Assessment

ByDan Shaw, Brad Bolender, Rick Meisner

chapter 14|20 pages

A Seamless Integration of Human and Automated Scoring

ByKyle Habermehl, Aditya Nagarajan, Scott Dooley

chapter 15|14 pages

Deep Learning Networks for Automated Scoring Applications

BySaad M. Khan, Yuchi Huang

chapter 16|22 pages

Validation of Automated Scoring Systems

ByDuanli Yan, Brent Bridgeman

part Part III|162 pages

Part III: Practical Illustrations

chapter 18|18 pages

Expanding Automated Writing Evaluation

ByJill Burstein, Brian Riordan, Daniel McCaffrey

chapter 19|18 pages

Automated Writing Process Analysis

ByPaul Deane, Mo Zhang

chapter 20|18 pages

Automated Scoring of Extended Spontaneous Speech

ByKlaus Zechner, Anastassia Loukina

chapter 21|20 pages

Conversation-Based Learning and Assessment Environments

ByArthur C. Graesser, Xiangen Hu, Vasile Rus, Zhiqiang Cai

chapter 22|20 pages

Automated Scoring in Intelligent Tutoring Systems

ByRobert J. Mislevy, Duanli Yan, Janice Gobert, Michael Sao Pedro

chapter 23|22 pages

Scoring of Streaming Data in Game-Based Assessments

ByRussell G. Almond

chapter 24|24 pages

Automated Scoring in Medical Licensing

ByMelissa J. Margolis, Brian E. Clauser

chapter 25|6 pages

At the Birth of the Future: Commentary on Part III

ByJohn T. Behrens

chapter 26|14 pages

Theory into Practice: Reflections on the Handbook

ByAndré A. Rupp, Peter W. Foltz, Duanli Yan