This comprehensive, interdisciplinary handbook reviews the latest methods and technologies used in automated essay evaluation (AEE) methods and technologies. Highlights include the latest in the evaluation of performance-based writing assessments and recent advances in the teaching of writing, language testing, cognitive psychology, and computational linguistics. This greatly expanded follow-up to Automated Essay Scoring reflects the numerous advances that have taken place in the field since 2003 including automated essay scoring and diagnostic feedback. Each chapter features a common structure including an introduction and a conclusion. Ideas for diagnostic and evaluative feedback are sprinkled throughout the book. 

Highlights of the book’s coverage include:

  • The latest research on automated essay evaluation.
  • Descriptions of the major scoring engines including the E-rater®, the Intelligent Essay Assessor, the Intellimetric™ Engine, c-rater™, and LightSIDE.
  • Applications of the uses of the technology including a large scale system used in West Virginia.
  • A systematic framework for evaluating research and technological results.
  • Descriptions of AEE methods that can be replicated for languages other than English as seen in the example from China.
  • Chapters from key researchers in the field.

The book opens with an introduction to AEEs and a review of the "best practices" of teaching writing along with tips on the use of automated analysis in the classroom.  Next the book highlights the capabilities and applications of several scoring engines including the E-rater®, the Intelligent Essay Assessor, the Intellimetric™ engine, c-rater™, and LightSIDE. Here readers will find an actual application of the use of an AEE in West Virginia, psychometric issues related to AEEs such as validity, reliability, and scaling, and the use of automated scoring to detect reader drift, grammatical errors, discourse coherence quality, and the impact of human rating on AEEs. A review of the cognitive foundations underlying methods used in AEE is also provided. The book concludes with a comparison of the various AEE systems and speculation about the future of the field in light of current educational policy.

Ideal for educators, professionals, curriculum specialists, and administrators responsible for developing writing programs or distance learning curricula, those who teach using AEE technologies, policy makers, and researchers in education, writing, psychometrics, cognitive psychology, and computational linguistics, this book also serves as a reference for graduate courses on automated essay evaluation taught in education, computer science, language, linguistics, and cognitive psychology.

chapter 1|15 pages

Introduction to Automated Essay Evaluation

ByMark D. Shermis, Jill Burstein, Sharon Apel Bursky

chapter 2|20 pages

Automated Essay Evaluation and the Teaching of Writing

ByNorbert Elliot, Andrew Klobucar

chapter 3|19 pages

English as a Second Language Writing and Automated Essay Evaluation

BySara C. Weigle

chapter 4|13 pages

The E-rater® Automated Essay Scoring System

ByJill Burstein, Joel Tetreault, Nitin Madnani

chapter 5|21 pages

Implementation and Applications of the Intelligent Essay Assessor

ByPeter W. Foltz, Lynn A. Streeter, Karen E. Lochbaum, Thomas K. Landauer

chapter 7|25 pages

Applications of Automated Essay Evaluation in West Virginia

ByChanghua S. Rich, M. Christina Schneider, Juan M. D’Brot

chapter 8|12 pages


Open Source Machine Learning for Text
ByElijah Mayfield, Carolyn Penstein Rosé

chapter 9|17 pages

Automated Short Answer Scoring

Principles and Prospects
ByChris Brew, Claudia Leacock

chapter 10|28 pages

Probable Cause

Developing Warrants for Automated Scoring of Essays
ByDavid M. Williamson

chapter 11|18 pages

Validity and Reliability of Automated Essay Scoring

ByYigal Attali

chapter 12|22 pages

Scaling and Norming for Automated Essay Scoring

ByKristin L. K. Koskey, Mark D. Shermis

chapter 13|12 pages

Human Ratings and Automated Essay Evaluation

ByBrent Bridgeman

chapter 14|18 pages

Using Automated Scoring to Monitor Reader Performance and Detect Reader Drift in Essay Scoring

BySusan M. Lottridge, E. Matthew Schulz, Howard C. Mitzel

chapter 15|16 pages

Grammatical Error Detection in Automatic Essay Scoring and Feedback

ByMichael Gamon, Martin Chodorow, Claudia Leacock, Joel Tetreault

chapter 16|14 pages

Automated Evaluation of Discourse Coherence Quality in Essay Writing

ByJill Burstein, Joel Tetreault, Martin Chodorow, Daniel Blanchard, Slava Andreyev

chapter 17|17 pages

Automated Sentiment Analysis for Essay Evaluation

ByJill Burstein, Beata Beigman-Klebanov, Nitin Madnani, Adam Faulkner

chapter 18|15 pages

Covering the Construct

An Approach to Automated Essay Scoring Motivated by a Socio-Cognitive Framework for Defining Literacy Skills
ByPaul Deane

chapter 19|34 pages

Contrasting State-of-the-Art Automated Scoring of Essays

ByMark D. Shermis, Ben Hamner

chapter 20|7 pages

The Policy Turn in Current Education Reform

The Common Core State Standards and Its Linguistic Challenges and Opportunities
ByKenji Hakuta