Automated Scoring of Complex Tasks in Computer-Based Testing

This is the first volume to provide the latest methods and examples of "best practices" in the design, implementation, and evaluation of automated scoring for complex assessments. The contributing authors, all noted leaders in the field, introduce each m

The use of computers and the Internet in the testing community has expanded the opportunity for innovative testing. Until now, there was no one source that reviewed the latest methods of automated scoring for complex assessments. This is the first volume to provide that coverage, along with examples of "best practices" in the design, implementation, and evaluation of automated complex assessment. The contributing authors, all noted leaders in the field, introduce each method in the context of actual applications in real assessments so as to provide a realistic view of current industry practices. Evidence Centered Design, an innovative approach to assessment design, is used as the book's conceptual framework. The chapters review both well known methods for automated scoring such as rule-based logic, regression-based, and IRT systems, as well as more recent procedures such as Bayesian and neural networks. The concluding chapters compare and contrast the various methods and provide a vision for the future. Each chapter features a discussion of the philosophical and practical approaches of the method, the associated implications for validity, reliability, and implementation, and the calculations and processes of each technique. Intended for researchers, practitioners, and advanced students in educational testing and measurement, psychometrics, cognitive science, technical training and assessment, diagnostic, licensing, and certification exams, and expert systems, the book also serves as a resource in advanced courses in educational measurement or psychometrics.

Contents: Preface. D.M. Williamson, I.I. Bejar,R.J. Mislevy, Automated Scoring of Complex Tasks in Computer-Based Testing: An Introduction. R.J. Mislevy, L.S. Steinberg, R.G. Almond, J.F. Lukas, Concepts, Terminology, and Basic Models of Evidence-Centered Design. I.I. Bejar, D.M. Williamson, R.J. Mislevy, Human Scoring. H. Braun, I.I. Bejar, D.M. Williamson, Rule-Based Methods for Automated Scoring: Application in a Licensing Context. M.J. Margolis, B.E. Clauser, A Regression-Based Procedure for Automated Scoring of a Complex Medical Performance Assessment. H. Wainer, L.M. Brown, E.T. Bradlow, X. Wang, W.P. Skorupski, J. Boulet, R.J. Mislevy, An Application of Testlet Response Theory in the Scoring of a Complex Certification Exam. D.M. Williamson, R.G. Almond, R.J. Mislevy,R. Levy, An Application of Bayesian Networks in Automated Scoring of Computerized Simulation Tasks. R.H. Stevens, A. Casillas, Artificial Neural Networks. P. Deane, Strategies for Evidence Identification Through Linguistic Assessment of Textual Responses. K. Scalise, M. Wilson, Analysis and Comparison of Automated Scoring Approaches: Addressing Evidence-Based Assessment Principles. R.E. Bennett, Moving the Field Forward: Some Thoughts on Validity and Automated Scoring.