Computerized Multistage Testing Theory and Applications

Devising tests that evaluate a nation’s educational standing and implement efficacious educational reforms requires a careful balance among the contributions of technology, psychometrics, test design, and the learning sciences. Unlike other forms of adaptive testing, multistage testing (MST) is highly suitable for testing educational achievement because it can be adapted to educational surveys and student testing. Computerized Multistage Testing: Theory and Applications covers the methodologies, underlying technology, and implementation aspects of this type of test design. The book discusses current scientific perspectives and practical considerations for each step involved in setting up an MST program. It covers the history of MST, test design and implementation for various purposes, item pool development and maintenance, IRT-based and classical test theory-based methodologies for test assembly, routing and scoring, equating, test security, and existing software. It also explores current research, existing operational programs, and innovative future assessments using MST. Intended for psychologists, social scientists, and educational measurement scientists, this volume provides the first unified source of information on the design, psychometrics, implementation, and operational use of MST. It shows how to apply theoretical statistical tools to testing in novel and useful ways. It also explains how to explicitly tie the assumptions made by each model to observable (or at least inferable) data conditions. Winner of the 2016 AERA Award for Significant Contribution to Educational Measurement and Research Methodology The 2016 American Education Research Association (AERA) Div. D award committee for Significant Contributions to Educational Measurement and Research Methodology has recognized unanimously this collaborative work advancing the theory and applications of computerized MST. This annual award recognizes published research judged to represent a significant conceptual advancement in the theory and practice of educational measurement and/or educational research methodology. The 2016 award was made under the heading: Measurement, Psychometrics, and Assessment. This collective work, published in 2014 as an edited volume titled Computerized Multistage Testing: Theory and Applications, was cited by the committee both for the originality of the conceptual foundations presented in support of multistage testing and for arguing persuasively for its potential impact on the practice of educational measurement.

Test Design, Item Pool, and Maintenance Overview of Computerized Multistage Tests Duanli Yan, Charles Lewis, and Alina A. von DavierLinear Tests and Computerized Adaptive Tests (CATs) Multistage Tests (MSTs) MST Designs for Different Purposes Implementation Schemes Designing MST Content Balance and Assembly Exposure Control Routing Scoring, Linking, and Equating Reliability, Validity, Fairness, and Test Security Current and Future Applications Logistic Challenges Summary Multistage Test Designs: Moving Research Results into Practice April L. Zenisky and Ronald K. HambletonThe MST Design Structure The State of Research: MST Development and Design ConsiderationsConclusions and Next Steps Item Pool Design and Maintenance for Multistage Testing Bernard P. VeldkampDesigning an Item Pool BlueprintApplications in Item Writing Maintenance Discussion Mixed-Format Tests: Issues and Methods Jiseon Kim and Barbara G. DoddLiterature Review on Design Components in Mixed-Format MST Comparing Other Testing Approaches Issues and Future Research Suggestions for Mixed-Format MSTConclusion Design and Implementation of Large-Scale Multistage Testing Systems Richard LuechtMST Design and Implementation Considerations Conclusions: A Research Agenda Test Assembly Overview of Test Assembly Methods in Multistage Testing Yi Zheng, Chun Wang, Michael J. Culbertson, and Hua-Hua ChangMST Framework MST Assembly Design Automated Assembly for MSTSetting Difficulty Anchors and Information Targets for Modules "On-the-Fly" MST (OMST) Assembly ParadigmMST, CAT, and Other Designs—Which Way to Go? Using a Universal Shadow-Test Assembler with Multistage Testing Wim J. van der Linden and Qi DiaoSolving Shadow-Test Assembly Problems Basic Design ParametersDifferent Testing Formats Relative Efficiency of Formats Empirical Study Concluding Comments Multistage Testing by Shaping Modules on the Fly Kyung (Chris) T. Han and Fanmin GuoMST by Shaping MST-S versus MST-R versus CAT Discussion and Conclusion Optimizing the Test Assembly and Routing for Multistage Testing Angela Verschoor and Theo EggenOptimizing MST Assembly: A Nonexhaustive Search Limited Item Pools, Two- and Three-Parameter Models Discussion Routing, Scoring, and Equating IRT-Based Multistage Testing Alexander WeissmanIntroduction Motivation for Tailored Testing Routing Rules Scoring and Classification Methodologies Final Comments A Tree-Based Approach for Multistage Testing Duanli Yan, Charles Lewis, and Alina A. von DavierRegression Trees Tree-Based Computerized Adaptive Tests Tree-Based Multistage Testing AlgorithmAn ApplicationDiscussion Limitations and Future Research Multistage Testing for Categorical Decisions Robert Smith and Charles LewisComputer-Mastery Methods Information Targeted at Cut versus at Ability Influence of Multiple Cut Scores Factors That Can Reduce Optimal Solutions Example Based on Smith and Lewis (1995) Adaptive Mastery Multistage Testing Using a Multidimensional IRT Model C.A.W. GlasIntroduction Definition of the Decision Problem Computation of Expected Loss and Risk Using Backward InductionSelection of Items and Testlets Simulation Studies Conclusions and Further Research Multistage Testing Using Diagnostic Models Matthias von Davier and Ying (Alison) ChengThe DINA Model and the General Diagnostic Model Experience with CD-CATs CD-MSTs Discussion Considerations on Parameter Estimation, Scoring, and Linking in Multistage Testing Shelby J. Haberman and Alina A. von DavierNotation The Item Response Model The Test ScoreApproaches to Parameter Estimation Conclusions Test Reliability, Validity, Fairness, and Security Reliability of Multistage Tests Using Item Response Theory Peter W. van RijnTest ReliabilityApplication: IRT Reliability for MST in NAEP Conclusion Multistage Test Reliability Estimated via Classical Test Theory Samuel A. Livingston and Sooyeon KimThe Estimation Procedure Testing the Accuracy of the Estimation Procedure How Accurate Were the Estimates? Evaluating Validity, Fairness, and Differential Item Functioning in Multistage TestingRebecca Zwick and Brent BridgemanContent Balancing Opportunities for Item Review and Answer Changing Skipping Strategies MST Routing Algorithms The Digital Divide Comparability of Computer Platforms Accommodations for Students with Disabilities and English Language Learners Differential Item Functioning Analysis in MSTs Application of the Empirical Bayes DIF Approach to Simulated MST Data Summary Test Security and Quality Control for Multistage Tests Yi-Hsuan Lee, Charles Lewis, and Alina A. von DavierAn Overview of a Three-Component Procedure Tools to Evaluate Test Security and Quality Control A Simulation Study Using CUSUM Statistics to Monitor Item Performance Discussion Applications in Large-Scale Assessments Multistage Test Design and Scoring with Small Samples Duanli Yan, Charles Lewis, and Alina A. von DavierSmall Data Sample Item Pool Various MST Module DesignsRouting and Scoring Comparisons of the Six MST Designs Discussion Limitations and Future Research The Multistage Test Implementation of the GRE Revised General Test Frédéric Robin, Manfred Steffen, and Longjuan LiangFrom CAT to MST MST Design Implementation Monitoring Summary The Multistage Testing Approach to theAICPA Uniform Certified Public Accounting Examinations Krista J. Breithaupt, Oliver Y. Zhang, and Donovan R. HareResearch on Multistage Testing Item Bank Development for MST Content Security Monitoring for MST Inventory Exposure Planning for MST Discussion Transitioning a K–12 Assessment from Linear to Multistage Tests Carolyn Wentzel, Christine M. Mills, and Kevin C. MearaAdministering CTP Items Online Creating a New MST Scale Using IRTMultistage-Adaptive Test DevelopmentScore Reporting Summary AMultistage TestingApproach to Group-Score Assessments Andreas Oranje, John Mazzeo, Xueli Xu, and Edward KulickTargeted Testing Goals of the Study Methods Results Discussion Controlling Multistage Testing Exposure Rates inInternational Large-Scale AssessmentsHaiwen Chen, Kentaro Yamamoto, and Matthias von DavierItem Exposure Rate Control for Multistage Adaptive AssessmentsMethod: How to Compute and Adjust the Item Exposure Rates Data Results Conclusion Software Tools for Multistage Testing Simulations Kyung (Chris) T. Han and Michal KosinskiMSTGen R Conclusions Closing Remarks Past and Future of Multistage Testing in Educational Reform Isaac I. BejarFuture of MST A Model-Based Three-Stage Design Item Generation and Automated Scoring and Broadly Accessible Test Content Summary and Conclusions Bibliography

Duanli Yan, Alina A. von Davier, Charles Lewis