Keywords
Standard Setting, Nominal Response Model, Two-Parameter Logistic IRT, Embedded Standard Setting
Abstract
The validity of score interpretation and application for specific purposes is dependent upon both the appropriate assignment of test scores and the methods through which those scores are given meaning by defining performance benchmarks. This study seeks to examine the combined utility of various scoring (i.e., number correct, two-parameter logistic, and nominal response model) and standard setting methods in accurately predicting performance on an external assessment. The study will highlight the implementation of the embedded standard setting method, a newer standard setting procedure that is not yet well represented in the literature. Embedded standard setting will be compared to a variation of the bookmark method and to variation of norm-referenced standard setting, the equi-distribution method.
The results of the study suggest that each of the combined approach performance levels, which were evaluated separately in respective multilevel logistic regression analyses, were statistically significant predictors of performance on the statewide assessment as defined by whether a student passed or did not pass the assessment. Each model estimation explained additional variance in the overall outcome, as suggested by the pseudo-R square values, over the covariates only model that was used as a reference for comparison. Outcomes of the regression analyses for each scoring and standard setting combined approach, including statistical significance of the predictors, model fit, and classification accuracy, are presented and discussed.
Completion Date
2024
Semester
Fall
Committee Chair
Sivo, Stephen
Degree
Doctor of Philosophy (Ph.D.)
College
College of Community Innovation and Education
Department
Learning Sciences and Educational Research
Degree Program
Methodology, Measurement & Analysis
Format
Identifier
DP0029024
Language
English
Release Date
12-15-2024
Access Status
Dissertation
Campus Location
Orlando (Main) Campus
STARS Citation
Paul, Michelle, "A Comparison of Combined Scoring and Standard Setting Approaches Applied to an Interim Assessment in Predicting Statewide Assessment Performance" (2024). Graduate Thesis and Dissertation post-2024. 59.
https://stars.library.ucf.edu/etd2024/59
Accessibility Status
PDF accessibility verified using Adobe Acrobat Pro Accessibility Checker