Computerized Adaptive Testing: A Primer

Item Pool Qualitydesign lever

The degree to which the pool of candidate items is well written, sensitivity-reviewed, pretested, covers a wide range of difficulty for each content area, and conforms to the assumptions of the psychometric model used for calibration and scoring.

IRT Model Fit / Unidimensionalitydesign lever

The extent to which the chosen item response theory model adequately describes examinee responses, including conformity to assumptions of unidimensionality, local (conditional) independence, item fungibility, and absence of differential item functioning.

Item Selection Algorithmdesign lever

The set of rules determining which item is presented first, which item follows each response, and when testing stops, typically maximizing information or expected posterior precision near the current proficiency estimate subject to content and exposure constraints.

Content Balancing Constraintsdesign lever

Constraints imposed on item selection so that each examinee receives a comparable mixture of items across content subdomains and informal contexts, preserving comparability of scores and content validity despite individualized item administration.

Item Exposure Controldesign lever

Mechanisms that limit how frequently any item is administered, either proportionally or in absolute number, to flatten the item usage distribution and reduce the chance that memorized items materially inflate examinee scores.

Administration Medium (Computer vs Paper)contextual condition

The contextual condition of whether items are presented and responded to via computer versus paper-and-pencil, which can change the cognitive task, difficulty, timing, and response process for some items, producing medium (mode) effects.

Continuous Testing Demandcontextual condition

The economically and operationally imposed requirement that computerized tests be administered continuously over time rather than at a few fixed mass-administration dates, shaping examinee access, item exposure, and security challenges.

Proficiency Estimation Accuracypsychological state

The accuracy and stability with which an examinee's latent proficiency is estimated during and at the end of an adaptive test, reflected in the width of the posterior distribution or the standard error of the proficiency estimate.

Examinee Engagement and Comfortpsychological state

The examinee's behavioral and affective state during testing—being challenged but not discouraged, avoiding boredom from too-easy items or frustration from too-hard items, and freedom from undue test or computer anxiety.

Measurement Precision / Informationoutcome metric

The amount of test information (inverse of error variance) achieved across the proficiency range, indicating how precisely the test measures each examinee and how efficiently it uses items.

Score Comparability / Equating Qualityoutcome metric

The degree to which scores from different adaptive forms, and from CAT versus legacy paper-and-pencil tests, can be placed on a common scale and used interchangeably, satisfying same-construct, equity, population invariance, and symmetry conditions.

Validity of Score Inferencesoutcome metric

The appropriateness, meaningfulness, and usefulness of the specific inferences (selection, placement, prediction) made from test scores, encompassing content, construct, and criterion-related validity and freedom from threats such as differential item functioning.

Test Securityoutcome metric

The degree to which the item pool is protected against compromise, theft, and preknowledge that would allow examinees to inflate scores without commensurate proficiency, threatening score validity.