The probability of exceedance as a nonparametric person-fit statistic for tests of moderate length

Abstract

To classify an item score pattern as not fitting a nonparametric item response theory (NIRT) model, the probability of exceedance (PE) of an observed response vector x can be determined as the sum of the probabilities of all response vectors that are, at most, as likely as x, conditional on the test’s total score. Vector x is to be considered not fitting when its PE is smaller than a prespecified level. Although this concept is not new, it is hardly if ever applied in practice. In the present paper, the authors show how the PE of a response vector x can be computed in a NIRT context and how misfitting response patterns are detected using the exact distribution of PE. Results from two empirical applications are discussed. A simulation study is conducted to investigate the robustness of the PE against violation of the invariant item ordering condition. Finally, considerations over possible asymptotic distributions of PE are discussed.

Publication
Applied Psychological Measurement, 37