--- title: "How often does a rater screen give the right warning?" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{How often does a rater screen give the right warning?} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include=FALSE} is_cran_check <- !isTRUE(as.logical(Sys.getenv("NOT_CRAN", "false"))) knitr::opts_chunk$set(eval = !is_cran_check, collapse = TRUE, comment = "#>", dev.args = list(bg = "white")) library(mfrmr) ``` An assessment coordinator wants to use fit statistics when giving feedback to raters. Before choosing a warning rule, two questions matter: how often does it warn about a rater whose ratings follow the intended model, and how often does it detect a specified departure? A difference in severity alone is not such a departure. A consistently strict rater can fit a model that estimates severity. `mfrm_screening_performance()` summarizes flags against **known simulation truth**. It accepts outcomes from your own generators, fitting methods and screening rules. It does not determine which rule is appropriate or label real raters as good or bad. In an actual assessment, unobserved rater quality cannot be supplied as known truth simply by accepting the screen's own verdict. ## Keep failed and incomplete trials in the planned roster Declare the conditions, independent replications and targets before running the study. `Affected = TRUE` means a target has the particular departure the screen is meant to detect. Put different designs or methods in separate conditions. Within each condition, keep the same targets and truth labels. This small invented example has two unaffected raters and four planned trials: ```{r incomplete-example} roster <- expand.grid(Condition = "Null", Replicate = 1:4, Target = c("R1", "R2"), stringsAsFactors = FALSE) roster$Affected <- FALSE results <- roster[c("Condition", "Replicate", "Target")] results$Flag <- c(FALSE, TRUE, FALSE, NA, FALSE, NA, FALSE, NA) performance <- mfrm_screening_performance(roster, results, rule = "Infit or Outfit outside [0.5, 1.5]; both raters; no exclusion/refit") summary(performance)$by_family ``` The family event is **at least one unaffected rater flagged** in a trial. Trials 1 and 3 are complete negatives. Trial 2 is a known positive because R1 was flagged, even though R2 is unavailable. Trial 4 is unresolved. Thus the conditional rate is 1/3, with 3 available outcomes out of 4 planned trials. The realized proportion over all four trials lies between 1/4 and 2/4. Those latter bounds describe unresolved outcomes; they are not confidence intervals. Omitting a result row also leaves it unavailable. Removing it from the roster would incorrectly change the intended experiment. ```{r incomplete-plot, fig.width=8, fig.height=4.8, fig.alt="One known positive family event among three available trials out of four planned. The conditional rate is one third; unresolved-outcome bounds span one quarter to one half and are not confidence intervals."} plot(performance) ``` Blue whiskers are exact binomial Monte Carlo intervals, conditional on known outcomes. The gray segment describes unresolved outcomes. The right column shows available/planned events. A completely unavailable condition has no point estimate. Use `plot(performance, scope = "target")` for individual rater rates, or `metric = "detection"` for affected targets when the roster includes them. `plot_data(plot(performance, draw = FALSE))$table` supplies the unrounded values. ## Connect simulated ratings to a declared screen This runnable example fits six small datasets generated without rater misfit. It demonstrates the workflow, **not** an adequately precise accuracy study. The band below is an explicit illustrative choice, not a universal diagnostic threshold. The model allows rater severity differences under this null. ```{r generate-and-screen} spec <- build_mfrm_sim_spec(n_person = 60, n_rater = 4, n_criterion = 3, raters_per_person = 2, assignment = "rotating") planned <- expand.grid(Condition = "Rotating null", Replicate = 1:6, Target = sprintf("R%02d", 1:4), stringsAsFactors = FALSE) planned$Affected <- FALSE observed <- planned[c("Condition", "Replicate", "Target")] observed$Flag <- NA observed$FitReady <- FALSE observed$Message <- "" observed$Infit <- observed$Outfit <- NA_real_ observed$InfitZSTD <- observed$OutfitZSTD <- NA_real_ for (i in 1:6) { rows <- which(observed$Replicate == i) messages <- character() tryCatch(withCallingHandlers({ ratings <- simulate_mfrm_data(sim_spec = spec, seed = 700 + i) fit <- fit_mfrm(ratings, "Person", c("Rater", "Criterion"), "Score", model = "RSM", quad_points = 31, maxit = 200) observed$FitReady[rows] <- isTRUE(fit$summary$InferenceReady[1]) diagnostic <- diagnose_mfrm(fit, residual_pca = "none") raters <- diagnostic$measures[diagnostic$measures$Facet == "Rater", ] raters <- raters[match(observed$Target[rows], raters$Level), ] observed[rows, c("Infit", "Outfit", "InfitZSTD", "OutfitZSTD")] <- raters[c("Infit", "Outfit", "InfitZSTD", "OutfitZSTD")] outside <- function(x) ifelse(is.finite(x), x < .5 | x > 1.5, NA) observed$Flag[rows] <- outside(raters$Infit) | outside(raters$Outfit) }, warning = function(w) { messages <<- c(messages, conditionMessage(w)) invokeRestart("muffleWarning") }), error = function(e) { messages <<- c(messages, conditionMessage(e)) }) observed$Message[rows] <- paste(unique(messages), collapse = " | ") } screen <- mfrm_screening_performance(planned, observed, rule = paste("Infit OR Outfit outside [0.5, 1.5]; all four observed raters;", "computable descriptive screens retained; no exclusion/refit")) screen$by_target screen$by_family[screen$by_family$Targets > 0, ] unique(observed[c("Replicate", "FitReady", "Message")]) ``` R's logical `|` retains an available positive when the other index is missing; one negative plus one missing index remains unknown. We record numerical readiness separately from computability. This example retains computable descriptive screens; an inferential procedure may require a stricter rule. Specify that rule in advance and preserve unavailable outcomes when it fails. Six datasets are six independent replications, even though each contains four correlated raters. The any-rater rate therefore has a denominator of six, not 24. With zero flags in six replications, the exact 95% upper Monte Carlo bound is about 46%. A reported Monte Carlo standard error of zero at a boundary is not evidence of certainty. More replications narrow simulation uncertainty; they cannot fix an inappropriate generator or screening rule. To evaluate sensitivity, add a prespecified alternative that generates the departure of interest and mark its affected targets. Inconsistency, Rater-by-Group differential functioning and Rater-by-Criterion interaction are different alternatives. The existing `evaluate_mfrm_signal_detection()` specializes in Group-by-Criterion DIF and Rater-by-Criterion interaction bias; it does not implement a general Rater-by-Group accuracy study. ## Compare thresholds without refitting the models Retain the actual mean squares, not just a yes/no warning. The following profiles are declared comparisons, not an ordered scale of model quality. The final asymmetric band is a published heuristic for judged performance; its label does not make it a validated rater cutoff for every assessment. ```{r threshold-sensitivity} bands <- data.frame( Profile = c("0.5-1.5", "0.7-1.3", "0.8-1.2", "0.4-1.2"), Lower = c(.5, .7, .8, .4), Upper = c(1.5, 1.3, 1.2, 1.2) ) sensitivity <- mfrm_screening_sensitivity(planned, observed, bands, rule = "Ordinary RSM MML; EAP plug-in residuals; all raters; no exclusion/refit") subset(sensitivity$by_family, Targets > 0) ``` ```{r directional-tiles, fig.width=9, fig.height=3.6, fig.alt="Separate underfit and overfit family false-flag rates for declared threshold profiles. Each cell shows the conditional percentage and known/planned replication count; these six illustrative trials do not qualify screening accuracy."} plot(sensitivity, direction = c("underfit", "overfit")) ``` ```{r directional-curves, fig.width=7, fig.height=4.5, fig.alt="Directional false-flag rates across discrete threshold profiles, with pointwise Monte Carlo intervals. Different symbols and line types distinguish underfit and overfit."} plot(sensitivity, style = "curves", direction = c("underfit", "overfit")) ``` The intervals describe Monte Carlo uncertainty across independent simulated datasets; six trials leave considerable uncertainty. Thresholds applied to the same ratings are paired, so comparing their interval overlap is not a paired test. `statistic = "infit"` and `"outfit"` allow separate index comparisons. Use `quantity = "unavailable"` to inspect unresolved family events, and `metric = "detection"` when the roster includes affected targets. All-trial bounds and complete counts remain in the source tables. `palette = "mono"` avoids reliance on colour. `show_title = FALSE` and `show_notes = FALSE` suppress annotations; `title`, `caption`, `text_scale` and `point_size` customize presentation. Tile percentages supplement shading, and curves use distinct symbols and line types. The returned plot data include a text alternative. With ggplot2 installed, use `as_ggplot(plot(sensitivity, draw = FALSE))` for further styling. If labels are hidden for a crowded figure, provide the retained table alongside it. ## Distinguish low mean squares from ZSTD-only warnings `fit_measures_table()` now uses mean squares for its default directional labels. It retains all ZSTD columns, flags additional ZSTD-only evidence in `ZSTDOnly`, and makes the earlier combined rule available explicitly: ```{r separate-zstd, eval=FALSE} review <- fit_measures_table(fit, flag_basis = "mnsq") combined_review <- fit_measures_table(fit, flag_basis = "mnsq_or_zstd") combined_study <- mfrm_screening_sensitivity(planned, observed, bands, rule = "Same saved residuals; engine ZSTD; no exclusion/refit", zstd_cut = 2) ``` Changing the mean-square band under the combined rule need not change a ZSTD-only warning. ZSTD depends on sample size and on the df convention. Its approximate transformation is not a calibrated 5% test after estimating parameters or screening many raters. Compare the magnitude of the mean square, the number of ratings, the residual basis and the substantive consequence. A low mean square means lower residual variability than predicted. This use of **overfit** does not mean machine-learning overfitting, poor rater quality or misconduct. It may reflect redundancy, dependence or fitting effects; it does not identify their cause. Review large mean squares and the data first, and do not automatically remove a consistent rater to make an index closer to one. The distinction between discrepancy magnitude and standardized significance follows [Linacre (2003)](https://www.rasch.org/rmt/rmt171n.htm). The bands summarized by [Wright and Linacre (1994)](https://www.rasch.org/rmt/rmt83b.htm) are contextual heuristics originally discussed for item fit, not universal error-controlled rater thresholds. ## Check the mathematical target before interpreting a threshold For unit-weight ratings, let $p_{ik}$ be the declared model probability of category score $k$, $\mu_i=\sum_k k p_{ik}$, $V_i=\sum_k(k-\mu_i)^2 p_{ik}$ and $e_i=y_i-\mu_i$. Then $$\mathrm{Outfit}=\frac{1}{n}\sum_i\frac{e_i^2}{V_i},\qquad \mathrm{Infit}=\frac{\sum_i e_i^2}{\sum_i V_i}.$$ For fixed correct probabilities with positive variances, each expected squared residual is $V_i$, giving a reference expectation of one. That calculation does not establish the sampling distribution after estimating the model and Person scores from the same responses. Ordinary MML diagnostics here use plug-in EAP Person estimates. Parameter fitting and shrinkage can shift the residual summaries; response dependence also changes uncertainty. Engine and fourth-moment df conventions do not repair an incorrect residual target. For an extended model integrating latent effects, the predictive variance must retain both terms in the law of total variance: $V_i=E[\mathrm{Var}(Y_i\mid b)]+\mathrm{Var}(E[Y_i\mid b])$, with expectations taken over the specified latent distribution. Integrating only conditional variance, or substituting rater modes, generally gives a different statistic. Moreover, using the same responses to obtain the latent posterior and to check fit does not give the ordinary mean-square reference distribution. The threshold workflow above evaluates ordinary-model screens; it does not authorize those bands for the extended-model posterior-predictive diagnostics. ## Design an accuracy study around the feedback decision 1. State the departure, intended warning and consequences of false warnings. If a procedure selects or excludes raters and refits, evaluate that entire procedure, including its effects on scores and intervals. 2. Specify nulls and alternatives, rater counts, rubric categories, ability distributions and rating assignments. Compare assignments at matched rating budgets when asking about connectivity. Keep planned nonassignment separate from missing scores on assigned ratings. 3. Fix thresholds and the comparison family before examining outcomes. One rater's false-flag rate does not control the probability of any false flag among many raters. A union of several diagnostics changes the procedure. 4. Retain all planned trials, errors and unavailable results. Choose the number of replications for the precision needed, and report Monte Carlo uncertainty. If conditions share simulated data, compare them with paired outcomes rather than pretending that their estimated rates are independent. The tables retain full precision. Save the result and its original inputs with `saveRDS(screen, "screening-study.rds")`; use `write.csv(screen$by_family, "screening-family.csv", row.names = FALSE)` for a table. These commands write to your working directory. The stored rule records your description; it cannot verify when that rule was chosen. ## Evidence and limits A bounded comparison used 120 persons, six fixed raters, three criteria and four score categories. Each person was assigned two raters: 720 planned ratings and 120 ratings per rater. A rotating cycle linked all raters through repeated overlap. A weak-bridge design kept the same workloads but linked two panels through only two persons. Both designs used the same 100 independent latent and response draws per scenario. The screen was the union of Infit and Outfit outside [0.5, 1.5], with no rater exclusion or refitting. | Generating scenario | Any unaffected rater flagged: rotating | Any unaffected rater flagged: weak bridge | Detection of affected R6: rotating / weak bridge | | --- | ---: | ---: | ---: | | Normal population, no inconsistency | 0/100 | 0/100 | Not applicable | | Each R6 response replaced by a uniform category with probability 0.5 | 0/100 | 0/100 | 6/100 / 2/100 | | Assignment by ability rank, no inconsistency | 0/100 | 0/100 | Not applicable | | 20% of assigned scores missing completely at random | 0/100 | 1/100 | Not applicable | | 20% missing, favoring generated scores of zero | 0/100 | 0/100 | Not applicable | For the contaminated R6, exact 95% Monte Carlo intervals are 2.23–12.60% in the rotating design and 0.24–7.04% with weak bridges. The paired difference is minus 4 percentage points (MCSE 2.81 points); this small comparison does not establish a general ordering of designs. Each 0/100 false-flag result has a 3.62% upper bound; 1/100 has an interval of 0.03–5.45%. These are pointwise intervals, not simultaneous protection across conditions. All 6,000 target screens were computable. Three of the 1,000 fits required category-support review, all under score-dependent missingness: two rotating and one weak-bridge fit. Their descriptive screens were retained according to the declared rule; their availability is not evidence of inference readiness. The missingness scenarios retain 576 observed ratings each. Score-dependent missingness selected exactly 144 assigned scores without replacement, with fourfold selection weight for generated zeros. Fits used observed scores only. Ability-linked assignment retained the same overall normal ability draws but changed their distribution across panels. These are nulls for rater inconsistency, not assertions that all analysis assumptions remain correct. **The low detection rates are a limitation of this screen under the specified alternative.** Rare false warnings do not make a screen useful when it misses most contaminated raters. An absence of flags cannot certify rater quality. The study does not qualify other thresholds, contamination strengths, rater counts, PCM/GPCM models, disconnected designs, differential rater functioning, post-selection inference or missing-score imputation. Thresholds were not retuned to these outcomes. The study reused 200 unchanged pre-selection screens and fitted the other 800 datasets; the reused results are not extra independent replications. ## Stress-test thresholds, sample size and low-side warnings A further bounded study compared five declared bands, Infit and Outfit separately and jointly, and mean-square-only versus mean-square-or-ZSTD rules. Each condition used 100 independent replications, categories 0--3, three criteria and an ordinary fixed-rater RSM fitted by MML with 61 quadrature points. Abilities were generated from the assumed standard normal population. Rater severities were equally spaced from -0.8 to 0.8, criterion effects were (-0.3, 0, 0.3), and the shared adjacent steps were (-1, 0, 1). The study retained computable descriptive screens without excluding fits that required inference review. It did not remove raters or refit after screening. | Condition | Persons / raters | Generating feature | | --- | --- | --- | | Small crossed null | 60 / 6 | Every person rated by every rater | | Short sparse null | 60 / 6 | Two raters per person, rotating overlap | | Large sparse null | 600 / 6 | Same six ratings per person; more persons per rater | | Twelve-rater null | 120 / 12 | Two raters per person, rotating overlap | | Heavy MCAR null | 120 / 6 | Exactly 60% of the 720 assigned scores missing at random | | Random R6 | 120 / 6 | Each R6 response independently replaced by a uniform category with probability 0.5 | | Predictable R6 | 120 / 6 | Each R6 response replaced by its model-modal category with probability 0.75 | | Local dependence | 120 / 6 | A normal Person-by-rater effect with SD 1 shared across three criteria | | Disconnected null | 120 / 6 | Two rater panels with no common persons; common criteria and ability-population assumption retained | Except for the crossed condition, each person was assigned two raters and three criteria. Nonassigned cells were absent, not imputed. The nulls permit severity differences. The disconnected condition is null for rater response inconsistency; it does not qualify cross-panel interpretation when the assumed ability distributions differ. Local dependence affects all raters and cannot be interpreted as identifying one bad rater. The following are **any-unaffected-rater overfit rates**, not individual-rater probabilities. Mean squares use [0.5, 1.5]; the combined rule also uses engine ZSTD at or below -2 for the low-side flag. | Null condition | Mean squares only | Mean squares or ZSTD | | --- | ---: | ---: | | Small crossed | 0/100 | 44/100 | | Short sparse | 1/100 | 59/100 | | Large sparse | 0/100 | 100/100 | | Twelve raters | 1/100 | 88/100 | | Heavy MCAR | 24/100 | 92/100 | | Disconnected panels | 0/100 | 93/100 | Thus ZSTD-only additions can produce frequent low-side warnings even under the generating null. The fourth-moment df convention did not resolve this: its corresponding rates were 48%, 71%, 100%, 96%, 96% and 96%. Mean-square-only screening also remained unreliable in the heavy-missingness condition. For 24/100, the exact pointwise 95% Monte Carlo interval is 16.02--33.57%; for 0/100 it is 0--3.62%, and for 100/100 it is 96.38--100%. None is a universal error guarantee. ```{r stress-overfit, echo=FALSE, out.width="100%", fig.alt="Overfit family false-flag rates in six null conditions across five threshold bands. Mean-square-only rates rise sharply as the lower bound increases. Adding engine ZSTD yields 44 to 100 percent false flags under the 0.5 to 1.5 band. Cell counts are known outcomes out of 100 planned trials; inference-review fits are retained."} knitr::include_graphics("figures/screening-overfit-stress.png") ``` Narrowing a band improved detection of the random-category departure but also increased low-side false flags. The R6 detection rates were 2%, 11%, 39% and 65% under [0.5,1.5], [0.6,1.4], [0.7,1.3] and [0.8,1.2]. Under [0.4,1.2], random-category detection was also 65%, but detection of the specified predictable-response departure fell to 18%, compared with 65% under [0.5,1.5]. No single band was selected as best from these outcomes. The chosen departure, the cost of false feedback and the observation design all matter. ```{r stress-detection, echo=FALSE, out.width="100%", fig.alt="R6 detection curves show the tradeoff between thresholds and departures, with pointwise Monte Carlo intervals. Uniform-category contamination produces upper-side flags; modal-category replacement produces lower-side flags. A 0.4 to 1.2 band detects 65 percent of the former and 18 percent of the latter."} knitr::include_graphics("figures/screening-detection-stress.png") ``` All 900 fits were numerically ready and all 6,000 target screens were computable, while 159 fits required inference review. Computability is not inferential eligibility. Independent reconstruction from observation-level residuals agreed with the reported mean squares to within $2.6\times10^{-15}$. Nevertheless, in the large sparse null, the average fitted Infit/Outfit was 0.832/0.806, versus 1.001/1.001 using the generating parameters. In the heavy MCAR null, the corresponding values were 0.704/0.689 versus 1.002/1.004. The formulas can therefore be implemented correctly while a reference rule centered at one is unsuitable for the fitted residuals. This comparison does not isolate every contribution of estimation, shrinkage and short Person records; it establishes the need to validate the actual diagnostic procedure. The previous matched-budget study also supplied 800 saved raw-statistic fits for threshold sensitivity without refitting. Two other cells retained only yes/no flags, so their 200 trials remain unavailable for new thresholds; the old flags cannot reconstruct unknown mean squares. These reanalyses are not new independent replications. The stress results do not qualify JML, other models, arbitrary sample sizes, extended-model posterior diagnostics, or automatic rater exclusion. The Monte Carlo reporting principles follow Morris, White and Crowther (2019), [Using simulation studies to evaluate statistical methods](https://doi.org/10.1002/sim.8086). Rater misfit and differential functioning should be distinguished when designing an evaluation; see Wind and Guo (2019), [Exploring the Combined Effects of Rater Misfit and Differential Rater Functioning in Performance Assessments](https://doi.org/10.1177/0013164419834613). Incomplete assignments are also part of the design question; see Wind and Jones (2019), [The Effects of Incomplete Rating Designs in Combination With Rater Effects](https://doi.org/10.1111/jedm.12201). A warning supports review of rubric use, shared rating examples and design coverage. It does not establish that a particular rater should be removed, that training will help, or that an imputation method repairs selective nonresponse. Conditions outside an evaluated generator remain unqualified.