--- title: "Rating agreement and score comparisons" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Rating agreement and score comparisons} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>", dev.args = list(bg = "white")) ``` Two raters can rank responses identically while assigning different scores. This tutorial shows how to detect that distinction, match the right rating records, and retain the sample behind each coefficient. It uses small synthetic data and fits no measurement model. Run the code blocks in order with `mfrmr` installed. For importing your own CSV or fitting a model, start with the [ordinary workflow](mfrmr-workflow.html). ## Choose the question before the coefficient | Your question | Starting point | What the result describes | | --- | --- | --- | | Do two raters give similar category scores to the same responses? | `pairwise_kappa_table()` | Weighted agreement for each pair on its common complete units. | | How much agreement is present across the panel? | `krippendorff_alpha_table()` or `gwet_ac2_table()` | A pooled coefficient under a stated distance/weighting and chance model. Their definitions differ. | | Does a score source agree with a chosen reference, track its variation, or shift its mean? | `score_comparison_table()` | QWK, correlation and standardized mean difference on aligned records. A reference is not automatically correct. | | Which rater is stricter after accounting for the fitted facets? | `fit_mfrm()`, then individual estimates and diagnostics | Model-adjusted severity. High rater separation reliability means distinguishable severities, not agreement. | | Does a rater tend to compress scores toward the middle? | `rater_centrality_table()` | Indirect RSM residual and/or rater-specific PCM threshold descriptions, with their source status. | | How accurately do raters match supplied expert benchmark ratings? | `fit_rater_accuracy()` | A separate Rasch model of exact-match indicators; the benchmark and its alignment must be justified. | This lesson works through the first three questions. A high correlation does not establish agreement, agreement does not establish benchmark accuracy, and a low Infit value alone does not diagnose centrality. For the distinction between observed agreement and model-based reporting, see [Reporting and APA](mfrmr-reporting-and-apa.html). The [methods guide](../reference/mfrmr_methods_reference.html#choose-an-agreement-target-and-keep-its-denominator) also distinguishes the denominators and saving routes. In particular, `interrater_agreement_table()` reports model-implied expected exact agreement from fitted category probabilities; this is not the chance-agreement term used by kappa or AC2. Residual association answers another question: whether departures from the fitted model vary together. Neither raw agreement nor residual association alone establishes the other. ## 1. Identify the rated unit Six people each complete two tasks, scored on the permitted categories `0, 1, 2, 3`. Rater R2 deliberately gives every response one category more than R1. R3 has a few different scores, one missing score and one absent row. These existing teaching data are included in mfrmr: `data()` loads the table, and the next line gives it the shorter name `ratings`. You do not need to construct scores or run a simulation before following this lesson. ```{r ratings} library(mfrmr) data("mfrmr_example_agreement", package = "mfrmr") ratings <- mfrmr_example_agreement head(ratings) with(ratings, table(Rater, Score, useNA = "ifany")) ``` Here **Person and Task together** identify one response to be compared. `Person` alone would mix different tasks. If your data also distinguish rubric criteria or occasions, include those identifiers in `unit`, for example `c("Person", "Task", "Criterion", "Occasion")`. There must be at most one row for each rater and unit: ```{r check-keys} key <- c("Person", "Task", "Rater") stopifnot(anyDuplicated(ratings[key]) == 0L) ``` An assertion failure means the key repeats. Check whether a task or occasion column is missing from the key, or whether a row was accidentally duplicated. Do not silently average or delete repeated ratings. The `unit` argument matches records; it does not fit separate Person and Task variance components. There are 35 input rows and 34 observed scores. The R3/P06/T2 row has an explicit missing score; R3/P02/T2 has no row. Neither is a score of zero. An absent row alone does not tell us whether a rating was unassigned or an assigned rating was never recorded; that requires the assignment roster. ## 2. Compare pairs on their shared responses Declare the complete rubric even if some categories are unused by a rater. Quadratic weighting gives quadratic weighted kappa (QWK): a larger category disagreement receives a larger penalty. ```{r pairwise-agreement} qwk <- pairwise_kappa_table( ratings, rater = "Rater", unit = c("Person", "Task"), score = "Score", categories = 0:3, weights = "quadratic" ) qwk$table[, c("Rater1", "Rater2", "N", "Kappa", "Status")] ``` `N` is the number of common complete units, not the number of people or input rows. R1/R2 uses all 12 responses. Each pair involving R3 uses 10. Each pair uses its own retained sample for both observed and expected agreement; the three coefficients do not automatically form a panel average or a ranking of rater quality. For numeric categories, linear and quadratic weights use the declared numeric spacing. With character categories they use positions in the supplied order. Choose this meaning from the rubric. Changing `c(0, 1, 2, 3)` to uneven numeric codes changes the weights; a larger resulting coefficient would not establish better agreement. Inspect exclusions before interpreting a coefficient. `Unit` is an index into `qwk$units`; attach the original identifiers to make the audit readable: ```{r pair-audit} pair_audit <- cbind(qwk$cases, qwk$units[qwk$cases$Unit, , drop = FALSE]) subset(pair_audit, !Included, select = c(Person, Task, Rater1, Rater2, Assigned1, Assigned2, Missing1, Missing2, Included)) ``` `Assigned2 = FALSE` means that no input row exists for the second rater on that unit. `Assigned2 = TRUE` with `Missing2 = TRUE` means a row exists but its score is missing. These are observations about the supplied file, not proof that the planned assignment was complete. ## 3. Separate agreement, correlation and mean differences Suppose R2 is the source under review and R1 is the comparison reference. Build the two score vectors by joining on the unit identifiers, not by assuming that two files have the same row order. `all = TRUE` retains unmatched units for the subsequent missing-pair audit instead of discarding them silently. ```{r align-sources} scored <- subset(ratings, Rater == "R2", select = c(Person, Task, Score)) reference <- subset(ratings, Rater == "R1", select = c(Person, Task, Score)) names(reference)[3] <- "Reference" stopifnot(anyDuplicated(scored[c("Person", "Task")]) == 0L, anyDuplicated(reference[c("Person", "Task")]) == 0L) aligned <- merge(scored, reference, by = c("Person", "Task"), all = TRUE) comparison <- score_comparison_table( scores = aligned$Score, reference = aligned$Reference, categories = 0:3 ) comparison$table[, c("N", "MeanScore", "MeanReference", "QWK", "Correlation", "SMD")] mean(aligned$Score == aligned$Reference) # Exact agreement for this complete pair ``` For these 12 responses, R2's mean is 2 and R1's mean is 1. Correlation is 1, yet exact agreement is 0 and QWK is about 0.571. R2 tracks every score difference while consistently assigning one category more. Thus perfect correlation can coexist with systematic disagreement. `SMD` is the mean difference divided by the selected score SD. Its default pooled-SD version is about +1.173 here; positive values mean the evaluated source scores higher. This example deliberately treats the 0-to-3 rubric codes as equally spaced for means and SMD. Check that score-scale interpretation for your application. SMD does not adjust for differing task/person assignment and is not an MFRM severity estimate. Both pooled-SD and reference-SD versions are retained in the table. The function does not align records, round continuous scores, or choose pass/fail criteria for you. For continuous source scores, pass the original values as `scores` and an explicitly categorized vector as `score_categories`; correlation and SMD use the original values, while QWK uses the category values. Document the categorization rule before evaluating the results. ```{r comparison-status} comparison$criteria[, c("Metric", "Value", "Threshold", "Status", "Reason")] comparison_cases <- cbind(aligned[c("Person", "Task")], comparison$cases) comparison_cases ``` `not_requested` means a statistic is available but no threshold was requested. `unavailable` means the statistic could not be computed; it does not mean zero agreement. Without a second reference, the two degradation statistics are unavailable. By default `screen = "none"`. The explicitly requested `"williamson2012"` screen uses numerical examples for an automated-scoring application; passing those thresholds is not a general validity certificate. See `?score_comparison_table` for the definitions and application limits. ### Add a second reference only when the design supplies one If R3 is a defensible second reference, its agreement with R1 can supply a baseline. Here it illustrates the calculation, not expert authority. ```{r second-reference} reference2 <- subset(ratings, Rater == "R3", select = c(Person, Task, Score)) names(reference2)[3] <- "Reference2" aligned3 <- merge(aligned, reference2, by = c("Person", "Task"), all = TRUE) with_baseline <- score_comparison_table( scores = aligned3$Score, reference = aligned3$Reference, reference2 = aligned3$Reference2, categories = 0:3 ) with_baseline$table[, c("NInput", "N", "NExcluded", "QWK", "ReferenceQWK", "QWKDegradation")] cbind(aligned3[c("Person", "Task")], with_baseline$cases) ``` All statistics now use the same 10 complete three-source responses. Adding R3 changes the sample even for the R2/R1 statistics, so do not interpret their change from the 12-response analysis as a change in scorer performance. `QWKDegradation` is `ReferenceQWK - QWK`: positive values mean less agreement with R1 than the R3/R1 baseline, and negative values mean more. Baseline agreement is itself sample-dependent; it does not establish either reference's correctness. ## 4. Describe panel agreement under a declared definition For a panel coefficient, compute the intended coefficient directly rather than averaging the pairwise kappas: ```{r panel-agreement} alpha <- krippendorff_alpha_table( ratings, rater = "Rater", unit = c("Person", "Task"), score = "Score", level = "ordinal", categories = 0:3 ) ac2 <- gwet_ac2_table( ratings, rater = "Rater", unit = c("Person", "Task"), score = "Score", categories = 0:3, weights = "quadratic" ) alpha$table[, c("Alpha", "Level", "Status", "PairableUnits", "PairableRatings")] ac2$table[, c("AC", "Coefficient", "Status", "PairableUnits", "NonmissingRatings")] alpha$unit_summary ``` These coefficients are not interchangeable implementations of the same formula. Ordinal alpha uses distances based on the frequencies of pairable categories; it is not QWK with a different name. AC2 here uses quadratic category weights and a different chance-agreement model. The help pages document the detailed definitions; the following differences matter when reading incomplete panels: | Feature | Krippendorff's alpha | Gwet AC2 in this call | | --- | --- | --- | | Units with two or more observed ratings | Supply observed and expected disagreement. Unequal rating counts affect unit contributions. | Supply observed agreement, with equal weight per unit. | | A unit with exactly one observed rating | Does not enter the coefficient. | Contributes to category proportions for expected agreement, but not observed agreement. | | Unused categories | Do not alter ordinal pairable distances. | Remain part of the declared category universe and can change expected agreement. | The present example has at least two observed ratings on every unit. Use `$unit_summary` and `$cases` to establish that for your data. These functions do not correct selective assignment or missingness. Choosing whichever coefficient is largest after looking at the results changes the reporting question. Krippendorff's alpha is also distinct from Cronbach's internal- consistency alpha. If your question instead concerns scores on a meaningful interval scale, `pairwise_icc_table()` distinguishes absolute agreement from consistency and a single rating from the mean of two ratings. Numeric ordinal codes alone do not establish its interval-scale assumption. See `?pairwise_icc_table`; the function's pairwise average-rating ICC concerns two ratings, not the average of a whole panel. ## 5. See the disagreement, then compare the coefficients The following figures answer two different questions: where do matched scores fall, and how much weighted agreement does each pair show? They use the saved `qwk` result directly. They do not refit a model or estimate uncertainty. The optional `ggplot2` package is needed only for these figures; all preceding tables remain usable without it. For model-based figures such as Wright maps and category curves, use the [visual diagnostics guide](mfrmr-visual-diagnostics.html). ### Matched-score counts A plain scatterplot would hide repeated score combinations. Use the retained contingency counts instead: each circle's area and printed number show how many matched responses occupy that cell. The dashed diagonal marks identical scores. Category indices in `$counts` are positions in `$categories`, not the rubric's score values; use the category labels on the axes. ```{r score-count-figure, fig.width=7, fig.height=4.5, fig.alt="R1 versus R2 score counts. Four responses occur at each of (0, 1), (1, 2) and (2, 3). All lie one category above the equal-score diagonal; none has identical scores."} can_plot <- requireNamespace("ggplot2", quietly = TRUE) figures <- list() selected_pair <- subset(qwk$table, Rater1 == "R1" & Rater2 == "R2") score_counts <- subset(qwk$counts, Pair == selected_pair$Pair) category_index <- qwk$categories$CategoryIndex category_labels <- qwk$categories$Category if (can_plot) { figures$score_counts <- ggplot2::ggplot( score_counts, ggplot2::aes(x = Category1, y = Category2) ) + ggplot2::geom_abline(slope = 1, intercept = 0, linetype = "dashed", colour = "grey45") + ggplot2::geom_point(ggplot2::aes(size = N), shape = 21, fill = "grey85", colour = "grey30") + ggplot2::geom_text(ggplot2::aes(label = N), size = 3.5) + ggplot2::scale_size_area(max_size = 14, guide = "none") + ggplot2::scale_x_continuous(breaks = category_index, labels = category_labels) + ggplot2::scale_y_continuous(breaks = category_index, labels = category_labels) + ggplot2::coord_equal(xlim = range(category_index) + c(-0.4, 0.4), ylim = range(category_index) + c(-0.4, 0.4)) + ggplot2::labs(x = "R1 score", y = "R2 score", title = NULL, caption = NULL) + ggplot2::theme_classic(base_size = 12) print(figures$score_counts) } ``` All 12 responses sit one category above the diagonal. This makes the systematic shift visible despite the correlation of 1. Blank cells mean no retained responses with that score combination; they do not identify missing ratings. The `$cases` audit remains the source for exclusions. R1 is the comparison reference here, not an established correct answer. ### Pairwise QWK with its sample size Keep pairs in their original order and show each retained sample size. Sorting by QWK would invite a ranking even though the pairs can involve different responses. The zero line is the coefficient's chance-agreement reference, not a quality threshold. There are no confidence intervals or quality bands. ```{r pairwise-qwk-figure, fig.width=7, fig.height=3.5, fig.alt="Pairwise quadratic weighted kappa: R1/R2, 0.571 on 12 responses; R1/R3, 0.833 on 10; R2/R3, 0.455 on 10. Points have no uncertainty bars. The vertical dotted line marks zero, not a pass threshold."} pair_plot_data <- qwk$table pair_plot_data$PairLabel <- with(pair_plot_data, paste0(Rater1, " / ", Rater2, " (N = ", N, ")", ifelse(Status == "available", "", paste0("; ", Status)))) pair_plot_data$PairLabel <- factor(pair_plot_data$PairLabel, levels = rev(pair_plot_data$PairLabel)) if (can_plot) { figures$pairwise_qwk <- ggplot2::ggplot( pair_plot_data, ggplot2::aes(x = Kappa, y = PairLabel) ) + ggplot2::geom_vline(xintercept = 0, linetype = "dotted", colour = "grey45") + ggplot2::geom_point(data = subset(pair_plot_data, Status == "available" & is.finite(Kappa)), size = 3) + ggplot2::scale_y_discrete(drop = FALSE) + ggplot2::expand_limits(x = c(0, 1)) + ggplot2::labs(x = "Quadratic weighted kappa", y = NULL, title = NULL, caption = NULL) + ggplot2::theme_classic(base_size = 12) print(figures$pairwise_qwk) } ``` `expand_limits()` includes 0 and 1 without removing negative estimates. Unavailable pairs retain their labelled row and status, with no point; an unavailable value must not be drawn at zero. This example's three estimates are all available. Read the values with the exclusion audit rather than as evidence that one rater is more accurate or less severe. These are custom ggplot recipes from `pairwise_kappa_table()` results, not new mfrmr plot methods. The existing `plot()` displays exact agreement or correlation from a fitted-model workflow, not QWK. Keep suggested titles and interpretation notes outside the drawing and save them with the source data: ```{r figure-notes} figure_titles <- c(score_counts = "Matched score counts for R1 and R2", pairwise_qwk = "Pairwise quadratic weighted agreement") figure_notes <- list( score_counts = c( "Circle area and printed counts describe common complete responses; the diagonal marks equal scores.", "Blank cells are zero retained counts, not missing-rating indicators. Inspect qwk$cases for exclusions.", "The reference is not assumed correct; observed score differences are not model-adjusted severity." ), pairwise_qwk = c(qwk$notes, "N is pair-specific. The zero line is a chance reference, not an acceptance threshold; no intervals are shown.") ) figure_titles figure_notes ``` ## 6. Save the evidence and state the answer Keep the full results, matched identifiers and resolved settings together. CSV tables are useful for inspection, but a coefficient table alone loses the exclusion audit and interpretation notes. ```{r save-and-reopen} analysis <- list( ratings = ratings, qwk = qwk, alpha = alpha, ac2 = ac2, aligned = aligned, comparison = comparison, aligned3 = aligned3, with_baseline = with_baseline, figures = figures, figure_titles = figure_titles, figure_notes = figure_notes, figure_data = list(score_counts = score_counts, pairwise_qwk = pair_plot_data), session = sessionInfo() ) output_dir <- tempfile("rating-agreement-") dir.create(output_dir) saveRDS(analysis, file.path(output_dir, "analysis.rds")) write.csv(qwk$table, file.path(output_dir, "pairwise-qwk.csv"), row.names = FALSE) write.csv(comparison$table, file.path(output_dir, "score-comparison.csv"), row.names = FALSE) write.csv(comparison_cases, file.path(output_dir, "matched-cases.csv"), row.names = FALSE) if (can_plot) { ggplot2::ggsave(file.path(output_dir, "score-counts.png"), figures$score_counts, width = 7, height = 4.5, units = "in", dpi = 150) ggplot2::ggsave(file.path(output_dir, "pairwise-qwk.png"), figures$pairwise_qwk, width = 7, height = 3.5, units = "in", dpi = 150) } restored <- readRDS(file.path(output_dir, "analysis.rds")) restored$comparison$table[, c("N", "QWK", "Correlation", "SMD")] restored$qwk$notes # With ggplot2 installed: print(restored$figures$pairwise_qwk) ``` `output_dir` contains the practice files. Use a new permanent study directory for work you want to retain, and keep the analysis script. Reopening the RDS does not recompute coefficients. Check identifiers before sharing real data. The saved figures, drawing data, suggested titles and notes remain together in `analysis.rds`; PNG files alone do not preserve that context. Redrawing a saved ggplot needs a compatible ggplot2 installation. The recorded session helps identify the version used. A concise description of the worked pair would be: > On 12 matched person-by-task responses with declared categories 0 through 3, > R2 scored every response one category higher than R1. Correlation was 1.00, > but exact agreement was 0% and quadratic weighted kappa was 0.571. These > results describe the observed pair; they do not estimate model-adjusted > severity or establish reference accuracy. For a real study, also report which records were excluded, the category weights or distance definition, the assignment design and why the chosen reference is suitable. These interfaces return descriptive point estimates, not confidence intervals or universal quality bands. Two tasks from one person are two matched response units here, not evidence of two independent people for subsequent sampling inference. ## When a result needs investigation | What you see | How to investigate | | --- | --- | | A duplicate rater-unit error | Check the composite key and distinguish accidental duplication from separate tasks, criteria or occasions. | | Few common complete units | Inspect `$cases` and the assignment roster. Different pairs can involve different samples. Do not fill absent scores with zero. | | `NA` for kappa or alpha | Read `Status`; there may be too few pairable observations or no expected disagreement. Do not replace an unavailable coefficient with 0 or 1. | | A negative coefficient | Retain it and inspect the rating table and definition. These functions do not truncate negative estimates to zero. | | AC2 differs from alpha or kappa | Check the chance model, weighting, category universe and contributing units before attributing the difference to software. | | Fewer records after adding `reference2` | Every comparison statistic uses complete records across all supplied vectors; inspect the new case audit. | | High rater separation reliability alongside low agreement | Separation describes distinguishable model-adjusted severities; it is not an agreement coefficient. Low separation can reflect imprecision and does not prove agreement either. | Use `?pairwise_kappa_table`, `?krippendorff_alpha_table`, `?gwet_ac2_table` and `?score_comparison_table` for formulas, argument details and references. For a broader question-to-function map, use `mfrmr_output_guide("feedback")`. Future pass/fail accuracy under an explicit rating plan is a different question; see `?classification_table` for its fixed-calibration assumptions and cut scales.