mfrmr Reporting and APA

A useful report explains the question, why the scoring design and model can answer it, what the results show, and how far the answer applies. This guide connects those steps to actual mfrmr tables and figures. The Manuscript coverage map also identifies information that the researcher must provide outside the fitted model.

For a literature-based comparison with FACETS/ConQuest and actual L2 speaking/writing papers, see connect designs, networks and reports. That lesson also distinguishes the analysis units of MFRM and G/D studies.

Choose report sections by question

A report needs a common core: the question, rating unit and assignment, rubric endpoints/direction, retained sample, method and scale, uncertainty, and the substantive answer. Add the sections below when they answer that question. An unrequested analysis is not a reporting defect, and a longer coefficient list is not stronger evidence.

When this is your question	Report these outputs together	Existing route
Can the model distinguish people or facet levels, and where does it misfit?	Estimates/intervals, Separation, Strata, model or fit-adjusted Reliability, fit basis, Wright map and relevant category curves	`diagnose_mfrm()` → `mfrm_results()` → `mfrm_report()`; this lesson below
Do two named raters agree on the same responses?	Exact/adjacent agreement when useful, Cohen’s kappa with declared unweighted/linear/quadratic weights, common-unit counts and exclusions	`pairwise_kappa_table()`; agreement lesson. Several raters produce separate pair results, not a single panel kappa.
How much observed agreement does the panel show?	Krippendorff’s alpha with its distance and pairable units, or a justified alternative such as Gwet AC2	`krippendorff_alpha_table()` / `gwet_ac2_table()`. Choose the chance/distance model before interpreting the coefficient.
How reproducible are numeric ratings under the specified rater model?	The stated ICC definition, single/average-score target, consistency/absolute-agreement target and analyzed sample	`pairwise_icc_table()` and its help. An ICC is not interchangeable with separation reliability, alpha or kappa.
Which future assessment plan suits the intended score?	G-study variance/covariance components; D-study G/Phi/SEM, score weights, count grid, uncertainty and eligible paired contrasts	G/D-study reporting structure, `mfrm_report(g/d, intervals = saved_ci)` → `build_summary_table_bundle()`; `plot(g/d)` and the editable G/D template
Can scores be compared across forms or waves?	Anchor source, values, scale/direction, centering, overlap, invariance assumption, drift and uncertainty basis	Linking workflow, `review_mfrm_anchors()` → anchored fit / `build_linking_review()` → `build_summary_table_bundle()`
Does a particular combination or group need investigation?	Signed bias/DFF contrast, cell evidence, reference group, multiplicity, uncertainty and follow-up	Specified `estimate_bias()` / `analyze_dff()` → `plot()` / tables. Supply matching bias results to `reporting_checklist()` and `build_apa_outputs()`; `dif_report()` serves DFF.
Does a model choice change the answer?	Comparison basis, AIC/BIC where comparable, eligible LRT, numerical checks and changes in the target estimates/conclusion	`compare_mfrm()` → `build_model_choice_review()` → `build_summary_table_bundle()`; model choice
What unexplained structure or grouping merits follow-up?	Node/feature definition, residual versus observed-score basis, overlap, missingness, selection choices and exploratory stability	Visual diagnostics and feature groups; keep exploration separate from confirmation

Separation describes spread relative to error; Strata transforms separation; Reliability expresses another ratio of the same underlying variance estimates. They are useful companion summaries, not three independent confirmations. Kappa and alpha instead concern observed agreement. Report both only when the pairwise and panel questions both matter. Select comparable criteria/occasions and name the pooling rule; a single pooled coefficient can hide differences among criteria. Never fill missing overlap with zero agreement.

The ordinary report already has fit, precision, bias and linking wording tables. include = "bias" supplies a facet screen, not a newly selected Rater-by-Criterion interaction; include = "linking" does not create evidence from other waves. Supply the targeted analyses through their supported routes. Observed alpha/kappa and external-feature groups remain separate result objects. G/D studies and residual networks now enter mfrm_report() directly and retain their own estimands and reporting tables. The editable GPCM example below joins one particular set of these outputs; it is not an automatic report for all package functions.

For assessment planning, copy and render system.file("templates", "gtheory-report.Rmd", package = "mfrmr"). See the G/D-study route for source-checked saved intervals and full CSV/HTML exports.

For selected anchor, bias and model-choice questions, copy system.file("templates", "assessment-review.Rmd", package = "mfrmr"). Its switches run only the requested modules. The template starts from bundled ratings, retains all bias cells and anchor values, and exports each report with a complete saved analysis. These objects now enter mfrm_report() directly; reporting does not recalculate estimates or test missing evidence.

Keep the study’s score meaning with each report:

context <- c(rating_unit = "One performance by judge by criterion rating",
  score_direction = "Higher Comprehensibility means harder to understand",
  rubric = "1 = very easy; 9 = very difficult", transformation = "None")
report <- mfrm_report(results, study_context = context)
report$tables$study_context

The declaration is copied into the object, HTML/Markdown and appendix tables; it does not recode the data or reverse a parameter’s sign. Reuse it across measurement, planning and feedback only when the units and meanings match. Record differences in aggregation, scale and anchor origin explicitly. The model cannot validate the meaning of these author-supplied declarations.

Start with an editable GPCM report

For a one-family GPCM MML analysis, the package includes a complete R Markdown starting point. It fits the bundled synthetic ratings and produces an HTML report, CSV tables, two PNG figures and a saved analysis:

template <- system.file("templates", "gpcm-report.Rmd", package = "mfrmr")
file.copy(template, "gpcm-report.Rmd")
rmarkdown::render("gpcm-report.Rmd")

# Then use your own data: Person, Rater, Criterion, Score; one rating per cell.
rmarkdown::render("gpcm-report.Rmd", params = list(
  ratings_csv = "ratings.csv", rating_min = 1, rating_max = 4,
  output_dir = "my-gpcm-report-files"
))

Edit the model and study-context sections for your assessment. The template uses criterion-specific steps and slopes; it is not a two-family GPCM or JML template. Repeated occasions require an explicit measurement unit and matching agreement keys. The default output folder is gpcm-report-files.

What you want to report	Where it comes from	Key distinction
Separation, Strata, Reliability	`diagnose_mfrm(fit)$reliability`	Separation-based precision, with its recorded SE basis and eligibility; not observed agreement.
Infit/Outfit and ZSTD	`diagnose_mfrm(fit)$fit`	Native MML EAP plug-in fit; not TAM/ConQuest posterior-averaged fit.
Wright map	`plot(fit, diagnostics = diagnostics)`	Locations and targeting; the template deliberately draws points without implying qualified intervals.
Category-probability curves	`plot(fit, type = "ccc")`; `category_curves_report(fit)`	Conditional reference profiles; retain their step/slope owners and reference settings.
Krippendorff’s alpha	`krippendorff_alpha_table()` on the ratings	A separate observed-agreement calculation. `mfrm_report()` does not automatically add it.
Draft Methods/Results and reporting gaps	`mfrm_report(..., style = "apa")`, `build_apa_outputs()`, `reporting_checklist()`	Draft text and evidence tracking; recruitment, design justification and the substantive answer still come from the researcher.

These outputs can support a paper, but their presence does not qualify every claim. In particular, a converged one-family GPCM can retain incomplete identification/boundary evidence and review-only precision summaries. Read the fit decision and any target-specific interval result; do not interpret missing intervals or formal-inference eligibility as established by a report preset. The template retains these states and separates model precision from alpha.

It asks a panel-level agreement question, which explains its alpha section. Separation is already included. For a specified rater-pair question, add pairwise_kappa_table() with explicit weights and the same matched rating units; the template includes an optional code example. The synthetic rating design has six raters but only six of the fifteen possible pairs share units. The remaining nine pair coefficients are unavailable, not zero. This is a reason to show the assignment/overlap evidence alongside agreement, not to substitute model-adjusted severity for missing pairwise observations.

The worked lesson below develops the reporting logic in more detail using RSM.

The running question is: after accounting for person ability and criterion difficulty, how much do the raters differ in severity? The synthetic data illustrate the reporting workflow; they are not evidence about a real assessment or population. For loading your own CSV, see vignette("mfrmr-workflow", package = "mfrmr").

1. Fit once and retain the analysis record

# Load the package and synthetic ratings
library(mfrmr)
toy <- load_mfrmr_data("example_operational")
head(toy)

# Estimate person ability, rater severity, and criterion difficulty
fit <- fit_mfrm(
  data = toy,
  person = "Person",
  facets = c("Rater", "Criterion"),
  score = "Score",
  method = "MML",
  model = "RSM"
)

# Reuse these diagnostics in the tables and report below
diag <- diagnose_mfrm(fit, diagnostic_mode = "both", residual_pca = "none")
results <- summary(fit, diagnostics = diag)

One row is one rating. The rubric uses 1 to 4, and all four categories occur. The example has 48 persons, six raters, and three criteria, with 282 observed ratings. Here N is the number of rating rows, not the number of independent participants. Repeated ratings from the same person share that person’s latent ability in the model.

MML means marginal maximum likelihood; RSM is a rating-scale model with shared category thresholds. In this default fit, the person distribution is fixed to standard normal, person scores are expected a posteriori (EAP) estimates, and rater and criterion effects are fixed parameters centered within each facet. Higher rater estimates mean stricter ratings, in logits. In a real study, explain why this population assumption and shared-threshold rubric match the question. Record any different estimator, constraints, anchors, weights, interactions, or score orientation.

results$overview[, c("Model", "Method", "N", "Persons", "Converged")]
results$settings_overview[, c(
  "QuadPoints", "RatingMin", "RatingMax", "RatingRangeSource", "NoncenterFacet"
)]
results$row_retention
results$decision

# Record the iteration limit, integration grid, and numerical stopping details
spec <- specifications_report(fit)
spec$convergence_control
results$overview[, c("OptimizerMethod", "ConvergenceStatus", "RequestedReltol", "EffectiveReltol")]

The default numerical integration uses 31 quadrature points. Retain the analysis script and recorded controls with the results; convergence alone does not establish numerical stability or model adequacy. If the conclusions hinge on a close model comparison, use mml_quadrature_sensitivity() with a prespecified denser common grid and report whether the conclusion changes.

Show what was planned, observed, and excluded

data("mfrmr_example_operational_design", package = "mfrmr")
data_review <- describe_mfrm_data(
  data = toy,
  person = "Person",
  facets = c("Rater", "Criterion"),
  score = "Score",
  rating_min = 1,
  rating_max = 4,
  keep_original = TRUE,
  expected_design = mfrmr_example_operational_design
)
data_summary <- summary(data_review)
data_summary$missing
data_summary$structural_missingness
data_summary$design_connectivity

# Describe the scores and the amount of rating evidence
data_summary$score_distribution
data_summary$facet_overview
data.frame(MeanScore = mean(toy$Score), SDScore = sd(toy$Score))
table(table(toy$Person)) # Number of persons with each observed rating count
as.data.frame(results$person_overview[, c(
  "Persons", "DistributionN", "ReviewExcludedExtremeEAPs", "Mean", "SD", "MeanPosteriorSD"
)])

The supplied roster contains 288 planned rating cells, of which six are absent. All 282 input rows are retained in fitting. These are different counts: absent planned ratings are not rows discarded by fit_mfrm(). Report the assignment scheme, overlap, exclusions, and reasons for missing ratings. Without a roster, do not label every unobserved person-by-rater combination as a missing planned rating. A connected design is relevant to comparability, but does not establish anchor invariance or sufficient precision.

These counts describe different units. In the example, 42 persons have six ratings and six have five; rater workloads range from 38 to 56 rating rows. For repeated testing, state whether a person ID identifies a person, a response, or an occasion, and how dependence across occasions is handled. Preserve the original rubric, any scoring exceptions, and the exact recoding rule. Such details affect what the model can answer: Goodwin (2016, Sections 3–4 and Table 1) documents score-use contexts, rating history, exclusions, and category recoding before reporting rater differences.

State which persons enter distribution summaries. Eckes (2005, Tables 1–2) marks summaries restricted to nonextreme examinees. In this example the person summary includes all 48 persons. SD describes variation across person EAP estimates; MeanPosteriorSD describes their average within-person uncertainty. They are not interchangeable, and neither is a rater or criterion SE.

2. Check uncertainty and fit before writing the answer

prec <- precision_review_report(fit, diagnostics = diag)
prec$profile[, c(
  "PrecisionTier", "SupportsFormalInference", "PersonSEBasis", "NonPersonSEBasis"
)]
prec$checks

# These diagnostics address departures from the fitted response model
diag_summary <- summary(diag)
diag_summary$overall_fit
diag_summary$diagnostic_basis[, c("DiagnosticPath", "Status", "ReportingUse")]

Read results$decision, including Why and NextAction, together with these checks. A model_based precision tier describes the uncertainty calculation; it does not by itself justify every inferential claim. Retain restrictions reported for the fit, design, or particular statistic. For this fit, person uncertainty is posterior SD and non-person SEs use observed information.

Infit and Outfit have a reference value of 1 and summarize departures from the model. Report the statistics and the source and purpose of any review cutoff; a flag is not an automatic exclusion rule. Global fit can hide local problems. The default diagnostic_mode = "both" retains residual/EAP diagnostics and latent-integrated marginal screens. Their reference bases differ; inspect prec$fit_separation_basis before describing MnSq, ZSTD, separation, or reliability as if they were a single adequacy test.

When reporting the frequency of large residuals, state the rule and denominator. For example, Eckes (2005, p. 204) reports counts at two absolute standardized- residual cutoffs. The following shows those descriptive counts for the current EAP-residual path:

z <- diag$obs$StdResidual
residual_counts <- data.frame(
  AbsoluteZCutoff = c(2, 3),
  Evaluated = sum(is.finite(z)),
  Unavailable = sum(!is.finite(z)),
  Flagged = c(sum(abs(z[is.finite(z)]) >= 2),
              sum(abs(z[is.finite(z)]) >= 3))
)
residual_counts$Percent <- 100 * residual_counts$Flagged / residual_counts$Evaluated
residual_counts

Use the residual basis appropriate to your analysis. These are descriptive tail counts, not a calibrated global test or a universal rule that a certain percentage establishes fit. The observed-category probability screen in unexpected_response_table() is an additional rule unless explicitly disabled; its default count is not the same as counting only abs(StdResidual) >= 2.

Keep the marginal screening results visible alongside those counts:

diag_summary$marginal_fit[, c(
  "OverallRMSD", "StepGroupsFlagged", "FacetLevelsFlagged",
  "PairwiseFlaggedLevelPairs", "ReportingUse"
)]
diag$marginal_fit$thresholds
diag$marginal_fit$pairwise$thresholds
subset(diag$marginal_fit$facet_level$summary_stats, Flagged,
       c("Facet", "Level", "GroupCount", "RMSD", "MaxAbsStdResidual"))
subset(diag$marginal_fit$pairwise$pair_stats, Flagged,
       c("Facet", "Level1", "Level2", "LevelPairCount", "ExactAgreement", "ExpectedExactAgreement"))

The package’s default marginal rules use an absolute standardized-residual cutoff of 2 or a root-mean-square proportion discrepancy of 0.05; pairwise screens also use an exact/adjacent agreement-gap cutoff of 0.10. Record the rules used in your run and inspect flagged groups with plot() and plot(). These remain screening results; an acceptable overall average does not resolve local warnings.

3. Show estimates, uncertainty, and the answer

measurements <- fit_measures_table(fit, diagnostics = diag, ci_level = 0.95)
rater_table <- subset(
  measurements$table, Facet == "Rater",
  c("Level", "N", "Measure", "SE", "CI_Lower", "CI_Upper", "Infit", "Outfit")
)
rater_table <- rater_table[order(rater_table$Measure), ]

rater_apa <- apa_table(
  rater_table,
  digits = 3,
  caption = "Rater severity and fit in the synthetic MML/RSM example",
  note = paste(
    "282 ratings of 48 persons by six raters on three criteria using a 1-4 rubric.",
    "N counts rating rows. Measures and SEs are in logits; higher values mean stricter ratings.",
    "Intervals are 95% normal approximations from observed-information SEs.",
    "Rater effects are centered within facet. Infit and Outfit have reference value 1."
  )
)
rater_apa

# Display the estimates on the common logit scale
plot(fit, diagnostics = diag, show_ci = TRUE)

Pass the same diag to the table and plot so their intervals use the same SEs. Without diagnostics, Wright-map intervals use exploratory observation-table SEs, which can differ from the MML observed-information SEs reported here. Steps on this map are point estimates; their uncertainty is reported in the separate threshold table below.

apa_table() prints the table, caption, and note; its $table member holds the data for further formatting. digits rounds all numeric columns uniformly. Review statistic-specific rounding, especially for p values and intervals, and the target journal’s format before submission.

A worked explanation for this synthetic example is:

We asked how much raters differed in severity after accounting for person ability and criterion difficulty. The model includes those facets so that raw rater averages are not the only basis for comparison. Estimated rater severity ranged from -0.606 logits (R01; SE = 0.224, 95% CI [-1.046, -0.166]) to 0.412 logits (R06; SE = 0.249, 95% CI [-0.077, 0.901]), a span of about 1.018 logits. Thus, the fitted model describes R06 as stricter than R01 in these data. This span is descriptive: it is not an interval or a hypothesis test for the difference. The evidence does not establish why the raters differ or how much a particular examinee’s decision would change.

The intervals compare each centered rater effect with its reference; their overlap is not a pairwise test. A contrast needs uncertainty for that contrast, including covariance, and a multiplicity plan if several comparisons are made. Whether a difference matters in practice depends on the rubric, the score use, and any decision rule. Do not convert a logit span directly into rubric points or claim that severity differences prove biased or inconsistent scoring. For person tables, state the EAP/posterior-SD basis and the interval method explicitly; normal approximations are not posterior quantile intervals.

Keep separation, agreement, and category functioning distinct

apa_table(
  diag_summary$reliability[, c("Facet", "Levels", "Separation", "Strata", "Reliability", "RealReliability")],
  digits = 3,
  caption = "Separation and reliability by facet",
  note = "High Person separation reliability indicates distinguishable abilities; high rater separation reliability indicates distinguishable severities, not agreement. Low rater values can also reflect imprecision."
)

agreement <- interrater_agreement_table(fit, diagnostics = diag)
agreement$summary[, c(
  "OpportunityCount", "ExactAgreement", "ExpectedExactAgreement", "AdjacentAgreement"
)]

A rater separation reliability describes how distinctly rater locations are estimated relative to their errors; a high value indicates distinguishable severities, not agreement. Low values can also reflect imprecise estimates and do not establish agreement or equivalence. Neither a high nor a low value tests every rater pair or quantifies the practical size of severity differences. Person separation reliability addresses differentiation among persons under its stated error basis. Specify whether the model or misfit-adjusted (RealReliability) index is reported, and the replication/error sources relevant to the intended score use. These coefficients do not describe every possible source of measurement error.

Also identify the formula. In diag$reliability, Reliability is the separation-based ratio max(V - E, 0) / V, where V is the sample variance of finite facet estimates and E is the mean squared model SE over available SEs. The ratio is unavailable when V is zero or unavailable. Separation is sqrt(max(V - E, 0) / E) when defined, and Strata is (4 * Separation + 1) / 3; they are distinct indices, not observed groups. The underlying quantities are available directly:

diag$reliability[, c("Facet", "Levels", "ObservedVariance", "ModelErrorVariance", "Reliability")]

This is not the posterior-variance EAP reliability reported by Xiao et al. (2026, Section 2.5, Equation 4), whose denominator combines the spread of EAP scores with posterior variances. Nor is it the squared correlation with known true abilities used in their simulation. Report the package’s coefficient under its own name and basis; numerical comparisons across papers require matching definitions and scoring designs. For example, this analysis’s person separation reliability is 0.664, not a TAM EAP-reliability estimate.

Here the observed exact agreement is 54/138 = 39.1% across available paired ratings of the same person and criterion; adjacent agreement (within one category) is 85.5%. ExpectedExactAgreement is the fitted-model baseline (35.1% here), not an independent chance-agreement correction. Report the opportunity count and overlap pattern, not just a percentage. These are observed-score summaries; keep them separate from the marginal diagnostic screens above. Use agreement$pairs when differences among rater pairs matter.

For observed QWK, ordinal Krippendorff alpha, Gwet AC2 or an aligned score-source comparison, follow Rating agreement and score comparisons. That tutorial shows the matching keys and sample audits, and distinguishes these chance-agreement definitions from the fitted-model expectation above.

scale <- rating_scale_table(fit, diagnostics = diag)
scale$category_table[, c("Category", "Count", "AvgMeasure", "ExpectedMeasure", "Infit", "Outfit")]
plot(scale)
scale$notes

AvgMeasure is the average oriented generating measure among responses in each observed category; ExpectedMeasure is its model expectation. Category residual mean squares are descriptive and have no universal unit baseline.

Check the threshold owner and the denominator before interpreting sparse use:

scale$model_context
scale$owner_category_table[, c(
  "StepFacet", "CategoryLabel", "RawCount", "Count", "Percent", "UsageStatus"
)]

This RSM uses a single Common threshold family. For PCM, StepOwner records the selected step_facet, and StepFacet identifies each level of that facet. For example, rater-owned PCM thresholds describe each rater’s category transitions; criterion-owned thresholds describe each criterion’s transitions. RawCount counts ratings and Count sums their weights. The percentage denominator is the total weight within that owner level, not the number of persons. Local zero counts remain visible even if a category is used elsewhere. No-exposure percentages are unavailable, and a zero count alone does not establish structural absence.

The threshold table already includes uncertainty from the supplied diagnostics:

steps <- scale$threshold_table
steps[, c("StepFacet", "Step", "Estimate", "SE", "CI_Lower", "CI_Upper")]
steps[, c("StepFacet", "Step", "SE_Status", "CIEligible", "CIUse",
          "UncertaintySourceStatus")]

Step_1, Step_2, and Step_3 are the transitions 1–2, 2–3, and 3–4 here. The supplied diagnostics use MML observed-information SEs and 95% normal intervals. Their estimates and owner/step keys must match the fit. New diagnostics also record the calibration identity; older diagnostics without that identity retain their numbers with a review status. Source verification does not establish interval coverage. Check availability and reporting status before quoting intervals. Calling rating_scale_table(fit) without diagnostics returns points and category counts without computing covariance; its SEs are unavailable.

Retain StepFacet in PCM output and original category labels when reporting recoded scores. A centered step offset is not the full ability-axis crossing: that crossing also includes the applicable facet locations. Its SE, and the SE of a difference between steps, require the relevant covariance terms. Do not substitute a criterion-location SE or an individual step SE. These distinctions matter when preparing tables such as the criterion-specific thresholds and SEs in Eckes (2005, Table 5).

The bars show observed category counts; the line shows fitted expected counts. Report category frequencies, threshold estimates and ordering, and any sparse or unused categories. Explain what the pattern means for the rubric. These checks can motivate rubric review; they do not automatically justify merging categories, changing scores, or claiming unidimensionality.

If the question concerns observed versus adjusted scores

Eckes (2005, Table 3) places observed scores, adjusted scores, logit measures, SEs, and rating counts together. State the adjustment reference when using a similar presentation. Here the reference is zero for the other facets:

fair <- fair_average_table(fit, diagnostics = diag, reference = "zero")
person_scores <- fair$raw_by_facet$Person
person_scores <- person_scores[order(person_scores$Level), ]
head(person_scores[, c(
  "Level", "TotalCount", "ObservedAverage", "FairZ", "Measure", "ModelSE"
)], 6)

These are the first six person IDs, selected for illustration. FairZ is an expected score on the fitted 1–4 scale, at the person’s estimated ability with the other facet references set to zero. It is not a z-score. ModelSE here is the posterior SD of person ability in logits, not the SE of FairZ. The RSM/PCM fair_average_table() does not supply fair-score SEs. Its mean-reference alternative, FairM, uses mean other-facet measures; for non-person rows that also includes the mean person measure. Neither averages over the actual assignment distribution.

A score-use claim needs the decision thresholds, rounding rule, reference environment, uncertainty treatment, and full-sample counts of changed decisions. The six-row preview is not that analysis. Statistical adjustment alone does not establish fairness or authorize replacing operational scores.

If the score will determine pass or fail

An additional score-use question is: under a specified future rating plan, how often would the scoring rule misclassify a person, and how precise is measurement near the cut? For this illustration, every person receives six ratings from two named raters across the three criteria. The true pass cut is 0 logits; the operational sum rule passes sums of at least 9. The sum uses category indices 0–3, so 9 corresponds to an original 1–4 score sum of 15 for six complete ratings. These are illustrative decisions, not recommended cuts or cuts inferred from the fitted data.

decision_plan <- expand.grid(
  Rater = sort(unique(as.character(toy$Rater)))[1:2],
  Criterion = sort(unique(as.character(toy$Criterion))),
  stringsAsFactors = FALSE
)
decisions <- classification_table(
  fit, rating_plan = decision_plan,
  cut_scores = c(Pass = 0), score_cuts = 9, operational_cuts = 0,
  theta = c(-1, -0.25, 0, 0.25, 1),
  theta_weights = c(0.10, 0.20, 0.30, 0.25, 0.15)
)
decisions$cuts
decisions$table[, c("Theta", "Method", "Accuracy", "Consistency", "SEM", "Status", "SourceStatus")]
decisions$marginal

The five ability points and their masses define an explicitly hypothetical discrete population. They are not fitted EAP scores or a continuous-normal quadrature rule. Equality enters the higher category: the mass at 0 belongs to the true-pass group. Thus the true-fail and true-pass proportions are 0.30 and 0.70. Without supplied population weights, the function returns conditional results but does not invent a population average.

lee_sum calculates the finite summed-score distribution for the stated sum rule. rudner_normal approximates decisions based on an ML ability estimate and its stated ability cut. Those operational rules need not agree; the expected sum at 0 in $cuts is a diagnostic, not an instruction to round a cut. Do not rank the two methods’ accuracy without first establishing a common decision rule. The normal approximation can be poor for short plans or extreme abilities. Neither route evaluates the earlier EAP scoring rule.

SEMAtTrueCut and the nearby SEM values are in ability logits, conditional on this calibration and locally independent ratings. They exclude uncertainty in estimated facet/step parameters and are not posterior SDs. Consistency concerns two conditionally independent repetitions of the same fixed plan; it does not generalize to different raters, retake policies or random tasks. This API supports ordinary RSM/PCM; two-family GMFRM is outside its scope.

Report an error’s denominator as part of its name. For a binary decision, Overclassification in $marginal is the population joint probability of false pass, and Underclassification is that of false fail. The joint matrix allows three different, useful descriptions of each error:

error_rates <- do.call(rbind, lapply(
  split(decisions$classification_matrix, decisions$classification_matrix$Method),
  function(d) {
    false_pass <- sum(d$Probability[d$TrueCategory == 1 & d$ObservedCategory == 2])
    false_fail <- sum(d$Probability[d$TrueCategory == 2 & d$ObservedCategory == 1])
    joint <- c(false_pass, false_fail)
    true_group <- c(sum(d$Probability[d$TrueCategory == 1]),
                    sum(d$Probability[d$TrueCategory == 2]))
    decision_group <- c(sum(d$Probability[d$ObservedCategory == 2]),
                        sum(d$Probability[d$ObservedCategory == 1]))
    # A missing or zero conditioning mass gives an undefined rate, not zero.
    given_true <- given_decision <- rep(NA_real_, 2)
    ok <- is.finite(true_group) & true_group > 0
    given_true[ok] <- joint[ok] / true_group[ok]
    ok <- is.finite(decision_group) & decision_group > 0
    given_decision[ok] <- joint[ok] / decision_group[ok]
    data.frame(
      Method = d$Method[1], Error = c("False pass", "False fail"),
      JointProbability = joint, TrueGroupProbability = true_group,
      RateGivenTrueGroup = given_true, DecisionGroupProbability = decision_group,
      RateGivenDecisionGroup = given_decision,
      Status = d$Status[1], SourceStatus = decisions$source$status
    )
  }
))
rownames(error_rates) <- NULL
error_rates

For false pass, the denominators are everyone, people truly below the cut, and people declared to pass, respectively. For false fail, the last two are people truly at/above the cut and people declared to fail. For example, a joint false-pass probability of 0.06 in a population with 0.30 true failures would mean a conditional false-pass rate of 0.06/0.30 = 0.20; those are illustrative arithmetic values, not the fitted results above. Missing method probabilities remain missing in the calculation; no na.rm = TRUE discards them. Zero denominators give NA. Retain Status and SourceStatus as well as the rates: numerical availability does not validate the source calibration, population, plan or operational cut. Inspect decisions$notes and ?classification_table before interpreting or exporting the results.

A short Methods and Results example

The following is an original write-up of this synthetic illustration, using the outputs above. It is not a reconstruction of any cited study. The research questions determine which of these supporting results belong in a real article’s main text or appendix.

Methods. We examined how much raters differed in severity after accounting for person ability and criterion difficulty. The synthetic dataset represented 48 persons, six raters, and three criteria scored from 1 to 4. Each person was assigned two raters across three criteria in an incomplete but connected design. Of 288 planned ratings, six were deliberately omitted in the example; all 282 observed ratings were retained. The raw rating mean was 2.38 (SD = 1.00), and individual raters contributed 38–56 ratings.

We fitted an RSM with person, rater, and criterion effects using marginal maximum likelihood in mfrmr (software versions are recorded below). Shared category steps represented a common transition structure across criteria. The working person distribution was fixed to standard normal, person scores were EAP estimates, and rater and criterion effects were centered within facet. Higher rater estimates indicated stricter scoring. The fit used 31 quadrature points, an iteration limit of 400, and the recorded L-BFGS-B optimizer; the requested relative tolerance was 1e-9 and the effective selected-stage tolerance was 1e-13. We examined the stored readiness and precision checks, residual/EAP and marginal diagnostics, and category use. Rater and step intervals were 95% normal approximations from observed-information SEs. Person uncertainty was reported as posterior SD.

Results. The fit’s readiness and precision checks passed. Estimated rater severity spanned 1.018 logits, from -0.606 (R01; SE = 0.224) to 0.412 (R06; SE = 0.249); the individual intervals are reported in the rater table above. Thus, the fitted model described a range of rater severity after accounting for the included facets. This descriptive span did not test a pairwise contrast or quantify changes to assessment decisions.

Global residual-path Infit and Outfit were 0.866 and 0.857. Seven of 282 standardized residuals (2.48%) had absolute values of at least 2, and one (0.35%) reached 3. The marginal screens flagged R06 and the R04–R05 rater pair under the stated rules; these local signals require follow-up despite an overall marginal RMSD of 0.0062. Categories 1–4 occurred 62, 96, 78, and 46 times. Shared step estimates were ordered (-1.223, 0.163, and 1.060 logits; SEs = 0.169, 0.167, and 0.177). Person and rater separation reliabilities were 0.664 and 0.677. Observed exact agreement was 54/138 (39.1%) of available paired ratings of the same person and criterion; agreement within one category was 85.5%. These statistics describe distinct aspects of this fit. They do not establish the cause of severity differences, rubric validity, or the adequacy of an operational scoring decision. The illustration does not evaluate a different population model or a pairwise inferential contrast, and it does not empirically validate classification decisions. The fixed-plan example above describes model-implied decision probabilities under hypothetical cuts and population weights.

For an empirical report, add recruitment, participant and rater characteristics, training and scoring procedures, rubric rationale, actual missingness reasons, and the study-level information in the map below. Report unresolved diagnostics and relevant sensitivity results even if the optimizer converged. Never fill unknown study details with assumptions from the synthetic example.

4. Review gaps and generate draft text

chk <- reporting_checklist(fit, diagnostics = diag, include_references = TRUE)
chk$section_summary
subset(chk$checklist, !DraftReady, c("Section", "Item", "NextAction"))

res <- mfrm_results(fit, diagnostics = diag, include = "publication")
report <- mfrm_report(res, style = "apa")
report$first_screen[, c("Area", "Status", "MainIssue", "NextAction")]

DraftReady means that drafting material is available with the documented caveats. Readiness flags and report$report_gaps concern the supplied analysis objects, not the completeness or validity of the research article. Check report$claim_readiness for restrictions on a planned claim. Add a bias, linking, or residual-structure analysis only if it answers a study question; an unrequested optional analysis is not necessarily a reporting omission.

apa <- build_apa_outputs(
  fit,
  diagnostics = diag,
  context = list(
    assessment = "Synthetic writing ratings",
    setting = "Illustration of the reporting workflow",
    scale_desc = "1-4 rubric scale",
    rater_facet = "Rater"
  )
)
cat(apa$report_text)

The draft reuses the supplied diagnostics. This example requested residual_pca = "none", so it contains no residual PCA findings. To report PCA, first request the intended scope in diagnose_mfrm() and review those results before regenerating the draft.

Verify every supplied context label against the study record. For example, scale_desc is author-provided text, not a check of the observed score range. summary(apa)$content_checks checks the generated content against its output contract; it cannot verify recruitment, rater training, a causal explanation, or whether the prose answers the question. Edit the draft to connect the numerical findings to the question, as in the worked example above.

Manuscript coverage map

Use this map while writing. The package supplies numerical evidence for part of a paper; the right-hand column identifies information to add or justify. Select items relevant to the design and intended claims. This is a practical MFRM reporting guide, not a certification of journal compliance.

What the reader needs	Evidence available in `mfrmr`	What the author supplies or verifies
Problem, intended score use, and research questions	No fitted-object substitute.	Who uses the scores, for what decision, what is unresolved, and the expected contribution. Identify confirmatory and exploratory questions and any preregistration.
Participants and sample rationale	`data_summary$overview`, `$facet_overview`; `results$overview`.	Recruitment, inclusion criteria, relevant characteristics, setting and dates, sampling unit, sample-size rationale, and generalization target. Distinguish people from rating rows.
Raters, tasks, and rubric	Input facet labels and `data_summary$score_distribution`.	Rater expertise, recruitment and training, calibration, scoring instructions, task selection, category meanings, administration, blinding where relevant, and rubric rationale. For automated scoring, record the model/version, scoring date, prompts, exemplars and other scoring settings. Labels alone do not document these.
Assignment, overlap, and linkage	`data_summary$structural_missingness`, `$design_connectivity`, `$linkage_summary`; diagnostic subset information.	Who rated whom, crossing/nesting, planned overlap, occasions/forms, allocation process, and substantive justification for common-scale comparisons. Supply the roster and any anchor provenance.
Missing data, duplicates, and exclusions	`data_summary$missing`, `$duplicate_cell_summary`, `$row_retention`; `results$row_retention`.	Missing codes and reasons, planned versus absent ratings, treatment of repeated records, exclusion rules and counts at each stage, and missingness assumptions. Explain differences between preparation and fitting counts.
Model and estimand	`results$settings_overview`, `$population_overview`; model call and `specifications_report(fit)`.	Why RSM/PCM/GPCM and MML/JML answer the question; population distribution, threshold structure, constraints, orientation, weights, anchors, and any interactions. State what score or contrast is estimated.
Computation and reproducibility	`results$overview`, `$decision`; recorded estimation controls and replay/manifest export.	Software versions, optimizer and integration settings, tolerances, iteration limits, convergence problems and remedies, and relevant numerical sensitivity. Retain the exact analysis script.
Estimates and uncertainty	`as.data.frame(fit)`; `fit_measures_table()`; person/facet/step summaries.	Units, direction, reference constraints, SE and interval basis and level, uncertainty for planned contrasts, effect magnitude, and practical interpretation. Do not invent unavailable contrast tests.
Model fit and local diagnostics	`diag_summary$overall_fit`, `$diagnostic_basis`; `plot()`, `unexpected_response_table()`, optional residual PCA.	Which assumptions each analysis probes, diagnostic mode, thresholds and their rationale/source, inspected flags, and follow-up decisions. Global fit does not establish validity; PCA does not prove unidimensionality.
Precision, separation, and agreement	`prec$profile`, `$checks`, `$fit_separation_basis`; `diag_summary$reliability`; `agreement$summary` and `$pairs`.	Interpreted score, relevant error sources/replications, index definition, adjustment and uncertainty basis, paired-rating denominator and overlap. Separate rater severity, inconsistency, agreement, and person-score precision.
Category functioning	`scale$category_table`, `$owner_category_table`, `$model_context`, `$threshold_table` and `$notes`; stored step SEs and status are joined from matching diagnostics.	Intended categories, threshold owner, counts and denominators, sparse-category implications, rubric-based interpretation, and reasons and consequences of any recoding. Distinguish step uncertainty from facet-location and full-crossing uncertainty.
Comparisons, bias, DFF, or linking, when relevant	`compare_mfrm()`, `build_model_choice_review()`, `estimate_bias()`, `analyze_dff()`, or `build_linking_review()`.	Planned contrasts, same-data/comparability checks, numerical sensitivity, reference groups and anchor rationale, uncertainty, multiplicity, and departures from plan. Screening flags alone do not establish unfairness or drift absence.
Latent regression or simulation, when relevant	Population coefficients/coding/caveats; recovery/evaluation helpers.	Covariate coding and omissions; or generating conditions, repetitions, random seeds, failures, evaluation metrics and Monte Carlo uncertainty. Explain each analysis’s role and distinguish empirical results from results under simulation assumptions.
Answer, limitations, and implications	Tables, plots, and generated draft text supply supporting results.	Answer each question in words, with direction, magnitude, uncertainty, conditions, alternative explanations, limitations, and implications for the intended score use. Do not infer validity or causality from fit alone.
Ethics and transparency	Local export bundle and software citation records cover only part of this.	Applicable ethics approval/consent, confidentiality, funding and conflicts, data/materials/code availability or access restrictions, and complete references. Review the actual files before sharing.

Journal-level reporting guidance emphasizes linking questions, design, analyses, and interpretation; use the applicable JARS-Quant items for your study. The Standards for Educational and Psychological Testing additionally connect documentation and precision evidence to the proposed score interpretation and use, including the error sources represented by a reliability coefficient (Appelbaum et al., 2018; AERA, APA, & NCME, 2014, Chapters 2 and 7).

5. Save the analysis and complete the references

# A new temporary folder makes this example repeatable.
# For your study, choose a new permanent project folder instead.
output_dir <- tempfile("mfrmr-report-")
bundle <- export_mfrm_bundle(
  fit,
  diagnostics = diag,
  data = toy,
  output_dir = output_dir,
  prefix = "analysis01",
  include = c(
    "core_tables", "checklist", "dashboard", "apa",
    "summary_tables", "manifest", "script", "html"
  ),
  acknowledge_sensitive = TRUE
)
selected_files <- subset(
  bundle$written_files,
  Component %in% c("core_measures", "replay_script", "bundle_html")
)
data.frame(
  Component = selected_files$Component,
  File = basename(selected_files$Path)
)

Full paths and data-handling notes remain in bundle$written_files. data = toy includes the original input CSV needed by the fit replay script. Omitting data leaves a your_data.csv placeholder to replace manually. The bundle’s core step CSV contains point estimates; save the joined threshold table below as well when reporting its SEs, intervals and source status.

Save the selected manuscript table, its caption and note, the threshold uncertainty table, and the draft carrying your supplied context:

# Keep full precision in CSV; the displayed table is rounded to three decimals
write.csv(rater_table, file.path(output_dir, "Table1-raters.csv"), row.names = FALSE)
writeLines(capture.output(print(rater_apa)), file.path(output_dir, "Table1-raters.txt"))
write.csv(steps, file.path(output_dir, "Step-uncertainty.csv"), row.names = FALSE)
write.csv(scale$owner_category_table, file.path(output_dir, "Category-owner-counts.csv"), row.names = FALSE)
writeLines(scale$notes, file.path(output_dir, "Category-notes.txt"))
saveRDS(list(classification = decisions, error_rates = error_rates), file.path(output_dir, "Decision-analysis.rds"))
write.csv(error_rates, file.path(output_dir, "Decision-error-rates.csv"), row.names = FALSE)
writeLines(c(decisions$notes,
  "False pass conditions on true failure or declared pass; false fail on true pass or declared failure.",
  "JointProbability uses the whole supplied population; missing/zero conditioning mass gives an undefined conditional rate.",
  "Use Decision-analysis.rds for the rating plan, category map, cuts, population weights and source assessment."),
  file.path(output_dir, "Decision-notes.txt"))
write.csv(mfrmr_example_operational_design, file.path(output_dir, "Planned-ratings.csv"), row.names = FALSE)
write.csv(data_summary$structural_missingness, file.path(output_dir, "Planned-missingness.csv"), row.names = FALSE)
writeLines(apa$report_text, file.path(output_dir, "Methods-Results-draft.txt"))

# Draw to the file device, then close it even if drawing fails
png(file.path(output_dir, "Figure1-Wright.png"), width = 1800, height = 1500, res = 200)
tryCatch(
  figure <- plot(fit, diagnostics = diag, show_ci = TRUE),
  finally = dev.off()
)
write.csv(plot_data(figure, "notes"),
  file.path(output_dir, "Figure1-Wright-notes.csv"), row.names = FALSE)
writeLines(c(
  "Figure 1. Person, facet, and step locations in the synthetic MML/RSM analysis.",
  "282 ratings; 48 persons, six raters, three criteria; rubric 1-4.",
  "The histogram shows person EAP estimates. All locations are in logits.",
  "Higher rater/criterion values mean stricter scoring/greater difficulty; effects are centered within facet.",
  "Facet whiskers are 95% normal intervals using MML observed-information SEs from the same diagnostics as Table 1.",
  "Steps are shared adjacent-category transitions shown as point estimates; see Step-uncertainty.csv for their intervals.",
  "The display does not test rater contrasts or establish unidimensionality."
), file.path(output_dir, "Figure1-Wright-caption.txt"))

# Filenames are shown here; output_dir retains the folder location.
list.files(output_dir, pattern = "^(Table1|Step-uncertainty|Category|Decision|Planned|Figure1|Methods-Results)")

Table1-raters.csv retains unrounded values and does not contain the table caption or note; keep the paired text file. The PNG likewise needs its accompanying caption. The bundle generates its own generic APA draft, so retain Methods-Results-draft.txt for the draft with the context supplied above. Check and edit that text against the study record before manuscript use. Decision-analysis.rds separately retains the fixed-plan calculation, its assumptions and the error-rate denominators. readRDS() reopens those results without fitting. Keep it with Decision-error-rates.csv and Decision-notes.txt; the generic result bundle does not automatically include this plain-list output. Figures omit titles and explanatory footnotes by default. Keep these separately in the manuscript: plot_data(figure, "title") retrieves the suggested title, and plot_data(figure, "notes") retrieves descriptions and cautions. export_mfrm_results(..., include = "plots") saves the suggested title and notes automatically in a matching _plot_{type}_notes.csv beside each PNG. The multi-plot fit route writes separate image/notes pairs for fit_wright_map, fit_pathway_map and fit_category_characteristic_curves, preserving all three figures. The starter index links each image to its corresponding notes. These notes also survive saveRDS() and readRDS(). Axis labels, legends, panel identifiers and unresolved-value symbols remain in the image. Use the journal’s figure dimensions and resolution for the final submission. Keep the planned-rating roster alongside its missingness summary: the observed input CSV alone cannot identify which unobserved combinations were planned.

To refit in a new R session with the same package version, run source("/path/to/analysis01_replay.R", chdir = TRUE), replacing the path with the exported file’s actual location. chdir = TRUE lets the script find the adjacent input CSV. This reruns the fit and diagnostics and writes a replayed_bundle folder; it does not recreate your edited prose or selected manuscript figures. Keep the analysis script containing those steps too.

The archive does not replace the study description or deidentify the data. Inspect its text as well as its tables, and use a new directory for a new analysis.

# Get the citation for the package version actually used, and for R
citation("mfrmr")
citation()

# Methodological reading prompts, not complete bibliography entries
chk$references

# Retain the computing environment with the analysis record
sessionInfo()

chk$references contains abbreviated citations and topics. Obtain and verify complete bibliographic records for the methods actually used, for example in Zotero, and cite the relevant source next to the model or diagnostic decision. Do not cite every listed method simply because it is available. A software citation identifies the tool; it does not replace methodological references.

For the conditional analyses in the map, runnable setups and reporting limits are provided in help("compare_mfrm"), help("estimate_bias"), and the mfrmr-linking-and-dff, mfrmr-gpcm-scope, and mfrmr-mml-and-marginal-fit vignettes. For latent regression, see the population model sections of help("fit_mfrm") and help("mfrmr_reporting_and_apa"). Fit any additional model explicitly before using it in a comparison or coefficient table.