--- title: "Binary ODA: Migraine Attacks in a Clinical Trial" author: "oda" date: "`r Sys.Date()`" output: rmarkdown::html_vignette vignette: > %\VignetteIndexEntry{Binary ODA: Migraine Attacks in a Clinical Trial} %\VignetteEngine{knitr::rmarkdown} %\VignetteEncoding{UTF-8} --- ```{r setup, include = FALSE} knitr::opts_chunk$set(collapse = TRUE, comment = "#>") ``` ## Research question Appleton (1995) reported a clinical trial in which 67 patients experiencing migraine were randomised to one of two treatments, and the number of migraine attacks was recorded.^[Appleton DR (1995). Pitfalls in the interpretation of studies: III. *Journal of the Royal Society of Medicine*, 88, 241-243.] Various conventional methods - Student's *t*-test (including square-root and log transformations), the Mann-Whitney *U*-test, and a Poisson normal test - either failed to reach conventional significance or violated their underlying assumptions. The analyst ultimately discretised the count at 0 vs. >=1 and validated the split with a one-tailed Fisher's exact test (*p* < 0.022). Because ODA is invariant under any monotonic transformation and requires no distributional assumptions, it can analyse the raw count directly. ODA is used here to determine whether number of migraine attacks discriminates treatment arm, and to quantify the strength of the association. ## Data Treatment arm (0 = Treatment 1, 1 = Treatment 2) is the class variable; number of migraine attacks (0-7, ordered) is the attribute. Published cell frequencies are reconstructed directly into observation-level vectors - no external data file is required. ```{r data} library(oda) # Cross-classification: rows = attacks (0-7), cols = treatment arm. # T1 (0) T2 (1) # 0 att: 13 5 # 1 att: 9 13 # 2 att: 4 6 # 3 att: 2 1 # 4 att: 1 2 # 5 att: 1 3 # 6 att: 3 3 # 7 att: 0 1 treatment <- c( rep(0L, 13), rep(1L, 5), # attacks = 0 rep(0L, 9), rep(1L, 13), # attacks = 1 rep(0L, 4), rep(1L, 6), # attacks = 2 rep(0L, 2), rep(1L, 1), # attacks = 3 rep(0L, 1), rep(1L, 2), # attacks = 4 rep(0L, 1), rep(1L, 3), # attacks = 5 rep(0L, 3), rep(1L, 3), # attacks = 6 rep(0L, 0), rep(1L, 1) # attacks = 7 ) attacks <- c( rep(0L, 18), rep(1L, 22), rep(2L, 10), rep(3L, 3), rep(4L, 3), rep(5L, 4), rep(6L, 6), rep(7L, 1) ) table(attacks, treatment, dnn = c("Migraine Attacks (0-7)", "Treatment (0=T1, 1=T2)")) ``` ## Fit the ODA model Number of attacks is an ordered integer; ODA scans it as an ordered attribute (no categorical flag), consistent with the MegaODA reference analysis. No directional hypothesis was specified *a priori*, so the default nondirectional search (`direction = "both"`) is used. Leave-one-out (LOO) jackknife validity analysis is included. ```{r fit-canonical, eval=FALSE} # Canonical reference run (mc_iter = 25000L; not evaluated in CRAN vignette) fit <- oda_fit( x = attacks, y = treatment, attr_type = "ordered", mc_iter = 25000L, loo = "on" ) ``` ```{r fit} # CRAN-safe run: mc_iter = 500L for vignette rendering speed. # Training rule, ESS, and confusion matrix are identical to the canonical run. fit <- oda_fit( x = attacks, y = treatment, attr_type = "ordered", mc_iter = 500L, mc_seed = 42L, loo = "on" ) ``` ## Rule and confusion matrix ```{r print-fit} print(fit) ``` ODA identified a single cut at 0.5, consistent with Appleton's hand-chosen spline: - If attacks <= 0.5 (zero attacks) -> predict Treatment 1 (0) - If attacks > 0.5 (one or more attacks) -> predict Treatment 2 (1) ```{r confusion} # Confusion matrix: actual treatment (rows) x predicted treatment (cols) conf_mat <- matrix( c(fit$confusion$TN, fit$confusion$FP, fit$confusion$FN, fit$confusion$TP), nrow = 2L, byrow = TRUE, dimnames = list(Actual = c("T1(0)", "T2(1)"), Predicted = c("T1(0)", "T2(1)")) ) print(conf_mat) ``` ## ESS / PAC / PV interpretation ```{r metrics} summary(fit) ``` ```{r pv} # Predictive value: accuracy when the model makes a prediction into each class pv_t1 <- fit$confusion$TN / (fit$confusion$TN + fit$confusion$FN) pv_t2 <- fit$confusion$TP / (fit$confusion$TP + fit$confusion$FP) cat("PV Treatment 1 (0):", round(pv_t1 * 100, 1), "%\n") cat("PV Treatment 2 (1):", round(pv_t2 * 100, 1), "%\n") ``` - **PAC (sensitivity per class):** 39.4% for Treatment 1 patients, 85.3% for Treatment 2 patients. Because 50% correct per class is expected by chance, the model classifies Treatment 2 patients well above chance while Treatment 1 classification is below chance - a notable asymmetry. - **ESS = 24.69%** is marginally below the conventional 25% threshold for moderate effect strength.^[Yarnold, P.R., & Soltysik, R.C. (2005). *Optimal Data Analysis: A Guidebook with Software for Windows.* Washington, D.C.: APA Books.] The asymmetry reflects a greater concentration of zero-attack patients in Treatment 1 relative to Treatment 2. - **PV:** When the model predicts Treatment 1, it is correct ~72.2% of the time; when it predicts Treatment 2, ~59.2%. ## Monte Carlo and LOO validity The MC p-value and LOO result are shown in the `summary` output above. - **MC p-value (one-tailed, non-directional permutation):** Each Monte Carlo permutation randomly shuffles class labels and refits ODA searching *both* directions, exactly as in the training analysis. The reported p is the proportion of permuted ESS values that equal or exceed the observed ESS. Because the permutation distribution accounts for optimizing over both directions, the MC p is more conservative for a non-directional analysis. The MegaODA reference value is p ≈ 0.086 (not significant at α = 0.05). - **LOO stability:** The leave-one-out ESS equals the training ESS (24.69%), indicating the rule is completely stable — no single observation materially alters the model. - **LOO Fisher exact p (one-tailed):** Per MPE p. 34, hold-out p is always one-tailed: "the null hypothesis is that the training model will not replicate when it is used to classify observations in the hold-out sample." The Fisher exact test (alternative = "greater") tests whether the LOO confusion matrix reflects above-chance classification. This test does *not* adjust for the direction search performed during training. Statistical significance is confirmed in LOO, consistent with Appleton's original one-tailed Fisher test. **Why MC p and LOO p diverge for non-directional analyses:** MC permutation p is more conservative than LOO Fisher p when the analysis is non-directional. The MC test accounts for the fact that training optimized over both directions (making the permutation baseline harder to beat); the LOO Fisher test does not apply that adjustment. Both values are valid for their respective purposes: MC p assesses training model significance with direction-search adjustment; LOO Fisher p assesses replication of the fixed training rule in held-out data. The divergence narrows or disappears when a directional hypothesis is declared *a priori* (see Notes). ## Notes on reproducibility and current scope **Fixture parity.** The training rule, confusion matrix, and ESS are verified against MegaODA.exe output in the package test suite (`tests/testthat/test-fixture-vignettes.R`, Example 4). **MC p-value calibration.** The MC p shown here reflects `mc_iter = 500L` for CRAN build speed and will differ from MegaODA's reported value (p = 0.086 at 25000 iterations). With only 500 permutations the estimate is noisy (Monte Carlo standard error ~1-2%). Use the canonical run with `mc_iter = 25000L` (chunk `fit-canonical`, `eval=FALSE`) for publication-quality results. Training ESS and confusion matrix are unaffected by `mc_iter`. **Directional analysis.** The original analysis did not specify a directional hypothesis *a priori*; the nondirectional default (`direction = "both"`) is therefore appropriate. If a directional ordered hypothesis had been specified in advance (e.g., more attacks predicts Treatment 2), `direction = "greater"` or `direction = "less"` could be used to enforce MPE Chapter 2 binary ordered directional ODA and obtain a one-tailed p-value.