Survey research often works with far fewer than the 30-or-more cases that justify asymptotic approximations: pilot studies, specialised populations, classroom studies, and early-stage organisational diagnostics routinely land at n = 10 to 25. Below that conventional threshold, p-values from tests that assume large-sample normality can be unreliable, and point estimates carry wide, often asymmetric uncertainty that a single p-value does not convey.
surveyframe’s small-sample tools do three things: flag when a sample has crossed below the n = 30 threshold, prefer exact or distribution-free alternatives to the asymptotic default where one exists, and pair every affected estimate with a confidence interval rather than a point value alone. Together they surface the cases where design judgement matters most, leaving that judgement itself to the researcher.
sample_size_plan() estimates a required sample size for
a design and attaches the same small-sample advisory used elsewhere in
the package when the estimate itself falls below 30.
sample_size_plan(type = "t_test", groups = 2)
#> Sample-Size Plan
#> Target: t_test
#> Estimated n: 128
#> Alpha: 0.050 Power: 0.80A worked instrument in this vignette targets n = 20 complete responses, well below that planning estimate. The instrument below has a two-level grouping question, a numeric outcome, and a binary outcome for the logistic example later in this vignette.
group_cs <- sf_choices(id = "grp_cs", values = c("control", "treatment"),
labels = c("Control", "Treatment"))
converted_cs <- sf_choices(id = "conv_cs", values = c("no", "yes"),
labels = c("No", "Yes"))
items <- list(
sf_item(id = "group", label = "Study arm", type = "single_choice", choice_set = "grp_cs"),
sf_item(id = "outcome", label = "Outcome score", type = "numeric"),
sf_item(id = "converted", label = "Converted (yes/no)", type = "single_choice",
choice_set = "conv_cs")
)
study <- sf_instrument(
title = "Small-sample pilot",
version = "0.1.0",
authors = "surveyframe",
components = c(list(group_cs, converted_cs), items)
)
study
#> <sframe>
#> Title: Small-sample pilot
#> Version: 0.1.0
#> Items: 3
#> Scales: 0
#> Analysis: 0 block(s)
#> Status: not validatedassumption_report() and sample_size_plan()
both attach an advisory element once a relevant n drops
below 30. It prints automatically and is also available as a field for
programmatic checks.
n20 <- data.frame(score = rnorm(20, mean = 50, sd = 10))
n50 <- data.frame(score = rnorm(50, mean = 50, sd = 10))
ar_small <- assumption_report(n20, variables = "score")
ar_small$advisory
#> [1] "Small sample detected (n = 20). For these assumption checks, consider non-parametric alternatives. Asymptotic p-values may be unreliable. Bootstrap or exact confidence intervals are provided where available."
ar_large <- assumption_report(n50, variables = "score")
is.null(ar_large$advisory)
#> [1] TRUEThe Mann-Whitney runner now reports the Hodges-Lehmann shift estimate alongside the rank-biserial effect size, with a confidence interval on the shift rather than a point estimate alone.
pilot <- data.frame(
group = rep(c("control", "treatment"), each = 10),
outcome = c(rnorm(10, 48, 8), rnorm(10, 55, 8)),
converted = sample(c("no", "yes"), 20, replace = TRUE, prob = c(0.6, 0.4)),
covariate = rnorm(20, 0, 1)
)
sf_plan(study) <- list(
list(id = "RQ1",
research_question = "Does the treatment arm score higher than control?",
family = "group_comparison", method = "mann_whitney",
roles = list(group = "group", outcome = "outcome"),
options = list(alpha = 0.05))
)
mw_results <- run_analysis_plan(pilot, study)
results_table(mw_results)| RQ | Research question | Method | Result (APA) |
|---|---|---|---|
| RQ1 | Does the treatment arm score higher than control? | U = 34, z = -1.17, p = 0.241, r = 0.26 [0.02, 0.65], Hodges-Lehmann shift = -3.99 [-14.38, 5.36] |
The paired runner reports a Hodges-Lehmann pseudomedian for the within-pair shift, again with an interval rather than a point value.
before <- rnorm(8, 50, 6)
after <- before + rnorm(8, 4, 5)
sf_plan(study) <- list(
list(id = "RQ2",
research_question = "Did scores change from before to after?",
family = "group_comparison", method = "wilcoxon_pair",
roles = list(before = "before", after = "after"),
options = list(alpha = 0.05))
)
wp_results <- run_analysis_plan(data.frame(before = before, after = after), study)
results_table(wp_results)| RQ | Research question | Method | Result (APA) |
|---|---|---|---|
| RQ2 | Did scores change from before to after? | V = 0, z = -2.45, p = 0.014, r = 0.87 [0.89, 0.90], pseudomedian = -6.62 [-9.04, -4.38] |
For a 2x2 table, fisher_exact now attaches an exact
odds-ratio confidence interval alongside the test’s p-value. The
interval is only defined for a 2x2 table, and is dropped (not
approximated) when simulate_p_value is requested, since
base R’s fisher.test() cannot return both together.
sf_plan(study) <- list(
list(id = "RQ3",
research_question = "Is conversion associated with study arm?",
family = "association", method = "fisher_exact",
roles = list(row = "group", column = "converted"),
options = list(alpha = 0.05))
)
fe_results <- run_analysis_plan(pilot, study)
results_table(fe_results)| RQ | Research question | Method | Result (APA) |
|---|---|---|---|
| RQ3 | Is conversion associated with study arm? | Fisher’s exact test, p = 1.000, phi = 0.12 |
bootstrap_ci() computes a bootstrap interval for an
arbitrary statistic, useful when no exact interval exists for the
estimator you need. It is already exported and used elsewhere in the
package, applied here to the median of the pilot outcome scores.
Ordinary logistic regression can fail to converge or produce severely
biased estimates with small samples or separated data. Firth’s
penalised-likelihood correction (Firth, 1993) addresses both.
surveyframe’s firth_logistic runner needs the optional
logistf package.
sf_plan(study) <- list(
list(id = "RQ4",
research_question = "Does the covariate predict conversion?",
family = "regression", method = "firth_logistic",
roles = list(dependent = "converted", predictors = "covariate"),
options = list(conf.level = 0.95))
)
firth_results <- run_analysis_plan(pilot, study)
results_table(firth_results)| RQ | Research question | Method | Result (APA) |
|---|---|---|---|
| RQ4 | Does the covariate predict conversion? | Firth logistic regression (n = 20), likelihood ratio = 0.05 |
cohens_d_ci() pairs Cohen’s d with a bootstrap interval,
which is informative precisely where it matters most: interval width
grows visibly as n falls, making the extra uncertainty at small n
explicit rather than implicit in a single point estimate.
The small-sample methods demonstrated in this vignette are described in full, with guidance on when to prefer each one, in:
Sharafuddin, M. A., Jaleel, A. A., and Madhavan, M. (2026). Quantitative Analysis with Small Samples: A Practical Guide for Students and Early-Career Researchers (Version 0.1.0) [Book]. Zenodo. https://doi.org/10.5281/zenodo.20221929
A companion preprint validating these method choices by simulation is in preparation and will be added here once posted.