Package {whatifbandit}

Title:	Analyzing Randomized Experiments Using Multi-Arm Bandits
Version:	1.0.2
Description:	Simulates response-adaptive experimental trials using Multi-Arm Bandits. Adaptive robust estimators defined in Hadad et al. (2021) <doi:10.1073/pnas.2014602118> and Offer-Westort et al. (2021) <doi:10.1111/ajps.12597> are used to robustly estimate conditional expectations and treatment effects. Provides significant simulation customization options for imperfect information, non-stationary bandits, and increased exploration strategies for assignments.
License:	GPL (≥ 3)
URL:	https://github.com/Noch05/whatifbandit
BugReports:	https://github.com/Noch05/whatifbandit/issues
Depends:	R (≥ 4.1.0)
Imports:	bandit, clubSandwich, data.table (≥ 1.18.0), dplyr, estimatr, furrr, lubridate, methods, purrr, randomizr, rlang, tibble, tidyr
Suggests:	future, ggplot2, knitr, rmarkdown, testthat (≥ 3.0.0)
VignetteBuilder:	knitr
Config/testthat/edition:	3
Config/testthat/parallel:	true
Encoding:	UTF-8
LazyData:	true
RoxygenNote:	8.0.0
NeedsCompilation:	no
Packaged:	2026-07-22 23:17:28 UTC; noah
Author:	Noah Ochital [aut, cre, cph], Ryan T. Moore [ctb, cph]
Maintainer:	Noah Ochital <no9857a@american.edu>
Repository:	CRAN
Date/Publication:	2026-07-23 10:50:02 UTC

whatifbandit: Analyzing Randomized Experiments Using Multi-Arm Bandits

Description

Simulates response-adaptive experimental trials using Multi-Arm Bandits. Adaptive robust estimators defined in Hadad et al. (2021) doi:10.1073/pnas.2014602118 and Offer-Westort et al. (2021) doi:10.1111/ajps.12597 are used to robustly estimate conditional expectations and treatment effects. Provides significant simulation customization options for imperfect information, non-stationary bandits, and increased exploration strategies for assignments.

Author(s)

Maintainer: Noah Ochital no9857a@american.edu (ORCID) [copyright holder]

Authors:

• Noah Ochital no9857a@american.edu (ORCID) [copyright holder]

Other contributors:

• Ryan T. Moore rtm@american.edu (ORCID) [contributor, copyright holder]

See Also

Useful links:

• https://github.com/Noch05/whatifbandit

• Report bugs at https://github.com/Noch05/whatifbandit/issues

Adaptively Assign Treatments in a Period

Description

Assigns new treatments for an assignment wave based on the assignment probabilities provided from compute_bandit(). Assignments are made randomly with the given probabilities using randomizr::block_ra(), randomizr::complete_ra(), randomizr::cluster_ra(), or randomizr::block_and_cluster_ra() depending on whether blocking and/or clustering are used.

Usage

assign_treatments(
  current_data,
  probs,
  blocking = NULL,
  clustering = NULL,
  conditions,
  condition_col = NULL,
  cluster_col = NULL,
  sim_type
)

build_ra_args(
  current_data,
  probs = NULL,
  conditions,
  blocking,
  clustering,
  cluster_col = NULL,
  dt
)

compute_assignments(
  current_data,
  probs,
  conditions,
  blocking,
  clustering,
  cluster_col = NULL
)

## S3 method for class 'data.frame'
assign_treatments(
  current_data,
  probs,
  blocking,
  clustering,
  conditions,
  condition_col = NULL,
  cluster_col = NULL,
  sim_type
)

## S3 method for class 'data.table'
assign_treatments(
  current_data,
  probs,
  blocking,
  clustering,
  conditions,
  condition_col = NULL,
  cluster_col = NULL,
  sim_type
)

Arguments

Value

Updated tibble or data.table with the new treatment conditions for each observation, and whether imputation is required. If this treatment is different from the original experiment, impute_req = 1, otherwise it is 0.

A list with fn (the randomizr function) and args (its arguments).

Character vector of length nrow(current_data) with treatment assignments.

Functions

• build_ra_args(): Selects the appropriate {randomizr} function and constructs its argument list based on whether blocking and/or clustering are requested.

• compute_assignments(): Pre-allocates a character vector and fills treatment assignments for bandit and randomly assigned subsets separately, using the appropriate randomizr function built by build_ra_args().

See Also

• randomizr::block_ra()

• randomizr::complete_ra()

• randomizr::cluster_ra()

• randomizr::block_and_cluster_ra()

Other assign: compute_bandit(), compute_prior()

Recover Block-Specific Success and Total Counts for Bootstrap Null

Description

Recovers the number of successes and total observations within each group for use in constructing group-specific Beta posteriors for the parametric bootstrap joint test.

Usage

boot_null_counts(data, success_col, group = NULL)

## S3 method for class 'data.frame'
boot_null_counts(data, success_col, group = NULL)

## S3 method for class 'data.table'
boot_null_counts(data, success_col, group = NULL)

Arguments

Value

An aggregated data.frame or data.table with the appropriate counts.

Build Proper Arguments for RCT Bootstrap Joint Test

Description

Build Proper Arguments for RCT Bootstrap Joint Test

Usage

build_rct_clusters(mab, col_names)

build_p_cols(args)

boot_build_p(data, success_col, group_col, cols)

build_time_model_args(mab, args, col_names)

Arguments

Value

A named numeric vector, named list of vectors, or NULL.

A named numeric vector of proportions, or a scalar 1.

A list with elements cols, s (successes), and n (totals), all named and sorted consistently.

A list with elements time_model (function or NULL) and args (list of additional arguments for the time model).

Functions

• build_rct_clusters(): Resolves the clusters argument for a single_rct_mab bootstrap. When both blocks and clusters are present, clusters is a named list of per-block cluster proportion vectors (as documented in simulate_mab()). When only clusters are present, returns a flat named proportion vector via group_prop(). Returns NULL when no clustering was used.

• build_p_cols(): Returns the named vector of proportions that drives the columns of the null p matrix — clusters if present, blocks if present, or a scalar 1 for the no-blocking/no-clustering case. For the blocked-and-clustered case, clusters is a named list; this flattens it to a single named vector (since the p matrix columns are individual clusters, not blocks).

• boot_build_p(): Recovers successes and totals for each column group (cluster, block, or the whole dataset), then returns them alongside the resolved column proportions vector for use in joint_boot_null().

• build_time_model_args(): Constructs the time_model function and its associated argument list when delayed_feedback is enabled for a single_rct_mab bootstrap. Returns a list with time_model = NULL and args = list() when delayed feedback is not in use.

Checking Clusters Do Not Persist Across Periods

Description

Checks to ensure that each cluster only exists within a single simulation period, because if this is the case a true clustered design is no longer specified. See details.

Usage

check_clusters(data, cluster_col)

## S3 method for class 'data.frame'
check_clusters(data, cluster_col)

## S3 method for class 'data.table'
check_clusters(data, cluster_col)

cluster_throw(check_clusters, cluster_col)

Arguments

Details

The assignment algorithm in mab_loop() assumes that clusters do not persist across periods. For a true clustered design, if a cluster persisted across periods, all observations within it would have to be assigned to the same treatment as in the previous period. In an adaptive experiment this results in no adaptation, thus this is not implemented into the algorithm. Instead, the assumption is verified here.

Value

Nothing; Throws a warning if any clusters persist across multiple periods.

Nothing; Throws a warning if clusters persist across periods.

Functions

• cluster_throw(): Internal Helper which handles all data wrangling agnostic portions of check_clusters()

Argument Check Helper Functions

Description

This set of functions is common across the main argument checkers, and they each check a clear condition on a set of arguments, such as ensuring the proper data type.

Usage

check_logical(...)

check_prop(...)

check_posint(...)

check_sum1(...)

check_string(arg, valid, name)

check_names(...)

Arguments

Value

Throws an error if any input is not TRUE or FALSE

Throws an error if any input is not a valid proportion between 0 and 1.

Throws an error if any input is not a valid positive integer.

Nothing; Throws an error if a numeric vector does not sum to 1.

Nothing; Throws an error if the string argument is invalid.

Nothing; Throws an if an argument does not have names attribute.

Functions

• check_logical(): Checks for valid logical arguments

• check_prop(): This function accepts the user's settings for proportion arguments and checks if they are valid proportions between 0 and 1.

• check_posint(): This function accepts the user's settings for positive integer arguments and checks if they are valid positive integers.

• check_sum1(): Checks if specified numeric vectors each sum to 1.

• check_string(): Checks specific string arguments against provided valid arguments.

• check_names(): Checks if provided objects have names attribute.

See Also

Other checks: check_rct_args()

Perform Validation Checks for simulate_mab()

Description

This function checks to ensure that all required arguments have been properly passed to simulate_mab() before continuing with the simulation. When errors are thrown, user-friendly messages are provided to indicate which argument was misspecified. Additionally, when verbose = TRUE, additional warning messages may be shown if unnecessary arguments are passed.

Usage

check_mab_sim(
  n,
  t,
  p,
  blocks = NULL,
  clusters = NULL,
  control_augment,
  random_assign_prop,
  assignment_dates,
  delayed_feedback,
  time_model = NULL,
  period_sizes = NULL,
  prior_periods = NULL,
  discount_rate,
  dt,
  ndraws = 5000,
  r,
  keep_data,
  keep_models,
  verbose
)

check_p_colnames(p, expected)

Arguments

Value

Nothing; Throws an error if all checks are not met.

Nothing; Throws an error if colnames(p) doesn't match provided labels.

Functions

• check_p_colnames(): Checks if colnames(p) matches provided labels

Validates Inputs For mab_from_rct()

Description

This function checks to ensure that all required arguments have been properly passed to mab_from_rct() before continuing with the simulation. When errors are thrown, user-friendly messages are provided to indicate which argument was misspecified. Additionally, when verbose = TRUE, additional warning messages may be shown if unnecessary arguments are passed.

Usage

check_rct_args(
  data,
  control_augment,
  random_assign_prop,
  period_method,
  time_unit,
  period_length,
  prior_periods,
  discount_rate,
  delayed_feedback,
  whole_experiment,
  col_names,
  verbose,
  ndraws,
  r,
  keep_data,
  keep_models,
  blocking
)

check_cols(
  period_method,
  time_unit,
  delayed_feedback,
  col_names,
  data,
  verbose,
  blocking
)

check_data(
  data,
  col_names,
  period_method,
  period_length,
  time_unit,
  delayed_feedback
)

check_period_method(period_method, time_unit, verbose, period_length)

Arguments

Value

Throws an error if an argument is missing or misspecified.

Throws an error if columns which are required have not been declared or are not present in the data, or are the wrong primitive data type. Additionally throws warning messages, if unnecessary columns have been provided, only when verbose = TRUE.

Nothing; Throws an error if the provided dataset does not meet the specifications of the trial based on user settings.

Throws an error if the user is missing necessary arguments to assign treatments or passes invalid ones.

Functions

• check_cols(): Helper to check_rct_args(). This function accepts the user's settings for the Multi-Arm-Bandit trial, and checks whether columns in the data have been properly specified based on these settings.

• check_data(): Throws an error if the provided dataset does not meet the specifications of the trial based on user settings.

• check_period_method(): Helper to check_rct_args(). This function accepts arguments relating to how treatment waves are assigned, and checks if they are valid, and if all supporting arguments are passed as necessary.

See Also

Other checks: check_helpers

Check Conflicts with Internal Columns

Description

Identifies column name conflicts between internal reserved columns and input data for mab_from_rct()'s internal procedures

Usage

col_conflict_check(data)

reserved_cols()

Value

throws an error under any conflict

a named character vector of reserved column names.

See Also

mab_from_rct()

Ends Multi-Arm Bandit Trial

Description

Condenses output from mab_loop() into manageable structure. Contains methods for data.frame and data.table.

Usage

collect_mab_results(
  data,
  bandits,
  algorithm,
  periods,
  conditions,
  num_conditions,
  ndraws
)

## S3 method for class 'data.frame'
collect_mab_results(
  data,
  bandits,
  algorithm,
  periods,
  conditions,
  num_conditions,
  ndraws
)

## S3 method for class 'data.table'
collect_mab_results(
  data,
  bandits,
  algorithm,
  periods,
  conditions,
  num_conditions,
  ndraws
)

Arguments

Value

A named list containing:

• final_data: The processed tibble or data.table, with the trial's results.

• bandits: A tibble or data.table containing the UCB1 or Thompson Sampling values for each period.

• assignment_probs: A tibble or data.table containing the probability of being assigned each treatment arm at a given period.

• assignment_quantities: A numeric vector of the total number of observations assigned to each treatment arm.

See Also

Other simulation: mab_loop(), run_mab()

Calculate Multi-Arm Bandit Decision Based on Algorithm

Description

Calculates the best treatment for a given period using either a UCB1 or Thompson Sampling algorithm. Thompson sampling is done using bandit::best_binomial_bandit() from the bandit package and UCB1 values are calculated using the well-defined formula that can be found in Auer et al. (2002).

Usage

compute_bandit(
  past_results = NULL,
  algorithm,
  num_conditions,
  conditions,
  current_period,
  control_augment = 0,
  random_assign_prop = 0,
  ndraws
)

compute_bandit.thompson(past_results, conditions, current_period, ndraws)

bandit_invalid(bandit)

compute_bandit.ucb1(past_results, conditions, current_period = NULL)

Arguments

Details

Control augmentation adjustment is always done last, to ensure proper probability floor.

The Thompson assignment_probabilities are the same as the bandit vector except when control_augment or random_assign_prop are greater than 0, as these arguments will alter the probabilities of assignment.

Thompson sampling is calculated using the bandit package but the direct calculation can result in errors or overflow. If this occurs, a simulation based method from the same package is used instead to estimate the posterior distribution. If this occurs a warning will be presented. ndraws specifies the number of iterations for the simulation based method, and the default value is 5000.

The UCB1 algorithm only selects 1 treatment at each period, with no probability matching so assignment_probabilities will always have 1 element equal to 1, and the rest equal to 0, unless control_augment or random_assign_prop are greater than 0, which will alter the probabilities of assignment. For example, if the original vector is ⁠(0, 0, 1)⁠, and control_augment = 0.2, the new vector is ⁠(0.2, 0, 0.8)⁠ assuming the first element is control. If instead the 3rd element were the control group the resulting vector would not be changed because it already meets the control group threshold. Under ties, the assignment probability is split evenly among tied arms, so ⁠(0, 1, 1)⁠ would become ⁠(0, 0.5, 0.5)⁠.

Value

A list of length 2 containing:

• bandit: Bandit object, either a named numeric vector of Thompson sampling probabilities or UCB1 values.

• assignment_probabilities: Named numeric vector with probabilities of being assigned to the given treatment, where names(.) are the treatments. Adjusted for control augmentation and random assignment proportion weighting

Logical; TRUE if the vector is invalid, FALSE if valid

Functions

• bandit_invalid(): Checks if the Thompson Sampling probabilities either sum arbitrarily close to 0 or if any of them are NA, indicating the direct calculation failed or did not converge.

References

Auer, Peter, Nicolò Cesa-Bianchi, and Paul Fischer. 2002. "Finite-Time Analysis of the Multiarmed Bandit Problem." Machine Learning 47 (2): 235–56. doi:10.1023/A:1013689704352.

Kuleshov, Volodymyr, and Doina Precup. 2014. "Algorithms for Multi-Armed Bandit Problems." arXiv. doi:10.48550/arXiv.1402.6028.

Loecher, Thomas Lotze and Markus. 2022. "Bandit: Functions for Simple a/B Split Test and Multi-Armed Bandit Analysis." https://cran.r-project.org/package=bandit.

Thompson, William R. 1933. "On the Likelihood That One Unknown Probability Exceeds Another in View of the Evidence of Two Samples." Biometrika 25 (3/4): 285–94. doi:10.2307/2332286

See Also

Other assign: assign_treatments(), compute_prior()

Calculate Observation Level AW-AIPW For Each Treatment Condition

Description

Calculates the augmented inverse probability weighted estimate (AW-AIPW) of treatment success for each observation and treatment (i.e. on the level of a single unit).

Usage

compute_iaipw(data, assignment_probs, periods, conditions)

## S3 method for class 'data.frame'
compute_iaipw(data, assignment_probs, periods, conditions)

## S3 method for class 'data.table'
compute_iaipw(data, assignment_probs, periods, conditions)

iaipw(conditions_vec, success_vec, mhat, prob, condition)

Arguments

Details

The specification for the individual AW-AIPW estimates can be found in Hadad et al. (2021). The formulas in equation 5, formed the basis for this function's calculations. Here the regression adjustment used is the grouped mean of success by treatment, up until the current period of estimation (so at period 5, the grouped mean would be calculated using the results from periods 1 through 4).

Value

A named list with the individual aipw estimate vectors for each treatment condition.

Functions

• iaipw(): Small function to handle iaipw computation in both branches.

References

Hadad, Vitor, David A. Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. 2021. "Confidence Intervals for Policy Evaluation in Adaptive Experiments." Proceedings of the National Academy of Sciences of the United States of America 118 (15): e2014602118. doi:10.1073/pnas.2014602118.

See Also

Other estimation: estimate_aw_aipw(), estimate_lm(), inference_helpers, lin_contrast

Compute exact period sizes

Description

Compute exact period sizes

Usage

compute_period_sizes(data)

## S3 method for class 'data.frame'
compute_period_sizes(data)

## S3 method for class 'data.table'
compute_period_sizes(data)

Arguments

Value

Numeric vector of length(max(period_nummber)) with each element representing the number of units in each period.

Gather Past Results for Given Assignment Period

Description

Summarizes results of prior periods to update assignment probabilities in the current period. This function calculates the number of successes under each treatment and the total number of observations assigned to each treatment which are used to calculate UCB1 values or Thompson sampling probabilities. These values are weighted by the discount_rate provided.

Usage

compute_prior(
  current_data,
  prior_data,
  delayed_feedback,
  assignment_date_col = NULL,
  discount_rate,
  conditions,
  current_period
)

## S3 method for class 'fast'
compute_prior(
  current_data,
  prior_data,
  delayed_feedback,
  assignment_date_col = NULL,
  discount_rate,
  conditions,
  current_period
)

## S3 method for class 'data.frame'
compute_prior(
  current_data,
  prior_data,
  delayed_feedback,
  assignment_date_col = NULL,
  discount_rate,
  conditions,
  current_period
)

## S3 method for class 'data.table'
compute_prior(
  current_data,
  prior_data,
  delayed_feedback,
  assignment_date_col = NULL,
  discount_rate,
  conditions,
  current_period
)

finalize_prior_list(prior_list, conditions)

Arguments

Details

When delayed_feedback = TRUE, the maximum value from the specified assignment_date_col in the current data is taken as the last possible date the researchers conducting the experiment could have learned about a treatment outcome. All successes that occur past this date are masked and are treated as failures for the purposes this period's assignments.

Value

A list containing 2 named vectors: the weighted number of successes, and the weighted number of assignments, where the names of each vector correspond to the treatment condition.

A named list with elements successes, n, each a named numeric vector of length length(conditions).

Functions

• finalize_prior_list(): Accepts the raw list output of an aggregation over prior_data (from compute_prior()), names each vector by condition, fills any conditions absent from the prior window with zeros, and sorts alphabetically.

See Also

Other assign: assign_treatments(), compute_bandit()

Condenses results of repeated simulations.

Description

Condenses results of repeated simulations.

Usage

condense_results(dt, keep_data, keep_models, mabs)

Arguments

Details

This function iterates over every element in mabs and extracts the required element to place in a condensed list for the final output.

Value

A named list containing:

• final_data: tibble or data.table containing the nested tibbles/data.tables from each trial. Only provided when keep_data = TRUE.

• bandits: A tibble or data.table containing the UCB1 values or Thompson Sampling posterior distributions for each period and trial. Wide format, each row is a period, and each column is a treatment.

• assignment_probs: A tibble or data.table containing the probability of being assigned each treatment arm at a given period and trial. Wide format, each row is a period, and each column is a treatment.

• assignment_quantities: A tibble or data.table containing total number of observations assigned to each treatment arm in each trial. each treatment arm in each trial.

• contrasts: A tibble or data.table containing the contrast estimates of the specified estimators for the specified contrast structure for each trial.

• f_stats: Named tibble or data.table containing f_stat from IPW and OLS regressions for each trial.

• models: A nested list containing the lm_robust objects from regressions, only saved in clustered case, and when keep_models = TRUE

Constructs mab and its other class variants

Description

Simple constructor for proper mab subclasses as output to simulate_mab() and mab_from_rct().

Usage

construct_mab(mab, type, multi)

Arguments

Value

Input mab with appropriate S3 class, restructured for output

Create Treatment Wave Cutoffs

Description

Used to assign each observation a new treatment assignment period, based on user-supplied specifications, and user supplied data from date_col and month_col in col_names, and the period_length. Creates a new column indicating with period each observation belongs to.

Usage

create_cutoff(data, col_names, period_length = NULL, period_method, time_unit)

create_cutoff.date(data, time_unit, date_col, month_col, period_length)

create_cutoff.individual(data)

create_cutoff.batch(data, period_length)

Arguments

Details

The assignment periods do not strictly have to line up with the original experiment, it is up to the researcher to test the possible options.

Month based assignment can be specified either using the months inside the month_col or date_col, if month_col is passed into the function it will be used.

Value

Updated tibble/data.table with the new period_number column. period_number is an integer representing an observation's new assignment period.

Create Necessary Columns for Multi-Arm Bandit Trial

Description

Initializes partially empty columns in data to initialize them for the simulation. These are initialized as NA except for observations with period_number = 1, whose values are copied from the provided columns, and used as the starting point for the simulation.

Usage

create_new_cols(data, col_names, blocking, delayed_feedback, vars_keep)

## S3 method for class 'data.frame'
create_new_cols(data, col_names, blocking, delayed_feedback, vars_keep)

## S3 method for class 'data.table'
create_new_cols(data, col_names, blocking, delayed_feedback, vars_keep)

Arguments

Value

A data.frame/data.table subsetted to all the user provided columns plus these 6 new columns:

• mab_success: New variable to hold new success from Multi-arm bandit procedure, NA until assigned.

• mab_condition: New variable to hold new treatment condition from Multi-arm bandit procedure, NA until assigned.

• impute_req: Binary indicator for imputation requirement, NA until assigned.

• new_success_date: New variable to hold the new success date under Multi-arm bandit procedure, NA until assigned.

• block: New variable indicating the variables to block by for assignment.

• treatment_block: New variable combining block with original treatment condition.

Calculate Adaptive AW-AIPW Estimates

Description

Uses provided Individual AIPW scores created by compute_iaipw() and computes the final Adaptively Weighted AIPW estimate and variance using the formulas from Hadad et. al (2021). Uses the constant allocation rate adaptive weight.

Usage

estimate_aw_aipw(
  data,
  assignment_probs,
  conditions,
  iaipw,
  cluster_col,
  clustering,
  periods,
  num_clusters = NULL
)

Arguments

Details

The formulas for the calculations in this function can be found in Hadad et al. (2021) at equation 5 (estimate), equation 11 (variance), equation 15 (allocation rate).

The estimator assumes pure sequential assignment, but we adapt the estimator to our batched assignment procedure. In the computations, of the individual estimates, regression estimates were only computed for each period, instead of for each observation, and similarly the adaptive weights used will only be computed per period, and then simply assigned to all the observations in that period, thus resulting in only a few unique weights. This keeps effective sample size large, ensuring the asymptotic properties are realized in large samples with only a few assignment periods, while also properly accounting for the assignment procedure.

If clustering is specified, within each period individual AIPW estimates are aggregated by cluster, and then the sample size becomes the sum of the number of clusters in each period, the variance formula uses the cluster deviations instead (an effective CR0 style estimator), which is adjusted via the Stata CR1 estimator (\frac{G}{G-1} * \frac{N-1}{N-k}) where k is the number of treatments, and G is the number of clusters.

The AW-AIPW estimator is unbiased, consistent, and asymptotically normal when a sufficently large non-zero probability of assignment is guaranteed for each treatment condition in each period. The provided standard errors of the so can be used for valid inference with a normal distribution. Treatment effects can also be estimated as the difference in AW-AIPW estimates with the variance of the difference as the sum of the variances of the two arms. In the clustered case, we suggest a t distribution to be more conservative, given the sample size is now the cluster, we provide G-1 as degrees of freedom, as standard by regression packages for clustered inference (CR0, and CR1).

Value

A tibble/data.table containing 5 columns:

• means: AW-AIPW statistics.

• se: AW-AIPW standard errors.

• mab_condition: Treatment arm.

• estimator: "AW-AIPW".

• df: Degrees of freedom, NA implies normal distribution Z test.

References

Hadad, Vitor, David A. Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. 2021. "Confidence Intervals for Policy Evaluation in Adaptive Experiments." Proceedings of the National Academy of Sciences of the United States of America 118 (15): e2014602118. doi:10.1073/pnas.2014602118.

See Also

Other estimation: compute_iaipw(), estimate_lm(), inference_helpers, lin_contrast

OLS Estimates for Probability of Success

Description

Computes OLS estimates for true true probabilities of success using estimatr::lm_robust(). Supports IPW weighted and unweighted regression. If clustering is specified CR2 standard errors are reported. Otherwise HC2 standard errors are used. Appropriate degrees of freedom are supplied along with the regression's F-statistic

Usage

estimate_lm(
  data,
  cluster_col,
  clustering,
  conditions,
  num_clusters = NULL,
  ipw
)

Arguments

Details

If CR2 standard errors fail to be calculated, CR1S are computed, equivalent to vce(cluster, clustervar) in Stata, and se_type = "stata" in {estimatr}

These estimates follow the procedure in Offer-Westort et al. (2021). Degrees of freedom are not provided for the f-statistic, because the traditional F-distribution is invalid under the adaptive procedure. However, this f-statistic can be used for the randomization and bootstrap inference joint-tests provided.

The provided standard errors can be used to construct approximate confidence intervals using a t-distribution with the provided degrees of freedom. However the degrees of freedom provided are n - num_conditions, which is likely to be an overestimate given the potential for the number of observations assigned to each group to vary widely with the adaptive trial. The HC2 or CR2 corrections help but do not capture the temporal dependence created by adaptive assignment. Thus formal pairwise tests cannot be conducted using these estimates, because the estimator does not follow a precise t-distribution.

As mentioned in Offer-Westort et al. (2021), if there is a true best arm, and control augmentation is used, t-test can provide proper coverage in large samples because the algorithm converges to a two-arm comparison.

Block fixed effects are not used for estimation due to the prevalence of numerical instability in the estimates. Assignment probabilities to treatment are the same within each block, so the IPW estimator is still unbiased without the presence of the fixed effects.

Value

A list containing 3 elements:

• estimates: data.table/data.frame of coefficients, standard errors, and degrees of freedom.

• f_stat: F statistic from regression

• model: lm_robust object, only saved under clustering.

References

Offer‐Westort, Molly, Alexander Coppock, and Donald P. Green. 2021. "Adaptive Experimental Design: Prospects and Applications in Political Science." American Journal of Political Science 65 (4): 826–44. doi:10.1111/ajps.12597.

See Also

Other estimation: compute_iaipw(), estimate_aw_aipw(), inference_helpers, lin_contrast

Fit a Robust Linear Probability Model and Compute Contrasts

Description

Internal helper that wraps linear model estimation for both IPW and OLS estimators, returning arm-level mean estimates, precomputed contrasts, and optionally the fitted model object.

Usage

estimate_lm_bundle(
  ipw,
  estimator,
  sim_results,
  col_names,
  clustering,
  conditions,
  num_clusters,
  dt,
  contrasts_list
)

Arguments

Value

A named list with three elements: means, contrasts, model. contrasts are the computed contrasts, the other 2 elements are the output of estimate_lm()

Extract Success Probabilities Per-Unit

Description

Looks up the success probability for each unit given their treatment assignment and, optionally, their block and/or cluster membership. Handles all supported p structures.

Usage

extract_success_prob(p, conditions, other_idx = NULL)

Arguments

Value

A numeric vector of length n containing the per-unit success probabilities to be used for outcome observation.

Helpers for Joint F Test

Description

Takes the single_mab object provided and returns the proper null distribution for the randomization or the bootstrap joint test.

Usage

joint_random_null(mab, r)

joint_boot_null(mab, r)

joint_base_args(mab, sim_type)

joint_null_inner(args)

Arguments

Value

a numeric vector of simulated F-statistics

A named list of arguments to be used for run_mab_single()

The F-statistic from the IPW regression of the MAB Trial

Functions

• joint_random_null(): Prepares arguments for the randomization joint test.

• joint_boot_null(): Prepares arguments for the parametric bootstrap joint test.

• joint_base_args(): Extracts common arguments from run_mab_single() and the single_mab provided object's arguments slot.

• joint_null_inner(): inner function for furrr::future_map()

Formula Parser

Description

Parses the input formula for mab_from_rct()

Usage

formula_parse(formula)

gather_args(x)

.block(...)

.cluster(c)

Arguments

Value

List of columns specified from formula.

A list containing the function call, and the arguments so "block(x1)" gets turned into a list with elements ⁠block, "x1"⁠.

Vector of blocking variables

cluster variable

Functions

• gather_args(): Helper for formula parsing. Parses the expression, and splits the function call from the arguments.

• .block(): Block

• .cluster(): Cluster

Generate Assignment Dates

Description

Generates a length(n) vector of assignment dates based on provided information.

Usage

generate_assignment_dates(n, assignment_dates)

Arguments

Value

vector of assignment dates

See Also

Other param: generate_groups(), generate_outcomes(), generate_period_idx(), prep_sim_data(), setup_mab_sim(), split_args()

Generate Block and Cluster Memberships

Description

Takes a named probability vector for blocks and clusters and uses randomizr::complete_ra() to randomly assign each of n units to a block and cluster according to those probabilities.

Usage

generate_groups(n, blocks = NULL, clusters = NULL)

Arguments

Value

A list containing the factor vectors of group assignments for blocks, clusters, both or NULL depending on what was specified. Levels for each vector come from the labels for each block and cluster.

See Also

Other param: generate_assignment_dates(), generate_outcomes(), generate_period_idx(), prep_sim_data(), setup_mab_sim(), split_args()

Generate Outcomes Per-Unit

Description

Uses provided success probabilities to draw a Bernoulli outcome for each unit. If time_model is provided, it is also used to compute dates of success

Usage

generate_outcomes(
  current_data,
  current_period,
  data,
  p,
  idx,
  simulate_dates,
  time_model = NULL,
  time_model_args = NULL
)

Arguments

Value

Updated data object containing all the outcomes generated in the period, such as the treatment assignments, treatment outcomes, and new success dates

See Also

Other param: generate_assignment_dates(), generate_groups(), generate_period_idx(), prep_sim_data(), setup_mab_sim(), split_args()

Generate Start and End Indexes

Description

Generates the start and end indexes for each period based on provided information

Usage

generate_period_idx(n, t, period_sizes = NULL)

Arguments

Details

When not provided period sizes are calculated as n %/% t for all periods, with the last as n %/% t + n %% t.

Value

list of numeric vectors featuring start and end indexes for each period of the simulation

See Also

Other param: generate_assignment_dates(), generate_groups(), generate_outcomes(), prep_sim_data(), setup_mab_sim(), split_args()

Get Group Proportions

Description

Accepts input data, and a group column, and returns the proportion of the data that belongs to each group

Usage

group_prop(data, group)

## S3 method for class 'data.frame'
group_prop(data, group)

## S3 method for class 'data.table'
group_prop(data, group)

Arguments

Value

A named numeric vector with the names corresponding to the group, and the value to its proportion among the provided data.

Imputing New Outcomes of Multi-Arm-Bandit Trial

Description

Imputes outcomes for the current treatment assignment period. Uses randomizr::block_ra() to impute the outcomes for observations who were assigned new treatments. The probabilities used to guide the imputation of the outcomes are pre-computed using the existing data from the original randomized experiment.

Usage

impute_outcomes(
  data,
  imputation_info,
  success_col,
  success_date_col = NULL,
  delayed_feedback,
  idx
)

compute_impute(imputation_info, success_col)

## S3 method for class 'data.frame'
impute_outcomes(
  data,
  imputation_info,
  success_col,
  success_date_col,
  delayed_feedback,
  idx
)

## S3 method for class 'data.table'
impute_outcomes(
  data,
  imputation_info,
  success_col,
  success_date_col,
  delayed_feedback,
  idx
)

Arguments

Details

When delayed_feedback = TRUE, dates of success are imputed according to the average by each period and treatment block (treatment arm + any blocking). These imputations are required because these observations do not currently have dates of success, as no success was observed during the original experiment. Therefore if they go through the next iteration of the simulation without being imputed, the new successes will still be treated as failures because of the date masking mechanism.

Observations that were successful in the original experiment, got assigned a new treatment, and then imputed as success again, will have their original date kept. This assumes that the treatment has no individual treatment effect on the date of success, which may or may not be valid depending on the context of the experiment.

Value

Updated data object with all the updates from the current period of the simulation

Numeric vector of imputed outcomes

Functions

• compute_impute(): Performs all data.table/data.frame agnostic portions of impute_success

See Also

Other imputation: precompute_imputation(), prep_imputation()

Helper Functions for Inference

Description

Internal helpers for estimation in run_mab().

Accepts a data.frame like object, and a character of vector of conditions. It checks whether or not all provided conditions are present in the data, if not their values are initialized to NA

Combines the AW-AIPW, IPW, and Sample estimates into a single object to be returned.

Performs adjustment of CR0 SE to CR1 SE, using Stata's formula

Usage

strip_condition_prefix(x)

fill_missing_conditions(x, conditions, estimator)

## S3 method for class 'data.frame'
fill_missing_conditions(x, conditions, estimator)

## S3 method for class 'data.table'
fill_missing_conditions(x, conditions, estimator)

combine_estimates(...)

cr1(x, g, n, k)

Arguments

Value

x with the prefix removed.

An updated estimates object with missing conditions initialized.

A list of 2 elements:

• estimates: Input estimates bound together by rows.

• vcov: vcov input

An adjusted vector of variances

Functions

• strip_condition_prefix(): Strips coefficient name prefixes from named vector or matrix's dimnames.

See Also

Other estimation: compute_iaipw(), estimate_aw_aipw(), estimate_lm(), lin_contrast

Joint Hypothesis Test for Multi-Arm Bandit Trials

Description

Conducts a joint hypothesis test of no treatment effects across all arms, i.e. that all arms have the same true probability of success, either using a bootstrap procedure or the randomization inference procedure adapted from Offer-Westort et al. (2021). See details for a description of both methods

Usage

joint_test(mab, method = c("bootstrap", "randomization"), r = 100)

Arguments

Value

A named list object containing

• f_statistic: The observed F-statistic from the IPW regression.

• null_distribution: A numeric vector of F-statistics under the null.

• p_value: The proportion of simulated F-statistics more extreme than observed.

• method: The method used.

• r: Number of replications used.

• effective_r: Number of generated f_stats which are not NULL or NA.

NOTE

This procedure is experimental and has no Type I error guarantee. Offer-Westort et. al (2021) also note the test suffers from low power, but it is provided for experimentation nonetheless.

method = "randomization" operates under the sharp null that each unit would express the same outcome no matter the treatment they were assigned. To achieve this the trial is re-simulated but new outcomes are not generated or imputed, however the adaptive algorithm still changes the assignments. This results in a null distribution that captures how the adaptive algorithm will assign even when the outcomes are not related to treatments at all.

method = "bootstrap" operates under the null hypothesis that there is no difference between treatment arms within each block/cluster. If there are no blocks or clusters, a p-matrix is built from the pooled sample mean of the original trial. With block and or cluster pooled sample means are estimated within each block or cluster. The block and or cluster assignment proportions are taken from the original dataset.

References

Offer-Westort, Molly, Alexander Coppock, and Donald P. Green. "Adaptive Experimental Design: Prospects and Applications in Political Science." American Journal of Political Science 65, no. 4 (2021): 826–44. doi:10.1111/ajps.12597.

Examples

data(tanf)
set.seed(5)
adaptive <- mab_from_rct(success ~ condition, data = tanf, algorithm = "thompson",
period_method = "batch", period_length = 500)

# Low `r` for examples, use replications in practice
joint_test(adaptive, "randomization", r = 2)
joint_test(adaptive, "bootstrap", r = 2)

Computing Linear Contrasts

Description

A set of helper functions for computing linear contrasts of AW-AIPW, IPW, and OLS estimates

Usage

build_contrast_matrices(conditions, contrasts = NULL, bandits)

make_contrasts(conditions, ref_idx, type)

compute_contrast(
  C = NULL,
  coefs = NULL,
  vcov = NULL,
  df = NULL,
  model = NULL,
  data = NULL,
  dt,
  conditions = NULL,
  estimator
)

Arguments

Value

A list of 1 row contrast vectors, encoding a single pairwise comparison.

A matrix row vectors containing each contrast to test.

data.table or data.frame of linear contrasts with columns for each treatment arm, the estimated contrast, standard error, and degrees of freedom.

Functions

• build_contrast_matrices(): Constructs a matrix of contrast vectors for testing pairwise comparisons between treatment arms.

• make_contrasts(): Creates contrast vectors for linear hypothesis test

• compute_contrast(): Compute linear contrasts given a list of contrast vectors and model inputs. Dispatches to the appropriate method based on the estimator type and whether clustering is present.

See Also

Other estimation: compute_iaipw(), estimate_aw_aipw(), estimate_lm(), inference_helpers

Simulate a Multi-Arm-Bandit Trial from an Existing Randomized Controlled Trial, With Bernoulli Distributed Outcomes.

Description

Simulates a response-adaptive, Multi-Arm-Bandit (MAB) trial using experimental data from an original randomized controlled trial (RCT), and adaptive inference strategies described in Hadad et al. (2021) and Offer-Westort et al. (2021) to robustly estimate treatment effects. See the details to learn more.

Usage

mab_from_rct(
  formula,
  data,
  algorithm = c("thompson", "ucb1", "static"),
  random_assign_prop = 0,
  control_augment = 0,
  control_condition = NULL,
  period_method = c("batch", "date", "individual"),
  time_unit = NULL,
  period_length,
  prior_periods = NULL,
  discount_rate = 1,
  date_col = NULL,
  month_col = NULL,
  delayed_feedback = FALSE,
  assignment_date_col = NULL,
  success_date_col = NULL,
  whole_experiment = FALSE,
  ndraws = 5000,
  r = 1,
  verbose = FALSE,
  check_args = TRUE,
  keep_data = FALSE,
  keep_models = FALSE,
  contrasts = NULL,
  ...
)

Arguments

Details

Clustering

Clusters should be contained inside each assignment wave (a warning is thrown if this is not the case), so it is possible to have 2 observations in the same cluster assigned to different treatments if they were assigned in different waves. This is assumed because without it the adaptive probabilities will not be impacting assignments. For example if someone in cluster 1 is assigned in period 1, then all other members are forced to have the same treatment, even if they are assigned in period 5, 10, 20 etc.

Implementation

At each period, either the Thompson sampling probabilities or UCB1 values are calculated based on the outcomes from the number of prior_periods specified weighted by discount_rate. New treatments are then assigned randomly using the Thompson sampling probabilities via the randomizr package, or as the treatment with the highest UCB1 values, while implementing the specific treatment blocking and control augmentation specified.

After assigning treatments, observations with new treatments have their outcomes imputed using success rates from the original randomized trial. These rates are estimated as grouped means within each treatment arm. If blocking is specified, rates are estimated within each combination of treatment arm and block.

If delayed_feedback = TRUE, new dates of success will be imputed using the means of those dates in the period, grouped by treatment block if necessary. Observations for which their treatment changed, but their outcome was success in the original and simulation, do not have their date changed. When the next period starts, the success dates are checked against the maximum/latest assignment_date for the period, and if any success occurs after that, it is treated as a failure for the purpose of the bandit decision algorithms.

Inference

At the end of the simulation the results are aggregated together to calculate the Adaptively Weighted Augmented Inverse Probability Estimator (Hadad et al. 2021) using the mean and variance formulas provided, under the constant allocation rate adaptive schema. These estimators are unbiased and asymptotically normal under the adaptive conditions and their differences are also unbiased asymptotically normal estimators for treatment effects. See Hadad et al. (2021). Asymptotic validity hinges on sub-optimal arms continually being assigned, if arms have low to 0 probability of being assigned, the central limit theorem proved no longer applies. Thus it is recommended to use random_assign_prop and control_augment to ensure all arms have non-zero probabilities of assignment over the whole trial.

Under clustering the unit of observation becomes the cluster, the sample size becomes the number of clusters. Individual estimates are aggregated in each period by cluster before being used to compute the final AIPW estimate and variance (CR0 style). The variance is adjusted by the Stata CR1 adjustment, (\frac{G}{G-1} * \frac{N-1}{N-k}) where k is the number of treatments, and G is the number of clusters. Degrees of freedom of G-1 are also provided, for use of the more conservative t-distribution, though inference is still only valid asymptotically.

Inverse Probability Weighted (IPW) estimates are also provided using estimatr::lm_robust(). Offer-Westort et al. (2021). In clustered cases CR2 standard errors are used, and CR1 (Stata) used if CR2 computation fails. HC2 standard errors are used in non-clustered cases. In high sample sizes for the arms chosen, standard t-tests of the estimates and their contrasts can be asymptotically valid. F-statistics are provided for joint tests provided in joint_test().

AIPW and IPW are unbiased, with AIPW having lower variances generally, while standard unweighted OLS estimates will be biased with spuriously low variance, but are provided for comparisons.

Performance Concerns

This procedure has the potential to be computationally expensive and time-consuming. Performance depends on the relative size of each period, number of periods, the overall size of the dataset, and number of replications. This function has separate support for data.frames and data.tables. If a data.frame is passed, the function uses a combination of dplyr, tidyr and base R to shape data, and run the simulation. However, if a data.table is passed the function exclusively uses the data.table code for all the same operations.

In general, smaller batches run faster under base R, while larger ones could benefit from the performance and memory efficiencies provided by data.table. However, we've observed larger datasets can cause numerical instability with some calculations in the Thompson sampling procedure. Internal safeguards exist to prevent this, but the best way to preempt any issues is to set prior_periods to a low number.

r > 1

Multiple simulations allows researchers to gauge the variance of the simulation procedure itself, by repeating it several times under different random states, using the same fixed data.

Parallel Processing

The function provides support for parallel processing via the future and furrr packages. When conducting a large number of simulations, parallelization can improve performance if sufficient system resources are available. Parallel processing must be explicitly set by the user, through future::plan(). Windows users should set the plan to "multisession", while Linux and MacOS users can use "multicore" or "multisession". Users running in a High Performance Computing environment (HPC), are encouraged to use future.batchtools, for their respective HPC scheduler. Note that parallel processing is not guaranteed to work on all systems, and may require additional setup or debugging effort from the user. For any issues, users are encouraged to consult the documentation of the above packages.

Value

Depends on r value if r = 1, an S3 single_rct_mab class object, and if r > 1, an S3 muti_rct_mab, with the following:

• new_data: tibble or data.table containing the new treatment assignments and outcomes under the simulation. If r >1 and keep_data = TRUE, the tables from each trial are nested inside.

• bandits: A list with 3 elements:

  • statistic: Thompson Sampling or UCB1 statistics computed for each treatment at each period of each trial.

  • assignment_prob: Assignment probabilities for each treatment at each period of each trial.

  • assignment_quant: Assignment quantities for each treatment in each trial.

• means: A tibble or data.table containing point estimates, and standard errors for the AIPW, IPW, and OLS estimators for each treatment in each trial.

• f_stats: A named numeric vector of F statistics from IPW and OLS regressions. When r > 1, it is a data.frame/data.table of F statistics with columns corresponding to IPW and OLS.

• contrasts: A tibble or data.table containing point estimates, and standard errors for the estimated linear contrasts of treatment arm estimates for each trial. Only when contrasts is not NULL.

• models: List containing lm_robust objects from IPW and OLS regressions, only stored when keep_models = TRUE or r = 1 and clusters are provided.

• config: Configuration list of 3 elements:

  • args: List of arguments passed to simulate_mab().

  • call: The original call to simulate_mab().

  • parallel: The furrr::furrr_options() object used for parallelization.

References

Agrawal, Shipra, and Navin Goyal. 2012. "Analysis of Thompson Sampling for the Multi-Armed Bandit Problem." Proceedings of the 25th Annual Conference on Learning Theory, June 16, 39.1-39.26. https://proceedings.mlr.press/v23/agrawal12.html.

Asyuraa, F. C., S. Abdullah, and T. E. Sutanto. 2021. "Empirical Evaluation on Discounted Thompson Sampling for Multi-Armed Bandit Problem with Piecewise-Stationary Bernoulli Arms." Journal of Physics: Conference Series 1722 (1): 012096. doi:10.1088/1742-6596/1722/1/012096

Auer, Peter, Nicolò Cesa-Bianchi, and Paul Fischer. 2002. "Finite-Time Analysis of the Multiarmed Bandit Problem." Machine Learning 47 (2): 235–56. doi:10.1023/A:1013689704352.

Bengtsson, Henrik. 2025. "Future: Unified Parallel and Distributed Processing in R for Everyone." https://cran.r-project.org/package=future.

Bengtsson, Henrik. 2025. "Future.Batchtools: A Future API for Parallel and Distributed Processing Using ‘Batchtools.’" https://cran.r-project.org/package=future.batchtools.

Garivier, Aurélien, and Eric Moulines. 2008. "On Upper-Confidence Bound Policies for Non-Stationary Bandit Problems." arXiv:0805.3415. Preprint, arXiv, May 22. doi:10.48550/arXiv.0805.3415

Hadad, Vitor, David A. Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. 2021. "Confidence Intervals for Policy Evaluation in Adaptive Experiments." Proceedings of the National Academy of Sciences of the United States of America 118 (15): e2014602118. doi:10.1073/pnas.2014602118.

Kuleshov, Volodymyr, and Doina Precup. 2014. "Algorithms for Multi-Armed Bandit Problems." arXiv. doi:10.48550/arXiv.1402.6028.

Loecher, Thomas Lotze and Markus. 2022. "Bandit: Functions for Simple a/B Split Test and Multi-Armed Bandit Analysis." https://cran.r-project.org/package=bandit.

Offer‐Westort, Molly, Alexander Coppock, and Donald P. Green. 2021. "Adaptive Experimental Design: Prospects and Applications in Political Science." American Journal of Political Science 65 (4): 826–44. doi:10.1111/ajps.12597.

Slivkins, Aleksandrs. 2024. "Introduction to Multi-Armed Bandits." arXiv. doi:10.48550/arXiv.1904.07272.

Vaughan, Davis, Matt Dancho, and RStudio. 2022. "Furrr: Apply Mapping Functions in Parallel Using Futures." https://cran.r-project.org/package=furrr.

See Also

furrr, future, joint_test(), simulate_mab()

Examples

data(tanf)
set.seed(454)

mab_from_rct(success ~ condition, data = tanf, algorithm = "thompson",
period_method = "batch", period_length = 500, delayed_feedback = TRUE,
assignment_date_col = appt_date, success_date_col = date_of_recert)

mab_from_rct(success ~ condition, data = tanf, algorithm = "ucb1",
period_method = "date", time_unit = "day", date_col = appt_date,
 period_length = 60, r = 2, discount_rate = 0.8)

mab_from_rct(success ~ condition + block(service_center), data = tanf, algorithm = "thompson",
period_method = "batch", period_length = 500, control_condition = "no_letter",
control_augment = 0.2, prior_periods = 1)

Runs Multi-Arm Bandit Trial

Description

Performs a full Multi-Arm Bandit (MAB) trial using Thompson Sampling or UCB1. The function provides a loop around each step of the process for each treatment wave, performing adaptive treatment assignment, and outcome imputation or generation as needed.

Usage

mab_loop(
  data,
  sim_type,
  p,
  algorithm,
  control_augment,
  random_assign_prop,
  prior_periods,
  discount_rate,
  whole_experiment = NULL,
  simulate_dates,
  delayed_feedback,
  clustering,
  blocking,
  conditions,
  col_names,
  imputation_information = NULL,
  ndraws,
  verbose,
  period_idxs,
  periods,
  num_conditions,
  time_model = NULL,
  time_model_args = NULL
)

compute_lookback(prior_periods = NULL, current_period)

Arguments

Details

The first period is used to initialize the trial, so the loop starts at period number 2.

Value

A named list containing:

• final_data: The processed tibble or data.table, with the trial's results.

• bandits: A tibble or data.table containing the UCB1 or Thompson Sampling values for each period.

• assignment_probs: A tibble or data.table containing the probability of being assigned each treatment arm at a given period.

• assignment_quantities: A numeric vector of the total number of observations assigned to each treatment arm.

Numeric value referring to the period index to look back from.

Functions

• compute_lookback(): Used during mab_loop() to create a vector of prior periods dynamically based on the specified number of prior periods.

See Also

Other simulation: collect_mab_results(), run_mab()

Miscellaneous Helper Functions

Description

Converts data.frame column into a vector using another column as the names for the vector

Shorthand Function for checking verbose and then printing if TRUE

Usage

as_named_vec(df, val, name)

verbose_log(log, message)

Arguments

Value

vector with values val and names name

Text output of message to the console when log = TRUE. If log = FALSE, returns nothing.

Precomputing Key Values for Outcome Imputation

Description

Pre-computes key values required for the outcome imputation step of the MAB procedure. Calculates the probabilities of success for each treatment block (treatment arm + any blocking specified), using the grouped means of the original experimental data. When delayed_feedback = TRUE, the average date of success is calculated for each treatment block at every period.

Usage

precompute_imputation(data, whole_experiment, delayed_feedback, col_names)

## S3 method for class 'data.frame'
precompute_imputation(data, whole_experiment, delayed_feedback, col_names)

## S3 method for class 'data.table'
precompute_imputation(data, whole_experiment, delayed_feedback, col_names)

summary_to_matrix(df)

Arguments

Details

precompute_imputation() is an optimization, meant to reduce the cost of calculating these variables within the simulation loop. When whole_experiment = TRUE, ⁠success' is a single matrix, and used through the simulation. When ⁠whole_experiment = FALSE⁠, ⁠success⁠is a list of matrices, each containing the cumulative probabilities of all periods up to the index⁠i'.

If delayed_feedback = FALSE, dates is not calculated, and is NULL. No covariates are used in the calculation, these are all simply grouped means.

Value

A named list containing:

• success: The matrix or list of matrices containing the probability of success for each treatment block, at each period.

• dates: A list of vectors containing the average success date for each treatment block at each treatment period.

A numeric matrix with row names equal to treatment_block and columns failure_rate and success_rate.

Functions

• summary_to_matrix(): Converts a summarized data.frame or data.table containing treatment_block, success_rate, and failure_rate columns into a named matrix for use with randomizr::block_ra().

See Also

Other imputation: impute_outcomes(), prep_imputation()

Outcome Imputation Preparation

Description

Executes all preparations necessary to impute outcomes for each iteration of the simulation loop. Adds an additional column to the current data, subsets necessary information from the precompute_imputation() output, and ensures compatibility with randomizr::block_ra().

Usage

prep_imputation(
  current_data,
  block_cols,
  imputation_information,
  whole_experiment,
  blocking,
  delayed_feedback,
  current_period
)

check_impute(impute_success, current_data, impute_idx)

Arguments

Details

The goal of this function is to set up the imputation procedure and prevent errors from occurring. randomizr::block_ra() does not see the names of the probabilities passed per block, so the imputation information must be subsetted to contain only the treatment blocks which exist in a given period.

When blocks are required but do not exist in the information provided it is added to the matrix, with an estimated conditional probability of success as the average across other blocks. When blocks are present but not required, they are removed. Then the rows are sorted by their names.

impute_block is the observation's new treatment block, combining any blocking variables with their new treatment assigned via the Multi-Arm-Bandit procedure.

Value

A named list containing:

• current_data: A tibble or data.table containing impute_block column to guide the outcome imputations

• impute_success: A matrix object containing probabilities of success by treatment_block used to impute outcomes taken from precompute_imputation(). Modified to remove unnecessary rows, or add necessary ones, and order the matrix appropriately, as required by randomizr::block_ra().

• impute_dates: Named date vector by treatment condition, containing the dates of success to impute if delayed_feedback is FALSE. Subsetted from the precompute_imputation() output.

Proper impute_success matrix as required by randomizr::block_ra().

See Also

Other imputation: impute_outcomes(), precompute_imputation()

Pre-Simulation Setup to Simulate a MAB Trial From an RCT

Description

Common function for all the actions that need to take place before running the Multi-Arm-Bandit re-simulation. Intakes the data and column names to check for valid arguments, format and create new columns as needed, and pre-compute key values to avoid doing so within the simulation loop.

Usage

prep_rct_data(
  data,
  random_assign_prop,
  control_augment,
  control_condition,
  period_method,
  time_unit,
  period_length,
  prior_periods,
  discount_rate,
  col_names,
  delayed_feedback,
  whole_experiment,
  verbose,
  ndraws,
  check_args,
  r,
  keep_data,
  keep_models,
  blocking,
  clustering
)

create_conditions(control_condition, data, condition_col, control_augment)

Arguments

Details

If a data.table is passed it is copied to avoid modifying the original dataset in the users environment.

Value

Named list containing:

• data: Prepared data.frame or data.table containing all the necessary columns to conduct the adaptive trial simulation, subset from the originally provided data to reduce memory usage. columns required for run_mab().

• conditions: character vector of treatment arm names, named to reflect treatment or control status.

• imputation_information: List containing necessary information for outcome and date imputation for run_mab().

• period_idxs: List of numeric vectors containing period start and end indexes.

Character vector of unique treatment conditions. Throws error if an invalid specification is used.

Functions

• create_conditions(): This function creates a character vector of treatment conditions using the conditions column in the provided data, and if control_augment is greater than 0, it also labels the control condition. Throws an error of control_condition is not present.

Prepares Data for Simulated MAB

Description

Initializes the data a simulated MAB trial. Generates block and cluster assignments, allocates all required columns, and assigns treatments and outcomes for the first period using equal assignment probabilities.

Usage

prep_sim_data(
  n,
  p,
  blocks = NULL,
  clusters = NULL,
  blocking,
  clustering,
  conditions,
  equal_probs,
  period_idxs,
  simulate_dates,
  assignment_dates = NULL,
  time_model = NULL,
  time_model_args = NULL,
  dt
)

Arguments

Value

Initialized data.table or tibble with the first period simulation conducted, and all required columns for run_mab()

See Also

Other param: generate_assignment_dates(), generate_groups(), generate_outcomes(), generate_period_idx(), setup_mab_sim(), split_args()

Simulates a MAB Trial From Prepared Inputs and Performs Inference

Description

Internal helper. Centralizes necessary functions to conduct a a MAB trial with adaptive inference. It assumes all inputs have been preprocessed already.

Usage

run_mab(
  data,
  sim_type,
  estimators = c("aipw", "ipw", "ols"),
  p = NULL,
  algorithm,
  control_augment,
  random_assign_prop,
  prior_periods,
  discount_rate,
  simulate_dates = NULL,
  delayed_feedback,
  whole_experiment = NULL,
  conditions,
  blocking,
  clustering,
  col_names,
  imputation_information = NULL,
  verbose,
  ndraws,
  period_idxs,
  keep_data,
  keep_models,
  contrasts,
  time_model = NULL,
  time_model_args = NULL
)

Arguments

Value

A named list containing:

• final_data: The processed tibble or data.table, with the trial's results.

• bandits: A tibble or data.table containing the UCB1 or Thompson Sampling values for each period.

• assignment_probs: A tibble or data.table containing the probability of being assigned each treatment arm at a given period.

• assignment_quantities: A numeric vector of the total number of observations assigned to each treatment arm.

• means: A tibble or data.table containing the mean estimates of the specified estimators for each treatment arm.

• contrasts: A tibble or data.table containing the contrast estimates of the specified estimators for the specified contrast structure. #'

• f_stats: Named numeric vector containing f_stat from IPW and OLS regressions. #'

• models: A nested list containing the lm_robust objects from regressions, only saved in clustered case.

• call: NULL; initialized for later assignment.

• args: NULL; initialized for later assignment.

• furrr: NULL; initialized for later assignment.

See Also

Other simulation: collect_mab_results(), mab_loop()

Run a single MAB simulation iteration

Description

Runs a single iteration of a MAB simulation. Intended to be called either directly for r = 1 or as the mapped function inside furrr::future_map()for r > 1⁠. Handles all ⁠sim_type'.

Usage

run_mab_single(
  sim_type,
  algorithm,
  estimators = c("aipw", "ipw", "ols"),
  control_augment = 0,
  random_assign_prop = 0,
  prior_periods = NULL,
  delayed_feedback = FALSE,
  discount_rate = 1,
  conditions,
  blocking,
  clustering,
  col_names,
  ndraws = 5000,
  keep_data = FALSE,
  keep_models = FALSE,
  contrasts = NULL,
  verbose = FALSE,
  r = 1,
  imputation_information = NULL,
  whole_experiment = NULL,
  time_model = NULL,
  time_model_args = NULL,
  p = NULL,
  n = NULL,
  dt = NULL,
  blocks = NULL,
  clusters = NULL,
  equal_probs = NULL,
  assignment_dates = NULL,
  simulate_dates = NULL,
  period_idxs,
  data = NULL
)

Arguments

Value

A named list containing:

• final_data: The processed tibble or data.table, with the trial's results.

• bandits: A tibble or data.table containing the UCB1 or Thompson Sampling values for each period.

• assignment_probs: A tibble or data.table containing the probability of being assigned each treatment arm at a given period.

• assignment_quantities: A numeric vector of the total number of observations assigned to each treatment arm.

• means: A tibble or data.table containing the mean estimates of the specified estimators for each treatment arm.

• contrasts: A tibble or data.table containing the contrast estimates of the specified estimators for the specified contrast structure. #'

• f_stats: Named numeric vector containing f_stat from IPW and OLS regressions. #'

• models: A nested list containing the lm_robust objects from regressions, only saved in clustered case.

• call: NULL; initialized for later assignment.

• args: NULL; initialized for later assignment.

• furrr: NULL; initialized for later assignment.

Set Up MAB Simulation

Description

Performs all one-time set-up required for simulate_mab() as opposed to prep_sim_data() which needs to be re-run each period.

Usage

setup_mab_sim(
  n,
  t,
  p,
  blocks,
  clusters,
  assignment_dates,
  time_model,
  period_sizes
)

Arguments

Value

A named list containing:

• period_idxs: A list of 2 integer vectors of period boundary indices.

• assignment_dates: Vector of assignment dates based on provided dates.

• blocking: Logical; TRUE if blocks is non-null.

• clustering: Logical; TRUE if clusters is non-null.

• simulate_dates: Logical; TRUE if both time_model is a function and assignment_dates is non-null.

• p: The success probability matrix with lowercase and sorted rownames with rows reordered to match conditions.

• conditions: A named character vector of arm labels sorted alphabetically, with names "control" or "treatment" as appropriate.

• equal_probs: A named numeric vector of equal assignment probabilities 1 / K for each of the K arms.

• col_names: A fixed named list of output column name strings.

See Also

Other param: generate_assignment_dates(), generate_groups(), generate_outcomes(), generate_period_idx(), prep_sim_data(), split_args()

Simulate an Adaptive Trial With Bernoulli Distributed Outcomes

Description

Simulates a response-adaptive randomized experiment with Bernoulli distributed outcomes. At each period, observed outcomes are used to update assignment probabilities according to the specified algorithm. algorithm = "static" is the non-adaptive uniform baseline, where probabilities of being assigned to one treatment is the same as any other.

Usage

simulate_mab(
  n,
  t = n,
  p,
  algorithm = c("thompson", "ucb1"),
  blocks = NULL,
  clusters = NULL,
  control_augment = 0,
  random_assign_prop = 0,
  delayed_feedback = FALSE,
  assignment_dates = NULL,
  time_model = NULL,
  period_sizes = NULL,
  prior_periods = NULL,
  discount_rate = 1,
  dt = FALSE,
  ndraws = 5000,
  r = 1,
  keep_data = FALSE,
  keep_models = FALSE,
  contrasts = NULL,
  check_args = TRUE,
  verbose = FALSE,
  ...
)

Arguments

Details

Blocking and Clustering

When blocking and/or clustering are specified, these assignments will be randomly pregenerated before the start of the adaptive sequential assignment. These arguments allow simulating a trial when there may be heterogeneous outcomes across a treatment block or treatment cluster, so different assignment probabilities can be provided for the same treatment, depending on the block and/or cluster of a unit.

Clusters should be contained inside each assignment wave (a warning is thrown if this is not the case), so it is possible to have 2 observations in the same cluster assigned to different treatments if they were assigned in different waves. This is assumed because without it the adaptive probabilities will not be impacting assignments. For example if someone in cluster 1 is assigned in period 1, then all other members are forced to have the same treatment, even if they are assigned in period 5, 10, 20 etc.

Implementation

At each period, either the Thompson sampling probabilities or UCB1 values are calculated based on the outcomes from the number of prior_periods specified weighted by discount_rate. New treatments are then assigned randomly using the Thompson sampling probabilities via the randomizr package, or as the treatment with the highest UCB1 values, while implementing the specific treatment blocking and control augmentation specified.

After assigning treatments, observations will have their outcomes generated via a Bernoulli draw associated to the probability in the p matrix corresponding to their treatment and block/cluster. If delayed_feedback = TRUE, dates of success will be generated via the provided time_model() function. When the next period starts, the success dates are checked against the maximum/latest assignment_date for the period, and if any success occurs after that, it is treated as a failure for the purpose of the bandit decision algorithms.

Inference

At the end of the simulation the results are aggregated together to calculate the Adaptively Weighted Augmented Inverse Probability Estimator (Hadad et al. 2021) using the mean and variance formulas provided, under the constant allocation rate adaptive schema. These estimators are unbiased and asymptotically normal under the adaptive conditions and their differences are also unbiased asymptotically normal estimators for treatment effects. Hadad et al. (2021). Asymptotic validity, hinges on sub-optimal arms continually being assigned, if arms have low to 0 probability of being assigned, the central limit theorem proved no longer applies. Thus it is recommended to use random_assign_prop and control_augment to ensure all arms non-zero probabilities of assignment over the whole trial.

Under clustering the unit of observation becomes the cluster, the sample size the number of clusters. Individual estimates are aggregated in each period by cluster before being used to compute the final AW-AIPW estimate and variance (CR0 style). The variance is adjusted by the Stata CR1 adjustment, (\frac{G}{G-1} * \frac{N-1}{N-k}) where k is the number of treatments, and G is the number of clusters. Degrees of freedom of G-1 are also provided, for use of the more conservative t-distribution, though inference is still only valid asymptotically.

Inverse Probability Weighted (IPW) estimates are also provided using estimatr::lm_robust(). Offer-Westort et al. (2021). In clustered cases CR2 standard errors are used, and CR1 (Stata) used if CR2 computation fails. HC2 standard errors are used in non-clustered cases. In high sample sizes for the arms chosen, standard t-tests of the estimates and their contrasts can be asymptotically valid. F-statistics are provided for joint tests provided in joint_test().

AW-AIPW and IPW are unbiased, with AW-AIPW having lower variances generally, while standard unweighted OLS estimates will be biased with spuriously low variance but are provided for comparisons.

r > 1

Multiple simulations allow researchers to gauge the variance of the procedure and produce bootstrap estimates of variance of the procedure under the passed parameters. For each simulation new data is drawn according to the passed population parameters. This differs from mab_from_rct() where resimulations occurs on the same fixed dataset.

Further details about the adaptive procedure can be found in mab_from_rct()

Performance Concerns

This procedure has the potential to be computationally expensive and time-consuming. Performance depends on the relative size of each period, t, n, and r. This function has separate support for data.frames and data.tables, selected by the dt argument. This flag defines two separate tracks where either combination of dplyr, tidyr and base R to shape data, and run the simulation or only data.table code operations are used.

In general, smaller batches run faster under base R, while larger ones could benefit from the performance and memory efficiencies provided by data.table. However, we've observed larger sizes can cause numerical instability with some calculations in the Thompson sampling procedure. Internal safeguards exist to prevent this, but the best way to preempt any issues is to set prior_periods to a low number.

Parallel Processing

The function provides support for parallel processing via the future and furrr packages. When conducting a large number of simulations, parallelization can improve performance if sufficient system resources are available. Parallel processing must be explicitly set by the user, through future::plan(). Windows users should set the plan to "multisession", while Linux and MacOS users can use "multicore" or "multisession". Users running in a High Performance Computing environment (HPC), are encouraged to use future.batchtools, for their respective HPC scheduler. Note that parallel processing is not guaranteed to work on all systems, and may require additional setup or debugging effort from the user. For any issues, users are encouraged to consult the documentation of the above packages.

Value

Depends on r value if r = 1, an S3 single_param_mab class object, and if r > 1, an S3 muti_param_mab, with the following:

• new_data: tibble or data.table containing the new treatment assignments and outcomes under the simulation. If r >1 and keep_data = TRUE, the tables from each trial are nested inside.

• bandits: A list with 3 elements:

  • statistic: Thompson Sampling or UCB1 statistics computed for each treatment at each period of each trial.

  • assignment_prob: Assignment probabilities for each treatment at each period of each trial.

  • assignment_quant: Assignment quantities for each treatment in each trial.

• means: A tibble or data.table containing point estimates, and standard errors for the AW-AIPW, IPW, and OLS estimators for each treatment in each trial.

• f_stats: A named numeric vector of F statistics from IPW and OLS regressions. When r > 1, it is a data.frame/data.table of F statistics with columns corresponding to IPW and OLS.

• contrasts: A tibble or data.table containing point estimates, and standard errors for the estimated linear contrasts of treatment arm estimates for each trial. Only when contrasts is not NULL.

• models: List containing lm_robust objects from IPW and OLS regressions, only stored when keep_models = TRUE or r = 1 and clusters are provided.

• config: Configuration list of 3 elements:

  • args: List of arguments passed to simulate_mab().

  • call: The original call to simulate_mab().

  • parallel: The furrr::furrr_options() object used for parallelization.

References

Agrawal, Shipra, and Navin Goyal. 2012. "Analysis of Thompson Sampling for the Multi-Armed Bandit Problem." Proceedings of the 25th Annual Conference on Learning Theory, June 16, 39.1-39.26. https://proceedings.mlr.press/v23/agrawal12.html.

Asyuraa, F. C., S. Abdullah, and T. E. Sutanto. 2021. "Empirical Evaluation on Discounted Thompson Sampling for Multi-Armed Bandit Problem with Piecewise-Stationary Bernoulli Arms." Journal of Physics: Conference Series 1722 (1): 012096. doi:10.1088/1742-6596/1722/1/012096

Auer, Peter, Nicolò Cesa-Bianchi, and Paul Fischer. 2002. "Finite-Time Analysis of the Multiarmed Bandit Problem." Machine Learning 47 (2): 235–56. doi:10.1023/A:1013689704352.

Bengtsson, Henrik. 2025. "Future: Unified Parallel and Distributed Processing in R for Everyone." https://cran.r-project.org/package=future.

Bengtsson, Henrik. 2025. "Future.Batchtools: A Future API for Parallel and Distributed Processing Using ‘Batchtools.’" https://cran.r-project.org/package=future.batchtools.

Garivier, Aurélien, and Eric Moulines. 2008. "On Upper-Confidence Bound Policies for Non-Stationary Bandit Problems." arXiv:0805.3415. Preprint, arXiv, May 22. doi:10.48550/arXiv.0805.3415

Hadad, Vitor, David A. Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. 2021. "Confidence Intervals for Policy Evaluation in Adaptive Experiments." Proceedings of the National Academy of Sciences of the United States of America 118 (15): e2014602118. doi:10.1073/pnas.2014602118.

Kuleshov, Volodymyr, and Doina Precup. 2014. "Algorithms for Multi-Armed Bandit Problems." arXiv. doi:10.48550/arXiv.1402.6028.

Loecher, Thomas Lotze and Markus. 2022. "Bandit: Functions for Simple a/B Split Test and Multi-Armed Bandit Analysis." https://cran.r-project.org/package=bandit.

Offer‐Westort, Molly, Alexander Coppock, and Donald P. Green. 2021. "Adaptive Experimental Design: Prospects and Applications in Political Science." American Journal of Political Science 65 (4): 826–44. doi:10.1111/ajps.12597.

Slivkins, Aleksandrs. 2024. "Introduction to Multi-Armed Bandits." arXiv. doi:10.48550/arXiv.1904.07272.

Vaughan, Davis, Matt Dancho, and RStudio. 2022. "Furrr: Apply Mapping Functions in Parallel Using Futures." https://cran.r-project.org/package=furrr.

See Also

mab_from_rct()

Examples

n <- 100
t <- 10

p <- matrix(c(0.2, 0.5, 0.45, 0.5), ncol =1, dimnames = list(paste0("T", 1:4)))

set.seed(543)
simulate_mab(n, t, p = p, random_assign_prop = 0.1,
algorithm =  "ucb1", discount_rate  = 0.5)

simulate_mab(n, t, p = p, random_assign_prop = 0.1,
period_sizes = c(37, rep(7, 9)), algorithm = "thompson")


simulate_mab(n, t, p = matrix(c(0.1, 0.5, 0.3, 0.2, 0.3, 0.3),
dimnames = list(c("T1", "T2"), c("B1", "B2", "B3")),
ncol = 3, nrow = 2), blocks = c("B1" =0.3, "B2" = 0.5, "B3" = 0.2),
algorithm = "thompson")

Split Function Arguments

Description

Uses methods::formalArgs() to match arguments provided to ... of simulate_mab() to furrr::furrr_options() and the user specified time_model

Usage

split_args(time_model = NULL, ...)

Arguments

Value

A named list with 2 elements, furr_args and time_model_args each a list of the respective arguments to furrr::furrr_options() and the user specified time_model

See Also

Other param: generate_assignment_dates(), generate_groups(), generate_outcomes(), generate_period_idx(), prep_sim_data(), setup_mab_sim()

Public TANF Recipient Data From Washington D.C

Description

A modified version of the data set used in https://thelabprojects.dc.gov/benefits-reminder-letter with one additional column added for analysis.

Usage

data(tanf)

Format

An object of class tbl_df (inherits from tbl, data.frame) with 3517 rows and 21 columns.

Details

Variables are as follows:

ic_case_id

Unique, anonymized case identifier.

service_center

DC Department of Human Services Center assigned each case.

condition

The assigned letter condition: "No Letter", "Open Appointment", or "Specific Appointment".

recert_month

Recertification Month.

letter_sent_date

Date the second (treatment) letter was sent.

recert_id

Administrative recertification identifier.

return_to_sender

Indicates whether letter was returned as undeliverable

pdc_status

PDC Status

renewal_date

Date by which renewal must be completed.

notice_date.x

Date the first notice was sent (initial legal communication)

days_betwn_notice_and_recert_due

Number of days between the first notice and the recertification due date.

cert_period_start

Start date of the recertification period.

cert_period_end

End date of recertification period.

recert_status

Status of recertification process (Pending, Denied, etc.)

denial_reason

Reason for denial if recertification was not approved.

recert_month_year

Combined recertification month and year.

notice_date.y

Alternate record of first notice date.

recert_status_dcas

Official recertification status from DCAS

date_of_recert

Date the recertification was successfully submitted (if applicable).

success

Binary variable indicating successful recertification based on recert_status (newly added column).

Source

https://github.com/thelabdc/DHS-TANFRecertification-Public/blob/main/data/df_replication_anonymized.csv