Simulation based calibration for RBesT

Sebastian Weber

Fri Sep 18 08:00:47 2026

This report documents the results of a simulation based calibration (SBC) run for RBesT. The calibration data will be generated whenever relevant changes to the gMAP function were made. The calibration runs are performed for typical use cases of gMAP. These include the three likelihoods (binomial, gaussian & Poisson), a sparse (\(2\) trials) and dense (\(10\) trials) data situation and finally a run with a very/less conservative prior choice for between-trial heterogeniety parameter.

The calibration data presented here has been generated at and with the RBesT git version as:

## Created:  2026-09-17 12:05:03 UTC
## git hash: 4c6c925273420bd42e043205618ad3c66105f364
## MD5:      81ad9dececcbcdaaf7f97fd43f7768f0

The MD5 hash of the calibration data file presented here must match the above listed MD5:

## /home/runner/work/RBesT/RBesT/inst/sbc/calibration.rds 
##                     "81ad9dececcbcdaaf7f97fd43f7768f0"

Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

  1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

  2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

  3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

  4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

Hierarchical intercept only (random-effects intercept) model for binomial, gaussian and Poisson likelihood

Likelihood:

Hierarchical prior:

\[ g(\theta_j)|\mu,\tau \sim \mbox{Normal}(\mu, \tau^2)\]

\[\mu \sim \mbox{Normal}(m_\mu, s^2_\mu)\] \[\tau \sim \mbox{Normal}^+(0, s^2_\tau)\]

The fake data simulation function returns for binomial and Poisson data the sum of the responses while for normal the mean summary is used. Please refer to the sbc_tools.R and make_reference_rankhist.R R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

SBC results

Sampler Diagnostics Overview

family data_scenario sd_tau N total_divergent total_divergent_sim_fraction min_ess max_Rhat total_large_Rhat min_lp_ess_bulk min_lp_ess_tail
binomial dense 0.5 10000 4 0.000 850 1.008 0 984.875 929.862
binomial dense 1.0 10000 1 0.000 859 1.008 0 1021.432 964.322
binomial sparse 0.5 10000 11 0.001 790 1.010 0 778.490 425.093
binomial sparse 1.0 10000 93 0.005 122 1.017 0 142.436 26.561
gaussian dense 0.5 10000 0 0.000 509 1.008 0 1054.370 1058.490
gaussian dense 1.0 10000 0 0.000 852 1.006 0 989.745 1468.484
gaussian sparse 0.5 10000 43 0.003 667 1.006 0 751.046 249.874
gaussian sparse 1.0 10000 297 0.015 94 1.021 0 94.597 27.566
poisson dense 0.5 10000 24 0.001 343 1.007 0 976.693 1269.858
poisson dense 1.0 10000 20 0.001 726 1.006 0 1062.405 1339.773
poisson sparse 0.5 10000 210 0.011 252 1.010 0 267.079 58.943
poisson sparse 1.0 10000 734 0.023 74 1.036 0 86.254 46.044

Note: Large Rhat is defined as exceeding 1.2.

Summary Statistics

\(\chi^2\) Statistic, \(\mu\)

data_scenario likelihood sd_tau parameter statistic df p.value
1 dense binomial 0.5 mu 66.355 63 0.362
13 dense binomial 1 mu 48.896 63 0.904
25 dense gaussian 0.5 mu 64.896 63 0.410
37 dense gaussian 1 mu 60.288 63 0.574
49 dense poisson 0.5 mu 47.834 63 0.922
61 dense poisson 1 mu 69.798 63 0.260
73 sparse binomial 0.5 mu 54.170 63 0.778
78 sparse binomial 1 mu 86.170 63 0.028
83 sparse gaussian 0.5 mu 67.507 63 0.326
88 sparse gaussian 1 mu 58.355 63 0.642
93 sparse poisson 0.5 mu 59.610 63 0.598
98 sparse poisson 1 mu 73.830 63 0.165

\(\chi^2\) Statistic, \(\tau\)

data_scenario likelihood sd_tau parameter statistic df p.value
2 dense binomial 0.5 tau 60.774 63 0.556
14 dense binomial 1 tau 82.304 63 0.052
26 dense gaussian 0.5 tau 64.397 63 0.427
38 dense gaussian 1 tau 61.722 63 0.522
50 dense poisson 0.5 tau 53.798 63 0.789
62 dense poisson 1 tau 81.510 63 0.058
74 sparse binomial 0.5 tau 49.818 63 0.886
79 sparse binomial 1 tau 79.885 63 0.074
84 sparse gaussian 0.5 tau 69.056 63 0.280
89 sparse gaussian 1 tau 69.056 63 0.280
94 sparse poisson 0.5 tau 62.694 63 0.487
99 sparse poisson 1 tau 63.782 63 0.449

\(\chi^2\) Statistic, group estimates \(\theta\)

data_scenario likelihood sd_tau parameter statistic df p.value
3 dense binomial 0.5 theta[10] 45.030 63 0.958
4 dense binomial 0.5 theta[1] 70.042 63 0.253
5 dense binomial 0.5 theta[2] 50.752 63 0.867
6 dense binomial 0.5 theta[3] 73.574 63 0.170
7 dense binomial 0.5 theta[4] 54.029 63 0.782
8 dense binomial 0.5 theta[5] 45.043 63 0.958
9 dense binomial 0.5 theta[6] 53.594 63 0.795
10 dense binomial 0.5 theta[7] 81.843 63 0.056
11 dense binomial 0.5 theta[8] 62.336 63 0.500
12 dense binomial 0.5 theta[9] 59.354 63 0.607
15 dense binomial 1 theta[10] 48.128 63 0.917
16 dense binomial 1 theta[1] 55.654 63 0.733
17 dense binomial 1 theta[2] 61.734 63 0.522
18 dense binomial 1 theta[3] 61.555 63 0.528
19 dense binomial 1 theta[4] 55.168 63 0.748
20 dense binomial 1 theta[5] 50.547 63 0.871
21 dense binomial 1 theta[6] 65.664 63 0.385
22 dense binomial 1 theta[7] 71.846 63 0.208
23 dense binomial 1 theta[8] 62.106 63 0.508
24 dense binomial 1 theta[9] 66.368 63 0.362
27 dense gaussian 0.5 theta[10] 63.168 63 0.470
28 dense gaussian 0.5 theta[1] 76.710 63 0.115
29 dense gaussian 0.5 theta[2] 56.410 63 0.708
30 dense gaussian 0.5 theta[3] 60.698 63 0.559
31 dense gaussian 0.5 theta[4] 53.478 63 0.798
32 dense gaussian 0.5 theta[5] 63.808 63 0.448
33 dense gaussian 0.5 theta[6] 75.034 63 0.143
34 dense gaussian 0.5 theta[7] 54.259 63 0.776
35 dense gaussian 0.5 theta[8] 66.074 63 0.371
36 dense gaussian 0.5 theta[9] 70.221 63 0.249
39 dense gaussian 1 theta[10] 66.995 63 0.342
40 dense gaussian 1 theta[1] 50.214 63 0.878
41 dense gaussian 1 theta[2] 58.227 63 0.647
42 dense gaussian 1 theta[3] 54.797 63 0.760
43 dense gaussian 1 theta[4] 55.693 63 0.732
44 dense gaussian 1 theta[5] 53.363 63 0.801
45 dense gaussian 1 theta[6] 78.003 63 0.097
46 dense gaussian 1 theta[7] 58.854 63 0.625
47 dense gaussian 1 theta[8] 74.726 63 0.148
48 dense gaussian 1 theta[9] 55.552 63 0.736
51 dense poisson 0.5 theta[10] 55.629 63 0.734
52 dense poisson 0.5 theta[1] 83.891 63 0.040
53 dense poisson 0.5 theta[2] 62.630 63 0.489
54 dense poisson 0.5 theta[3] 65.766 63 0.381
55 dense poisson 0.5 theta[4] 39.795 63 0.990
56 dense poisson 0.5 theta[5] 71.757 63 0.210
57 dense poisson 0.5 theta[6] 70.733 63 0.235
58 dense poisson 0.5 theta[7] 61.146 63 0.543
59 dense poisson 0.5 theta[8] 47.104 63 0.933
60 dense poisson 0.5 theta[9] 70.259 63 0.248
63 dense poisson 1 theta[10] 88.794 63 0.018
64 dense poisson 1 theta[1] 53.069 63 0.809
65 dense poisson 1 theta[2] 70.669 63 0.237
66 dense poisson 1 theta[3] 69.427 63 0.270
67 dense poisson 1 theta[4] 79.808 63 0.075
68 dense poisson 1 theta[5] 58.074 63 0.652
69 dense poisson 1 theta[6] 77.094 63 0.109
70 dense poisson 1 theta[7] 76.941 63 0.111
71 dense poisson 1 theta[8] 52.096 63 0.835
72 dense poisson 1 theta[9] 70.438 63 0.243
75 sparse binomial 0.5 theta[1] 66.906 63 0.345
76 sparse binomial 0.5 theta[2] 60.685 63 0.559
77 sparse binomial 0.5 theta[3] 62.976 63 0.477
80 sparse binomial 1 theta[1] 60.698 63 0.559
81 sparse binomial 1 theta[2] 60.378 63 0.570
82 sparse binomial 1 theta[3] 57.677 63 0.666
85 sparse gaussian 0.5 theta[1] 73.050 63 0.181
86 sparse gaussian 0.5 theta[2] 53.824 63 0.788
87 sparse gaussian 0.5 theta[3] 71.232 63 0.223
90 sparse gaussian 1 theta[1] 52.070 63 0.836
91 sparse gaussian 1 theta[2] 74.637 63 0.150
92 sparse gaussian 1 theta[3] 53.018 63 0.811
95 sparse poisson 0.5 theta[1] 71.898 63 0.207
96 sparse poisson 0.5 theta[2] 89.638 63 0.015
97 sparse poisson 0.5 theta[3] 67.315 63 0.332
100 sparse poisson 1 theta[1] 81.114 63 0.062
101 sparse poisson 1 theta[2] 63.475 63 0.460
102 sparse poisson 1 theta[3] 54.323 63 0.774

Session Info

## R version 4.6.1 (2026-06-24)
## Platform: x86_64-pc-linux-gnu
## Running under: Ubuntu 24.04.5 LTS
## 
## Matrix products: default
## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
## 
## locale:
##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8       
##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8   
##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C          
## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   
## 
## time zone: UTC
## tzcode source: system (glibc)
## 
## attached base packages:
## [1] tools     stats     graphics  grDevices utils     datasets  methods  
## [8] base     
## 
## other attached packages:
##  [1] purrr_1.2.2        rstan_2.32.7       StanHeaders_2.39.1 here_1.0.2        
##  [5] ggplot2_4.0.3      broom_1.0.13       tidyr_1.3.2        dplyr_1.2.1       
##  [9] assertthat_0.2.1   knitr_1.52        
## 
## loaded via a namespace (and not attached):
##  [1] sass_0.4.10        generics_0.1.4     digest_0.6.39      magrittr_2.0.5    
##  [5] evaluate_1.0.5     grid_4.6.1         RColorBrewer_1.1-3 fastmap_1.2.0     
##  [9] rprojroot_2.1.1    jsonlite_2.0.0     pkgbuild_1.4.8     backports_1.5.1   
## [13] gridExtra_2.3.1    QuickJSR_1.11.0    scales_1.4.0       codetools_0.2-20  
## [17] jquerylib_0.1.4    cli_3.6.6          rlang_1.3.0        withr_3.0.3       
## [21] cachem_1.1.0       yaml_2.3.12        otel_0.2.0         parallel_4.6.1    
## [25] inline_0.3.21      vctrs_0.7.3        R6_2.6.1           matrixStats_1.5.0 
## [29] stats4_4.6.1       lifecycle_1.0.5    pkgconfig_2.0.3    RcppParallel_6.2.1
## [33] pillar_1.11.1      bslib_0.12.0       gtable_0.3.6       loo_2.10.1        
## [37] glue_1.8.1         Rcpp_1.1.2         xfun_0.61          tibble_3.3.1      
## [41] tidyselect_1.2.1   farver_2.1.2       htmltools_0.5.9    rmarkdown_2.32    
## [45] compiler_4.6.1     S7_0.2.2