This vignette shows how to use the latent projection predictive feature selection from Catalina, Bürkner, and Vehtari (2021) in projpred. We recommend to read the main vignette first, as the latent-projection vignette presented here will skip some of the details explained in the main vignette.
The response families used in GLMs (McCullagh and Nelder 1989, chap.
2), GLMMs, GAMs, and GAMMs (in particular, the
gaussian(), the binomial(), and the
poisson() family which are supported by
projpred’s traditional projection) may be termed
exponential dispersion (ED) families (Jørgensen 1987)1. For a
response family that is not an ED family, the Kullback-Leibler (KL)
divergence minimization problem (see Piironen, Paasiniemi, and Vehtari
2020) is often not easy to solve analytically (exceptions are
non-ED families that are discrete and have finite support; see the
comment on the augmented-data projection in section “Implementation”). In order to bypass this issue, the
latent projection (Catalina, Bürkner, and Vehtari
2021) solves the KL minimization problem in the predictive
space of the latent predictors2 instead of in the predictive space of the
original response values.
To this end, the latent predictor is assumed to have a Gaussian distribution, since it (i) constitutes a combination of predictor data and regression parameters which is often linear (in the parameters, but—less—often also in the predictor data) or at least additive (across the predictor terms) and (ii) has the complete real line as support. Furthermore, the Gaussian distribution has the highest differential entropy among all distributions with two finite moments and with the real line as support (see, e.g., Cover and Thomas 1991). In some cases, e.g., for the probit link, the Gaussian distribution is even part of the original statistical model. In case of the logit link, the Gaussian distribution with a standard deviation of 1.6 approximates the logistic distribution (with a scale parameter of 1).
The assumption of a Gaussian distribution for the latent predictors makes things a lot easier because it allows us to make use of projpred’s traditional projection.
As illustrated by the Poisson example below, the latent projection can not only be used for families not supported by projpred’s traditional projection, but it can also be beneficial for families supported by it. The remaining examples show that the latent projection offers support for models (and censored observations) that the traditional (or augmented-data) projection do not support.
To use the latent projection in projpred, argument
latent of extend_family() needs to be set to
TRUE. Since extend_family() is called by
init_refmodel() which in turn is called by
get_refmodel() (more precisely, by the
get_refmodel() methods) which in turn is called at the
beginning of the top-level functions project(),
varsel(), and cv_varsel(), it is possible to
pass latent = TRUE from such a top-level function down to
extend_family() via the ellipsis (...).
However, we recommend to define the reference model object of class
refmodel explicitly (as illustrated in the examples below)
to avoid repetitive and inefficient code3.
After performing the projection (either as a stand-alone feature via
project() or embedded in a variable selection via
varsel() or cv_varsel()), the post-processing
(e.g., the estimation of the performance statistics in
summary.vsel()) can be performed on the original response
scale. For this, there are three arguments of
extend_family() which accept R functions:
latent_ilink (responsible for the inverse-link
transformation from latent scale to response scale),
latent_ll_oscale (responsible for the calculation of
log-likelihood values on response scale), and
latent_ppd_oscale (responsible for drawing from the
(posterior-projection) predictive distribution on response scale). For
some families, these three arguments have internal defaults implemented
natively in projpred. These families are listed in the
main vignette (section “Supported
types of models”). For all other families, projpred
either tries to infer a reasonable function internally (in case of
latent_ilink) or uses a dummy function returning only
NAs (in case of latent_ll_oscale and
latent_ppd_oscale), unless the user supplies custom
functions. When creating a reference model object for a family that
lacks projpred’s native support for full response-scale
post-processing, projpred will throw messages stating
whether (and which) features will be unavailable unless at least some of
these three arguments are provided by the user. Again, the ellipsis
(...) can be used to pass these arguments from a top-level
function such as cv_varsel() down to
extend_family(). In the post-processing functions,
response-scale analyses can usually be deactivated by setting argument
resp_oscale to FALSE, with the exception of
predict.refmodel() and proj_linpred() where
arguments type and transform serve this
purpose (see the documentation).
Apart from the arguments mentioned above,
extend_family() also features the latent-projection
argument latent_y_unqs whose purpose is described in the
documentation.
While the latent projection is an approximate solution to the KL divergence minimization problem in the original response space4, the augmented-data projection (Weber, Glass, and Vehtari 2025) gives the exact5 solution for some non-ED families, namely those where the response distribution has finite support. However, the augmented-data projection comes with a higher runtime than the latent projection. The families currently supported by projpred’s augmented-data projection are also listed in the main vignette (again section “Supported types of models”).
In this example, we will illustrate that in case of a family
supported by projpred’s traditional projection (here
the poisson() family), the latent projection can reduce the
runtime of the variable selection compared to the traditional
projection.
First, we generate a dataset with a Poisson-distributed response. The linear predictor depends on 4 continuous predictors and 1 categorical predictor, and there are 20 noise predictors:
# Number of observations:
N <- 100
# Data-generating function:
sim_poiss <- function(nobs = N, ncon = 4, ncats = 4, nnoise = 20) {
# Regression coefficients for continuous predictors:
coefs_con <- seq(from = 0.2, to = 0.8, length.out = ncon)
# Continuous predictors:
dat_sim <- matrix(rnorm(nobs * ncon), ncol = ncon)
# Start linear predictor:
linpred <- 1.4 + dat_sim %*% coefs_con
# Categorical predictor:
dat_sim <- data.frame(
x = dat_sim,
xcat = gl(n = ncats, k = nobs %/% ncats, length = nobs,
labels = paste0("cat", seq_len(ncats)))
)
# Regression coefficients for the categorical predictor:
coefs_cat <- seq(from = 0.2, to = 0.8, length.out = ncats)
# Continue linear predictor:
linpred <- linpred + coefs_cat[dat_sim$xcat]
# Noise predictors:
dat_sim <- data.frame(
dat_sim,
xn = matrix(rnorm(nobs * nnoise), ncol = nnoise)
)
# Poisson response, using the log link (i.e., exp() as inverse link):
dat_sim$y <- rpois(nobs, lambda = exp(linpred))
# Shuffle order of observations:
dat_sim <- dat_sim[sample.int(nobs), , drop = FALSE]
# Drop the shuffled original row names:
rownames(dat_sim) <- NULL
return(dat_sim)
}
# Generate data:
set.seed(300417)
dat_poiss <- sim_poiss()Next, we fit the reference model that we consider as the best model (in terms of predictive performance) that we can construct (here, we assume that we don’t know about the true data-generating process even though the dataset was simulated):
[1] 25
# Prior guess for the number of relevant (i.e., non-zero) regression
# coefficients:
p0 <- 5
# Prior guess for the overall magnitude of the response values, see Table 1 of
# Piironen and Vehtari (2017, DOI: 10.1214/17-EJS1337SI):
mu_prior <- 10
# Hyperprior scale for tau, the global shrinkage parameter:
tau0 <- p0 / (D - p0) / sqrt(mu_prior) / sqrt(N)
# Set this manually if desired:
ncores <- parallel::detectCores(logical = FALSE)
### Only for technical reasons in this vignette (you can omit this when running
### the code yourself):
ncores <- min(ncores, 2L)
###
options(mc.cores = ncores)
# All other columns of the data are predictors:
refm_fml <- y ~ .
refm_fit_poiss <- stan_glm(
formula = refm_fml,
family = poisson(),
data = dat_poiss,
prior = hs(global_scale = tau0, slab_df = 100, slab_scale = 1),
### Only for the sake of speed (not recommended in general):
chains = 2, iter = 1000,
### In general, use defaults or explicitly define
### chains = 4, iter = 4000,
###
refresh = 0
)Within projpred, we define the reference model
object explicitly and set latent = TRUE in the
corresponding get_refmodel() call (see section “Implementation”) so that the latent projection is used
in downstream functions. We run cv_varsel() with its
defaults: PSIS-LOO CV and validated search. Furthermore, we measure the
runtime to be able to compare it to the traditional projection’s
later:
refm_poiss <- get_refmodel(refm_fit_poiss, latent = TRUE)
time_lat <- system.time(cvvs_lat <- cv_varsel(
refm_poiss,
### In interactive use, we recommend not to deactivate the verbose mode:
verbose = 0,
###
seed = 95930
))Cutting off the search at size 19. Use argument `nterms_max` to customize this.
Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Warning: Some (3 / 100) Pareto k's for the reference model's PSIS-LOO weights
are > 0.67.
Warning: In the recalculation of the latent response values, some (3 / 100) expectation-specific Pareto k-values are > 0.67.
In general, we recommend K-fold CV in this case.
user system elapsed
11.558 33.686 6.105
The warnings about high Pareto \(\hat{k}\) values concern only a few of the
100 observations. With the short MCMC run of this vignette (1000
posterior draws), the threshold for a reliable Pareto \(\hat{k}\) is about 0.67 (Vehtari et al.
2022). In a real analysis, we would run more MCMC iterations,
check the Pareto \(\hat{k}\) values
again, and use \(K\)-fold CV
(cv_method = "kfold") if many of them are high.
In order to decide for a submodel size, we first inspect the
plot() results. In contrast to the main vignette where we
used the mean log predictive density (MLPD) as predictive performance
statistic for a gaussian() family reference model (and
gaussian() submodels), we have a discrete family
(poisson()) here, so it makes sense to exponentiate the
MLPD to obtain the geometric mean predictive density (GMPD; in case of a
discrete response, the predictive density values are actually predictive
probabilities and hence the GMPD is bounded by 0 and 1). As in
the main vignette, we plot with deltas = TRUE (in case of
the GMPD, this means that the ratio of the submodel GMPD
vs. the reference model GMPD is shown). Via global option
projpred.plot_vsel_size_position, we set argument
size_position of plot.vsel() to
"secondary_x" to make the submodel sizes readable in all of
the plots in this vignette.
ggplot2::theme_set(bayesplot::theme_default(base_family = "sans"))
options(projpred.plot_vsel_size_position = "secondary_x")
( gg_lat <- plot(cvvs_lat, stats = "gmpd", deltas = TRUE) )Based on this plot, we decide for a submodel size of 5:
This is also the size that suggest_size() would
suggest:
[1] 5
In the predictor ranking up to the selected size of 5, we can see that projpred has correctly selected the truly relevant predictors first and only then the noise predictors:
rk_lat <- ranking(cvvs_lat)
( predictors_final_lat <- head(rk_lat[["fulldata"]], size_decided_lat) )[1] "x.4" "x.3" "x.2" "xcat" "x.1"
We will skip post-selection inference here (see the main vignette for
a demonstration of post-selection inference), but note that
proj_predict() has argument resp_oscale for
controlling whether to draw from the posterior-projection predictive
distributions on the original response scale (TRUE, the
default) or on latent scale (FALSE) and that analogous
functionality is available in proj_linpred() (argument
transform) and predict.refmodel() (argument
type).
We now run the same cv_varsel() call with
projpred’s traditional projection. For a Poisson
submodel, each projection is an iterative fit, and validated search
repeats the search for each of the 100 LOO CV folds. To keep the runtime
of this vignette short, we use fewer clusters in the performance
evaluation (nclusters_pred = 20) and search only up to size
10 (nterms_max = 10):
time_trad <- system.time(cvvs_trad <- cv_varsel(
refm_fit_poiss,
### Only for the sake of speed (not recommended in general):
nclusters_pred = 20,
nterms_max = 10,
###
### In interactive use, we recommend not to deactivate the verbose mode:
verbose = 0,
###
seed = 95930
))Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Warning: Some (3 / 100) Pareto k's for the reference model's PSIS-LOO weights
are > 0.67.
latent traditional
6.105 73.881
[1] 5
[1] "x.4" "x.3" "x.2" "xcat" "x.1"
The traditional projection takes much longer than the latent projection (see the elapsed times in seconds above), although it uses the smaller settings. It selects the same 5 predictor terms first.
This example showed that the latent projection can be advantageous also for families supported by projpred’s traditional projection. Both projections selected the truly relevant predictor terms first, but the latent projection was much faster.
In this example, we will illustrate the latent projection in case of
the negative binomial family (more precisely, we will use the
rstanarm::neg_binomial_2() family here) which is a family
that is not supported by projpred’s traditional
projection6.
We will re-use the data generated above in the Poisson example.
We now fit a reference model with the negative binomial distribution
as response family. For the sake of simplicity, we won’t adjust
tau0 to this new family, but in a real-world example, such
an adjustment would be necessary. However, since Table 1 of Piironen and Vehtari (2017) does not list the
negative binomial distribution, this would first require a manual
derivation of the pseudo-variance \(\tilde{\sigma}^2\).
refm_fit_nebin <- stan_glm(
formula = refm_fml,
family = neg_binomial_2(),
data = dat_poiss,
prior = hs(global_scale = tau0, slab_df = 100, slab_scale = 1),
### Only for the sake of speed (not recommended in general):
chains = 2, iter = 1000,
###
refresh = 0
)If a warning about a low effective sample size (ESS) appears, we
ignore it here because of the technical reasons for which we reduced
chains and iter in this vignette.
To request the latent projection with latent = TRUE, we
now need to specify more arguments (latent_ll_oscale and
latent_ppd_oscale; the internal default for
latent_ilink works correctly in this example) which will be
passed to extend_family()7:
refm_prec <- as.matrix(refm_fit_nebin)[, "reciprocal_dispersion", drop = FALSE]
latent_ll_oscale_nebin <- function(ilpreds,
dis = rep(NA, nrow(ilpreds)),
y_oscale,
wobs = rep(1, ncol(ilpreds)),
cens,
cl_ref,
wdraws_ref = rep(1, length(cl_ref))) {
y_oscale_mat <- matrix(y_oscale, nrow = nrow(ilpreds), ncol = ncol(ilpreds),
byrow = TRUE)
wobs_mat <- matrix(wobs, nrow = nrow(ilpreds), ncol = ncol(ilpreds),
byrow = TRUE)
refm_prec_agg <- cl_agg(refm_prec, cl = cl_ref, wdraws = wdraws_ref)
ll_unw <- dnbinom(y_oscale_mat, size = refm_prec_agg, mu = ilpreds, log = TRUE)
return(wobs_mat * ll_unw)
}
latent_ppd_oscale_nebin <- function(ilpreds_resamp,
dis_resamp = rep(NA, nrow(ilpreds_resamp)),
wobs = rep(1, ncol(ilpreds_resamp)),
cl_ref,
wdraws_ref = rep(1, length(cl_ref)),
idxs_prjdraws) {
refm_prec_agg <- cl_agg(refm_prec, cl = cl_ref, wdraws = wdraws_ref)
refm_prec_agg_resamp <- refm_prec_agg[idxs_prjdraws, , drop = FALSE]
ppd <- rnbinom(prod(dim(ilpreds_resamp)), size = refm_prec_agg_resamp,
mu = ilpreds_resamp)
ppd <- matrix(ppd, nrow = nrow(ilpreds_resamp), ncol = ncol(ilpreds_resamp))
return(ppd)
}
refm_nebin <- get_refmodel(refm_fit_nebin, latent = TRUE,
latent_ll_oscale = latent_ll_oscale_nebin,
latent_ppd_oscale = latent_ppd_oscale_nebin)
cvvs_nebin <- cv_varsel(
refm_nebin,
### In interactive use, we recommend not to deactivate the verbose mode:
verbose = 0
###
)Cutting off the search at size 19. Use argument `nterms_max` to customize this.
Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Warning: Some (1 / 100) Pareto k's for the reference model's PSIS-LOO weights
are > 0.67.
Warning: In the recalculation of the latent response values, some (1 / 100) expectation-specific Pareto k-values are > 0.67.
In general, we recommend K-fold CV in this case.
Again, we first inspect the plot() results to decide for
a submodel size:
Based on this plot, we decide for a submodel size of 5:
This is also the size that suggest_size() would
suggest:
[1] 5
As we can see from the predictor ranking included in the plot, our selected 5 predictor terms are exactly the truly relevant ones. More explicitly, our selected predictor terms are:
rk_nebin <- ranking(cvvs_nebin)
( predictors_final_nebin <- head(rk_nebin[["fulldata"]],
size_decided_nebin) )[1] "x.4" "x.3" "x.2" "xcat" "x.1"
Again, we will skip post-selection inference here (see the main vignette for a demonstration of post-selection inference).
This example demonstrated how the latent projection can be used for those families which are neither supported by projpred’s traditional nor by projpred’s augmented-data projection, which reflects the flexibility of the latent approach.
Response-scale analyses for models with censored responses (e.g., for
time-to-event models as occurring in survival analysis) are possible
when using the latent projection in combination with a custom
latent_ll_oscale function (see ?extend_family)
that has an attribute called cens_var and makes use of its
argument cens8.
We will illustrate this here using simulated data and right censoring, first with a Weibull response and then with a log-normal response. Both examples are adapted from a thread on The Stan Forums. Here, we use the following common simulation part:
N_surv <- 500
n_pred <- 50
n_pred_truth <- 10
n_pred_noise <- n_pred - n_pred_truth
dat_sim_surv <- matrix(rnorm(N_surv * n_pred), ncol = n_pred)
colnames(dat_sim_surv) <- paste0("x", c(paste0(".", seq_len(n_pred_truth)),
paste0("n.", seq_len(n_pred_noise))))
linpreds_surv <- -0.1 +
dat_sim_surv[, seq_len(n_pred_truth), drop = FALSE] %*%
rep_len(c(0.3, -0.2), length.out = n_pred_truth)
epreds_surv <- exp(linpreds_surv)
dat_sim_surv <- as.data.frame(dat_sim_surv)
cens_surv <- runif(N_surv,
min = quantile(epreds_surv, probs = 0.4),
max = quantile(epreds_surv, probs = 0.9))For the Weibull model, we complete our data generation as follows:
shape_weib <- 1.2
scales_weib <- epreds_surv / (gamma(1 + (1 / shape_weib)))
y_weib <- rweibull(N_surv, shape = shape_weib, scale = scales_weib)
is_event_weib <- y_weib < cens_surv
yobs_weib <- y_weib
yobs_weib[!is_event_weib] <- cens_surv[!is_event_weib]
dat_sim_weib <- data.frame(yobs = yobs_weib,
is_censored = 1 - is_event_weib,
dat_sim_surv)We now fit a reference model in brms:
refm_fit_weib <- brms::brm(
formula = yobs | cens(is_censored) ~ .,
family = brms::weibull(),
data = dat_sim_weib,
prior = brms::prior(R2D2(mean_R2 = 1/3, prec_R2 = 3)),
### Only for the sake of speed (not recommended in general):
chains = 2,
###
silent = 2,
refresh = 0
)The following code prepares the cv_varsel() run and the
downstream projpred steps:
refm_shape <- as.matrix(refm_fit_weib)[, "shape", drop = FALSE]
latent_ll_oscale_weib <- structure(function(
ilpreds,
dis = rep(NA, nrow(ilpreds)),
y_oscale,
wobs = rep(1, ncol(ilpreds)),
cens,
cl_ref,
wdraws_ref = rep(1, length(cl_ref))
) {
idxs_cens <- which(cens == 1)
idxs_event <- setdiff(seq_along(cens), idxs_cens)
wobs_mat <- matrix(wobs, nrow = nrow(ilpreds), ncol = ncol(ilpreds),
byrow = TRUE)
refm_shape_agg <- cl_agg(refm_shape, cl = cl_ref, wdraws = wdraws_ref)
ll_unw <- matrix(nrow = nrow(ilpreds), ncol = ncol(ilpreds))
for (idx_cens in idxs_cens) {
ll_unw[, idx_cens] <- pweibull(
y_oscale[idx_cens],
shape = refm_shape_agg,
scale = ilpreds[, idx_cens] / gamma(1 + 1 / as.vector(refm_shape_agg)),
lower.tail = FALSE,
log.p = TRUE
)
}
for (idx_event in idxs_event) {
ll_unw[, idx_event] <- dweibull(
y_oscale[idx_event],
shape = refm_shape_agg,
scale = ilpreds[, idx_event] / gamma(1 + 1 / as.vector(refm_shape_agg)),
log = TRUE
)
}
return(wobs_mat * ll_unw)
}, cens_var = ~ is_censored)
latent_ppd_oscale_weib <- function(
ilpreds_resamp,
dis_resamp = rep(NA, nrow(ilpreds_resamp)),
wobs = rep(1, ncol(ilpreds_resamp)),
cl_ref,
wdraws_ref = rep(1, length(cl_ref)),
idxs_prjdraws
) {
warning("The draws from this `latent_ppd_oscale` function are uncensored.")
refm_shape_agg <- cl_agg(refm_shape, cl = cl_ref, wdraws = wdraws_ref)
refm_shape_agg_resamp <- refm_shape_agg[idxs_prjdraws, , drop = FALSE]
ppd <- rweibull(
prod(dim(ilpreds_resamp)),
shape = refm_shape_agg_resamp,
scale = ilpreds_resamp / gamma(1 + 1 / as.vector(refm_shape_agg_resamp))
)
ppd <- matrix(ppd, nrow = nrow(ilpreds_resamp), ncol = ncol(ilpreds_resamp))
return(ppd)
}
refm_weib <- get_refmodel(
refm_fit_weib,
latent = TRUE,
latent_ll_oscale = latent_ll_oscale_weib,
latent_ppd_oscale = latent_ppd_oscale_weib
)The functions above use the shape draws of the full-data fit
(refm_shape). This is correct for varsel() and
for the LOO CV that we use below. For K-fold CV, see section “K-fold CV with custom response-scale
functions”.
Run cv_varsel():
cvvs_weib <- cv_varsel(
refm_weib,
### In interactive use, we recommend not to deactivate the verbose mode:
verbose = 0
)Cutting off the search at size 19. Use argument `nterms_max` to customize this.
Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Warning: Some (1 / 500) Pareto k's for the reference model's PSIS-LOO weights
are > 0.7.
Warning: In the recalculation of the latent response values, some (2 / 500) expectation-specific Pareto k-values are > 0.7.
In general, we recommend K-fold CV in this case.
As in the Poisson example, warnings about high Pareto \(\hat{k}\) values can come from the short
MCMC run of this vignette. In a real analysis, we would run more MCMC
iterations, check the Pareto \(\hat{k}\) values again, and use \(K\)-fold CV
(cv_method = "kfold") if many of them are high.
Plot the results:
Hence, the truly relevant predictors are identified correctly.
Based on this plot, we decide for a submodel size of 10:
This is not the size that suggest_size() would suggest,
but as mentioned in the main vignette and in the documentation,
suggest_size() provides only a quite heuristic decision (so
we stick with our manual decision here):
[1] 8
Project onto the submodel of this size and perform a “posterior-projection predictive check” (PPPC):
predictors_final_weib <- head(ranking(cvvs_weib)[["fulldata"]],
size_decided_weib)
prj_weib <- project(refm_weib, predictor_terms = predictors_final_weib)
prj_predict_weib <- proj_predict(prj_weib)Warning in proj$refmodel$family$latent_ppd_oscale(mu_oscale_resamp, dis_resamp
= proj$dis[draw_inds], : The draws from this `latent_ppd_oscale` function are
uncensored.
bayesplot::ppc_km_overlay(y = dat_sim_weib$yobs, yrep = prj_predict_weib,
status_y = 1 - dat_sim_weib$is_censored)Note: `extrapolation_factor` now defaults to 1.2 (20%).
To display all posterior predictive draws, set `extrapolation_factor = Inf`.
As expected, this PPPC indicates that this submodel is a reasonable one (keeping in mind that such a PPPC is just a single model-diagnostic tool).
For the log-normal model, we complete our data generation as follows:
sdlog_lnorm <- 0.3
y_lnorm <- rlnorm(N_surv, meanlog = linpreds_surv, sdlog = sdlog_lnorm)
is_event_lnorm <- y_lnorm < cens_surv
yobs_lnorm <- y_lnorm
yobs_lnorm[!is_event_lnorm] <- cens_surv[!is_event_lnorm]
dat_sim_lnorm <- data.frame(yobs = yobs_lnorm,
is_censored = 1 - is_event_lnorm,
dat_sim_surv)Again, we fit a reference model in brms:
refm_fit_lnorm <- brms::brm(
formula = yobs | cens(is_censored) ~ .,
family = brms::lognormal(),
data = dat_sim_lnorm,
prior = brms::prior(R2D2(mean_R2 = 1/3, prec_R2 = 3)),
### Only for the sake of speed (not recommended in general):
chains = 2,
###
silent = 2,
refresh = 0
)The following code prepares the cv_varsel() run and the
downstream projpred steps:
latent_ll_oscale_lnorm <- structure(function(
ilpreds,
dis = rep(NA, nrow(ilpreds)),
y_oscale,
wobs = rep(1, ncol(ilpreds)),
cens,
cl_ref,
wdraws_ref = rep(1, length(cl_ref))
) {
idxs_cens <- which(cens == 1)
idxs_event <- setdiff(seq_along(cens), idxs_cens)
wobs_mat <- matrix(wobs, nrow = nrow(ilpreds), ncol = ncol(ilpreds),
byrow = TRUE)
ll_unw <- matrix(nrow = nrow(ilpreds), ncol = ncol(ilpreds))
for (idx_cens in idxs_cens) {
ll_unw[, idx_cens] <- plnorm(
y_oscale[idx_cens],
meanlog = ilpreds[, idx_cens],
sdlog = dis,
lower.tail = FALSE,
log.p = TRUE
)
}
for (idx_event in idxs_event) {
ll_unw[, idx_event] <- dlnorm(
y_oscale[idx_event],
meanlog = ilpreds[, idx_event],
sdlog = dis,
log = TRUE
)
}
return(wobs_mat * ll_unw)
}, cens_var = ~ is_censored)
latent_ppd_oscale_lnorm <- function(
ilpreds_resamp,
dis_resamp = rep(NA, nrow(ilpreds_resamp)),
wobs = rep(1, ncol(ilpreds_resamp)),
cl_ref,
wdraws_ref = rep(1, length(cl_ref)),
idxs_prjdraws
) {
warning("The draws from this `latent_ppd_oscale` function are uncensored.")
ppd <- rlnorm(
prod(dim(ilpreds_resamp)),
meanlog = ilpreds_resamp,
sdlog = dis_resamp
)
ppd <- matrix(ppd, nrow = nrow(ilpreds_resamp), ncol = ncol(ilpreds_resamp))
return(ppd)
}
refm_lnorm <- get_refmodel(
refm_fit_lnorm,
latent = TRUE,
latent_ilink = function(lpreds, cl_ref,
wdraws_ref = rep(1, length(cl_ref))) lpreds,
latent_ll_oscale = latent_ll_oscale_lnorm,
latent_ppd_oscale = latent_ppd_oscale_lnorm,
dis = as.matrix(refm_fit_lnorm)[, "sigma", drop = FALSE]
)Argument latent_ilink returns the latent predictors
unchanged, because the two custom functions use them as
meanlog. Argument dis contains the
sigma draws of the full-data fit. For K-fold CV, see
section “K-fold CV with custom response-scale
functions”.
Run cv_varsel():
cvvs_lnorm <- cv_varsel(
refm_lnorm,
### In interactive use, we recommend not to deactivate the verbose mode:
verbose = 0
)Cutting off the search at size 19. Use argument `nterms_max` to customize this.
Warning: Some Pareto k diagnostic values are too high. See help('pareto-k-diagnostic') for details.
Warning: Some (6 / 500) Pareto k's for the reference model's PSIS-LOO weights
are > 0.7.
Warning: In the recalculation of the latent response values, some (11 / 500) expectation-specific Pareto k-values are > 0.7.
In general, we recommend K-fold CV in this case.
As in the Poisson example, warnings about high Pareto \(\hat{k}\) values can come from the short
MCMC run of this vignette. In a real analysis, we would run more MCMC
iterations, check the Pareto \(\hat{k}\) values again, and use \(K\)-fold CV
(cv_method = "kfold") if many of them are high.
Plot the results:
Hence, the truly relevant predictors are identified correctly.
Based on this plot, we decide for a submodel size of 10:
This is also the size that suggest_size() would
suggest:
[1] 10
Project onto the submodel of this size and perform a “posterior-projection predictive check” (PPPC):
predictors_final_lnorm <- head(ranking(cvvs_lnorm)[["fulldata"]],
size_decided_lnorm)
prj_lnorm <- project(refm_lnorm, predictor_terms = predictors_final_lnorm)
prj_predict_lnorm <- proj_predict(prj_lnorm)Warning in proj$refmodel$family$latent_ppd_oscale(mu_oscale_resamp, dis_resamp
= proj$dis[draw_inds], : The draws from this `latent_ppd_oscale` function are
uncensored.
bayesplot::ppc_km_overlay(y = dat_sim_lnorm$yobs, yrep = prj_predict_lnorm,
status_y = 1 - dat_sim_lnorm$is_censored)Note: `extrapolation_factor` now defaults to 1.2 (20%).
To display all posterior predictive draws, set `extrapolation_factor = Inf`.
As expected, this PPPC indicates that this submodel is a reasonable one (keeping in mind that such a PPPC is just a single model-diagnostic tool).
In the negative binomial and Weibull examples above, the custom
response-scale functions use posterior draws from the full-data
reference model fit (refm_prec, refm_shape).
In the log-normal example, argument dis contains the
sigma draws of the full-data fit. This is correct for
varsel() and for cv_varsel() with
cv_method = "LOO", because these use only the full-data
fit. With cv_method = "kfold", the reference model is
refitted in each fold, and projpred creates the
fold-wise reference model objects with the function in element
cvrefbuilder of the reference model object. The default
cvrefbuilder functions of
get_refmodel.stanreg() and
brms::get_refmodel.brmsfit() pass the same custom functions
and the same dis to the fold-wise reference model objects,
so the folds use the draws from the full-data fit, meaning that
information from the held-out observations can enter the fold-wise
predictive evaluation.
To use the draws from the fold-wise refits, define the custom
functions inside a function that takes the fitted model object as its
argument and calls get_refmodel(). Then define a function
that calls this outer function with the fold-wise refit. For an
rstanarm reference model, pass this function as
argument cvrefbuilder of get_refmodel(). In
the call inside cvrefbuilder, argument
called_from_cvrefbuilder = TRUE suppresses some warnings
that are already shown for the full-data reference model object (see
?init_refmodel). Method
brms::get_refmodel.brmsfit() does not accept a
cvrefbuilder argument. For a brms
reference model, create the reference model object first, then set its
element cvrefbuilder to this function. The default
cvrefbuilder of brms::get_refmodel.brmsfit()
also passes its arguments resp and brms_seed.
If you use these arguments, pass them in the custom
cvrefbuilder too. For the negative binomial example, this
can look as follows:
get_refmodel_nebin <- function(fit, ...) {
refm_prec <- as.matrix(fit)[, "reciprocal_dispersion", drop = FALSE]
latent_ll_oscale_nebin <- function(ilpreds,
dis = rep(NA, nrow(ilpreds)),
y_oscale,
wobs = rep(1, ncol(ilpreds)),
cens,
cl_ref,
wdraws_ref = rep(1, length(cl_ref))) {
y_oscale_mat <- matrix(y_oscale, nrow = nrow(ilpreds),
ncol = ncol(ilpreds), byrow = TRUE)
wobs_mat <- matrix(wobs, nrow = nrow(ilpreds), ncol = ncol(ilpreds),
byrow = TRUE)
refm_prec_agg <- cl_agg(refm_prec, cl = cl_ref, wdraws = wdraws_ref)
ll_unw <- dnbinom(y_oscale_mat, size = refm_prec_agg, mu = ilpreds,
log = TRUE)
return(wobs_mat * ll_unw)
}
latent_ppd_oscale_nebin <- function(
ilpreds_resamp,
dis_resamp = rep(NA, nrow(ilpreds_resamp)),
wobs = rep(1, ncol(ilpreds_resamp)),
cl_ref,
wdraws_ref = rep(1, length(cl_ref)),
idxs_prjdraws
) {
refm_prec_agg <- cl_agg(refm_prec, cl = cl_ref, wdraws = wdraws_ref)
refm_prec_agg_resamp <- refm_prec_agg[idxs_prjdraws, , drop = FALSE]
ppd <- rnbinom(prod(dim(ilpreds_resamp)), size = refm_prec_agg_resamp,
mu = ilpreds_resamp)
ppd <- matrix(ppd, nrow = nrow(ilpreds_resamp), ncol = ncol(ilpreds_resamp))
return(ppd)
}
get_refmodel(fit, latent = TRUE,
latent_ll_oscale = latent_ll_oscale_nebin,
latent_ppd_oscale = latent_ppd_oscale_nebin, ...)
}
refm_nebin_kfold <- get_refmodel_nebin(
refm_fit_nebin,
cvrefbuilder = function(cvfit) {
get_refmodel_nebin(cvfit, called_from_cvrefbuilder = TRUE)
}
)Then cv_varsel(refm_nebin_kfold, cv_method = "kfold")
uses the reciprocal_dispersion draws of the fold-wise refit
in each fold. We do not run K-fold CV here, because it refits the
reference model \(K\) times. For the
Weibull example, define refm_shape inside such a function
in the same way. For the log-normal example, pass
dis = as.matrix(fit)[, "sigma", drop = FALSE] to
get_refmodel() inside such a function.
Jørgensen (1987) himself only uses the term “exponential dispersion model”, but the discussion for that article mentions the term “ED [i.e., exponential dispersion] family”. Jørgensen (1987) also introduces the class of discrete exponential dispersion families (here abbreviated by “DED families”), see section “Example: Negative binomial distribution”.↩︎
The latent predictors are also known as the linear predictors, but “latent” is a more general term than “linear”.↩︎
If the refmodel-class object is not defined
explicitly but implicitly by a call to a top-level function such as
project(), varsel(), or
cv_varsel(), then latent = TRUE and all other
arguments related to the latent projection need to be set in
each call to a top-level function.↩︎
More precisely, the latent projection replaces the KL divergence minimization problem in the original response space by a KL divergence minimization problem in the latent space and solves the latter.↩︎
Here, “exact” means apart from approximations and simplifications which are also undertaken for the traditional projection.↩︎
The negative binomial distribution belongs to the class of discrete exponential dispersion families (Jørgensen 1987) (here abbreviated by “DED families”). DED families are closely related to ED families (Jørgensen 1987), but strictly speaking, the class of DED families is not a subset of the class of ED families. GitHub issue #361 explains why the “traditional” projection onto a DED-family submodel is currently not implemented in projpred.↩︎
The suffix _prec in refm_prec
stands for “precision” because here, we follow the Stan convention (see
the Stan documentation for the neg_binomial_2 distribution,
the brms::negbinomial() documentation, and the brms vignette
“Parameterization
of Response Distributions in brms”) and prefer the term
precision parameter for what is denoted by \(\phi\) there (confusingly, argument
size in ?stats::NegBinomial—which is the same
as \(\phi\) from the Stan notation—is
called the dispersion parameter there, although the variance is
increased by its reciprocal).↩︎
Briefly, the variable mentioned in the
cens_var right-hand side formula needs to contain the
censoring indicators (e.g., 0 = uncensored, 1
= censored) which can then be used within the custom
latent_ll_oscale function via its argument
cens.↩︎