This vignette demonstrates using the ale
package on
small datasets, as is often the case with statistical inference. You
should first read the introductory vignette that
explains general functionality of the package; this vignette goes beyond
those details to discuss issues unique to small datasets.
An obvious question is, “How small is ‘small’?” This is a very complex question and it is way beyond the scope of this vignette to try to answer it rigorously. But we can simply say that the key issue at stake is that applying the training-test split that is common in machine learning is a crucial technique for increasing the generalizability of data analysis. So, the question becomes focused to, “How small is too small for a training-test split for machine learning analysis?” The rule of thumb I am familiar with is that machine learning requires at least 200 rows of data for each predictor variable. So, for example, if you have five input variables, you would need at least 1000 rows of data. But note that this does not refer to the size of the entire dataset but to the minimum size of the training subset. So, if you carry out an 80-20 split on the full dataset (that is, 80% training set), then you would need at least 1000 rows for the training set and another 250 rows for the test set, for a minimum of 1250 rows. (And if you carry out hyperparameter tuning with cross validation on that training set, then you need even more data.) If you see where this is headed, you might quickly realize that most datasets of less than 2000 rows are probably “small”. You can see that even many datasets that are more than 2000 rows are nonetheless “small”, and so probably need the techniques mentioned in this vignette.
Most analyses use the attitude
dataset, built-in with R:
“From a survey of the clerical employees of a large financial
organization, the data are aggregated from the questionnaires of the
approximately 35 employees for each of 30 (randomly selected)
departments.” Since we’re talking about “small” datasets, we figure that
we might as well demonstrate principles with extremely small
examples.
From a survey of the clerical employees of a large financial organization, the data are aggregated from the questionnaires of the approximately 35 employees for each of 30 (randomly selected) departments. The numbers give the percent proportion of favourable responses to seven questions in each department.
A data frame with 30 observations on 7 variables. The first column are the short names from the reference, the second one the variable names in the data frame:
Variable | Type | Description | |
---|---|---|---|
Y | rating | numeric | Overall rating |
X[1] | complaints | numeric | Handling of employee complaints |
X[2] | privileges | numeric | Does not allow special privileges |
X[3] | learning | numeric | Opportunity to learn |
X[4] | raises | numeric | Raises based on performance |
X[5] | critical | numeric | Too critical |
X[6] | advance | numeric | Advancement |
Chatterjee, S. and Price, B. (1977) Regression Analysis by Example. New York: Wiley. (Section 3.7, p.68ff of 2nd ed.(1991).)
str(attitude)
#> 'data.frame': 30 obs. of 7 variables:
#> $ rating : num 43 63 71 61 81 43 58 71 72 67 ...
#> $ complaints: num 51 64 70 63 78 55 67 75 82 61 ...
#> $ privileges: num 30 51 68 45 56 49 42 50 72 45 ...
#> $ learning : num 39 54 69 47 66 44 56 55 67 47 ...
#> $ raises : num 61 63 76 54 71 54 66 70 71 62 ...
#> $ critical : num 92 73 86 84 83 49 68 66 83 80 ...
#> $ advance : num 45 47 48 35 47 34 35 41 31 41 ...
summary(attitude)
#> rating complaints privileges learning raises
#> Min. :40.00 Min. :37.0 Min. :30.00 Min. :34.00 Min. :43.00
#> 1st Qu.:58.75 1st Qu.:58.5 1st Qu.:45.00 1st Qu.:47.00 1st Qu.:58.25
#> Median :65.50 Median :65.0 Median :51.50 Median :56.50 Median :63.50
#> Mean :64.63 Mean :66.6 Mean :53.13 Mean :56.37 Mean :64.63
#> 3rd Qu.:71.75 3rd Qu.:77.0 3rd Qu.:62.50 3rd Qu.:66.75 3rd Qu.:71.00
#> Max. :85.00 Max. :90.0 Max. :83.00 Max. :75.00 Max. :88.00
#> critical advance
#> Min. :49.00 Min. :25.00
#> 1st Qu.:69.25 1st Qu.:35.00
#> Median :77.50 Median :41.00
#> Mean :74.77 Mean :42.93
#> 3rd Qu.:80.00 3rd Qu.:47.75
#> Max. :92.00 Max. :72.00
We first run ALE analysis on this dataset as if it were a valid
regular dataset, even though it is too small for a proper training-test
split. This is a small-scale demonstration mainly to demonstrate that
ale
package is valid for analyzing even small datasets, not
just the large datasets typically used for machine learning.
Ordinary least squares (OLS) regression is the most generic multivariate statistical technique. Thus, we use it as a baseline illustration to help motivate the value of ALE for interpreting the analysis of small data samples.
We train an OLS model to predict average rating:
lm_attitude <- lm(rating ~ ., data = attitude)
summary(lm_attitude)
#>
#> Call:
#> lm(formula = rating ~ ., data = attitude)
#>
#> Residuals:
#> Min 1Q Median 3Q Max
#> -10.9418 -4.3555 0.3158 5.5425 11.5990
#>
#> Coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 10.78708 11.58926 0.931 0.361634
#> complaints 0.61319 0.16098 3.809 0.000903 ***
#> privileges -0.07305 0.13572 -0.538 0.595594
#> learning 0.32033 0.16852 1.901 0.069925 .
#> raises 0.08173 0.22148 0.369 0.715480
#> critical 0.03838 0.14700 0.261 0.796334
#> advance -0.21706 0.17821 -1.218 0.235577
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Residual standard error: 7.068 on 23 degrees of freedom
#> Multiple R-squared: 0.7326, Adjusted R-squared: 0.6628
#> F-statistic: 10.5 on 6 and 23 DF, p-value: 1.24e-05
At the very least, the ale
is useful for visualizing the
effects of model variables. Note that for now, we run ale
with no bootstrapping (boot_it = 0) because small samples require a
special bootstrap approach, as explained below. For now, all we are
doing is using ALE to accurately visualize what the model estimates.
ale_lm_attitude_simple <- ale(attitude, lm_attitude, boot_it = 0)
# Print all plots
# Skip .common_data when iterating through the data for plotting
ale_lm_attitude_simple[setdiff(names(ale_lm_attitude_simple), '.common_data')] |>
purrr::map(\(.x) .x$plot) |> # extract plots as a list
gridExtra::grid.arrange(grobs = _, ncol = 2)
This visualization confirms what we see in the model coefficients above: complaints have a strong positive effect on ratings and learning has a more moderate effect. However, the ALE indicates a stronger effect of advance than the regression coefficients suggest. The other variables have relatively little effect on ratings. We will see shortly that proper bootstrapping of the model can shed some light on the discrepancies.
What is unique about ALE compared to other approaches is that it visualizes the effect of each variable irrespective of interactions that might or might not exist with other variables, whether these other interacting variables are included in the model or not.
We can use ale_ixn
to visualize the possible existence
of any interactions:
ale_lm_attitude_ixn <- ale_ixn(attitude, lm_attitude, boot_it = 0)
# Skip .common_data when iterating through the data for plotting
ale_lm_attitude_ixn[setdiff(names(ale_lm_attitude_ixn), '.common_data')] |>
purrr::walk(\(x1) { # extract list of x1 ALE outputs
purrr::map(x1, \(.x) .x$plot) |> # for each x1, extract list of x2 ALE outputs
gridExtra::grid.arrange(grobs = _, ncol = 2) # plot all x1 plots
})
This is a powerful use-case for the ale
package: it can
be used to explore the existence of interactions after the fact; they do
not need to be hypothesized beforehand. However, without bootstrapping,
such findings cannot be considered reliable. In this case, there are no
interactions in this dataset, so we will not continue exploring
them.
We have referred frequently to the importance of bootstrapping. None
of our model results, with or without ALE, should be considered reliable
without being bootstrapped. For large datasets with clear separation
between training and testing samples, ale
bootstraps the
ALE results of the test data. However, when a dataset is too small to be
subdivided into training and test sets, then the entire model should be
bootstrapped. That is, multiple models should be trained, one on each
bootstrap sample. The reliable results are the average results of all
the bootstrap models, however many there are.
The ale::model_bootstrap
function automatically carries
out full-model bootstrapping suitable for small datasets. Specifically,
it:
model_bootstrap
has two required arguments. Consistent
with tidyverse conventions, its first argument is a dataset,
data.
The second argument, model_call_string
is a character version of the full call expression for the model with
everything except for the data argument. Because
model_bootstrap
programmatically modifies the model call
and calls the model multiple times, it needs direct access to the
expression rather than to the model object, as is more typical in such
functions.
For example, the call for the OLS regression earlier was
lm(rating ~ ., data = attitude)
. To convert this to the
format required for model_call_string
, simply remove the
data argument (so, it becomes lm(rating ~ .)
) and then wrap
that in a string: 'lm(rating ~ .)'
. So, here is the full
call to model_bootstrap
:
By default, model_bootstrap
creates 100 bootstrap
samples of the provided dataset and creates 100 + 1 models on the data
(one for each bootstrap sample and then once for the original dataset).
Beyond ALE data, it also provides bootstrapped overall model statistics
(provided through broom::glance
) and bootstrapped model
coefficients (provided through broom::tidy
). Any of the
default options for broom::glance
,
broom::tidy
, and ale::ale
can be customized,
along with defaults for model_bootstrap
, such as the number
of bootstrap iterations. You can consult the help file for these
details.
model_bootstrap
returns a list with the following
elements (depending on values requested in the output
argument: * model_stats: bootstrapped results from
broom::glance
* model_coefs: bootstrapped results from
broom::tidy
* ale_data: bootstrapped ALE data and plots *
boot_data: full bootstrap data (not returned by default)
Here are the bootstrapped overall model statistics:
mb_lm$model_stats
#> # A tibble: 8 × 6
#> name conf_lo mean median conf_hi sd
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 r.squared 6.12e-1 0.767 0.788 0.877 0.0726
#> 2 adj.r.squared 5.11e-1 0.706 0.733 0.845 0.0915
#> 3 sigma 4.59e+0 6.25 6.27 7.55 0.823
#> 4 statistic 6.05e+0 14.3 14.2 27.3 5.86
#> 5 p.value 2.35e-9 0.0000730 0.000000983 0.000651 0.000261
#> 6 df 6 e+0 6 6 6 0
#> 7 df.residual 2.3 e+1 23 23 23 0
#> 8 nobs 3 e+1 30 30 30 0
Here are the bootstrapped model coefficients:
mb_lm$model_coefs
#> # A tibble: 7 × 6
#> term conf_lo mean median conf_hi std.error
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 (Intercept) -17.8 11.7 11.2 44.8 15.4
#> 2 complaints 0.347 0.632 0.648 0.903 0.141
#> 3 privileges -0.321 -0.0689 -0.0912 0.269 0.162
#> 4 learning -0.110 0.260 0.272 0.624 0.196
#> 5 raises -0.225 0.114 0.0958 0.508 0.203
#> 6 critical -0.366 0.0190 0.0397 0.320 0.194
#> 7 advance -0.683 -0.205 -0.197 0.193 0.219
Here we can visualize the results of the ALE plots.
mb_lm$ale_data[setdiff(names(mb_lm$ale_data), '.common_data')] |>
purrr::map(\(.x) .x$plot) |> # extract plots as a list
gridExtra::grid.arrange(grobs = _, ncol = 2)
The key to interpreting the effects in these models is contrasting the grey bootstrapped confidence bands surrounding the average (median) ALE effect with the thin horizontal grey band labelled ‘median \(\pm\) 2.5%’. Anything within \(\pm\) 2.5% of the median is in the 5% middle of the data. Only bootstrapped effects that are clearly beyond this middle band may be considered significant. By this criteria, considering that the median rating was 65.5%, we can conclude that:
Although this is a basic demonstration, it readily shows how crucial proper bootstrapping is to make meaningful inferences from data analysis.
A major limitation of OLS regression is that it models all relationships between the x variables and y as straight lines. But it is unlikely that all relationships are truly linear. OLS cannot accurately capture non-linear relationships.
Because the samples here are relatively small, we will use general additive models (GAM) for the modelling. To grossly oversimplify things, GAM is an extension of statistical regression analysis that lets the model fit flexible patterns in the data instead of being restricted to the best-fitting straight line. It is an ideal approach for samples that are too small for machine learning because it provides flexible curves unlike ordinary least squares regression yet will not overfit excessively as would most machine learning techniques when working with such small samples.
With GAM, the variables that we want to become flexible need to be
wrapped in the s
(smooth) function, e.g.,
s(complaints)
. For this example, we will smooth all our
numerical input variables:
gam_attitude <- mgcv::gam(rating ~ s(complaints) + s(privileges) + s(learning) +
s(raises) + s(critical) + s(advance),
data = attitude)
summary(gam_attitude)
#>
#> Family: gaussian
#> Link function: identity
#>
#> Formula:
#> rating ~ s(complaints) + s(privileges) + s(learning) + s(raises) +
#> s(critical) + s(advance)
#>
#> Parametric coefficients:
#> Estimate Std. Error t value Pr(>|t|)
#> (Intercept) 64.633 1.036 62.39 <2e-16 ***
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> Approximate significance of smooth terms:
#> edf Ref.df F p-value
#> s(complaints) 1.000 1.000 21.806 0.000146 ***
#> s(privileges) 1.000 1.000 1.527 0.230918
#> s(learning) 1.915 2.360 3.789 0.032466 *
#> s(raises) 1.000 1.000 0.171 0.683629
#> s(critical) 2.594 3.031 3.895 0.023980 *
#> s(advance) 1.000 1.000 2.403 0.136743
#> ---
#> Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#>
#> R-sq.(adj) = 0.783 Deviance explained = 84.6%
#> GCV = 47.13 Scale est. = 32.192 n = 30
By comparing the adjusted R2 of the OLS model (0.663) with that of the GAM model (0.783), we can readily see that the GAM model provides a superior fit to the data.
To understand which variables were responsible for this relationship, the results for the smooth terms in GAM are not readily interpretable. They need to be visualized for effective interpretation—ALE is perfect for such purposes.
ale_gam_attitude_simple <- ale(attitude, gam_attitude, boot_it = 0)
ale_gam_attitude_simple[setdiff(names(ale_gam_attitude_simple), '.common_data')] |>
purrr::map(\(.x) .x$plot) |> # extract plots as a list
gridExtra::grid.arrange(grobs = _, ncol = 2)
Compared to the OLS results above, the GAM results provide quite a surprise concerning the shape of the effect of employees’ perceptions that their department is too critical–it seems that both low criticism and very high criticism negatively affect ratings. However, before trying to interpret these results, we must remember that results that are not bootstrapped are simply not reliable. So, let us see what bootstrapping will give us.
mb_gam <- model_bootstrap(
attitude,
'mgcv::gam(rating ~ s(complaints) + s(privileges) + s(learning) +
s(raises) + s(critical) + s(advance))'
)
mb_gam$model_stats
#> # A tibble: 3 × 6
#> name conf_lo mean median conf_hi sd
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 df 16.0 19.0 19.0 22.0 1.66
#> 2 df.residual 8.00 11.0 11.0 14.0 1.66
#> 3 nobs 30 30 30 30 0
mb_gam$model_coefs
#> # A tibble: 6 × 6
#> term conf_lo mean median conf_hi std.error
#> <chr> <dbl> <dbl> <dbl> <dbl> <dbl>
#> 1 s(complaints) 1.00 4.24 3.36 8.90 3.10
#> 2 s(privileges) 1.00 3.90 2.72 8.91 2.98
#> 3 s(learning) 1.00 2.82 1.08 8.71 2.61
#> 4 s(raises) 1.00 3.26 1.43 8.93 2.93
#> 5 s(critical) 1.00 1.80 1.03 7.14 1.53
#> 6 s(advance) 1.00 2.00 1.04 6.30 1.60
mb_gam$ale_data[setdiff(names(mb_gam$ale_data), '.common_data')] |>
purrr::map(\(.x) .x$plot) |> # extract plots as a list
gridExtra::grid.arrange(grobs = _, ncol = 2)
The bootstrapped GAM results tell a rather different story from the OLS results. In this case, the bootstrap confidence bands of all the variables (even of complaints) fully overlap with the entirety of the median non-significance region. Even the average slopes have vanished from all variables except for complaint, where it remains positive, yet insignificant because of the wide confidence interval.
So, what should we conclude? First, it is tempting to retain the OLS results because they tell a more interesting story. But we consider that this would be irresponsible since the GAM model is clearly superior in terms of adjusted R2: it is the model that far more reliably tells us what is really going on. And what does it tell us?
No doubt, the inconclusive results are because the dataset is so small (only 30 rows). A dataset even double that size might show significant effects at least for complaints, if not for other variables.