Prediction pools and MFF aggregation

External regression predictions

MFF aggregates numeric predictions without requiring models to be trained inside the package. The bundled mff_diamonds data contain validation and test predictions of diamond prices from seven regression learners: four linear (least squares, lasso, ridge, elastic net) and three tree ensembles (random forest, XGBoost, LightGBM). A random sample of 2000 diamonds from the ggplot2 diamonds data was randomly split into 1000 training, 500 validation and 500 test rows, and the learners were trained on the training rows only (see ?mff_diamonds). mff_example() builds these predictions into a pool. The example demonstrates the interface; it does not estimate general predictive superiority.

pool <- mff_example()
pool
#> MFF prediction pool
#>   Task: regression | Pool: heterogeneous 
#>   Candidates: 7 | Validation: 500 | Test: 500
dim(validation_predictions(pool))
#> [1] 500   7
candidate_info(pool)
#>       candidate        family
#> 1            lm        linear
#> 2         lasso        linear
#> 3         ridge        linear
#> 4   elastic_net        linear
#> 5 random_forest tree_ensemble
#> 6       xgboost tree_ensemble
#> 7      lightgbm tree_ensemble

A fixed configuration and a grid search share the same fitted-object contract.

fixed <- fit_mff(pool, clusters = 2, method = "fcm", seed = 19)
fit <- tune_mff(pool, clusters = 2:3, method = "fcm", m = c(1.5, 2),
                nstart = 3, reltol = 1e-6, control = mff_control(seed = 19))
print(fit)
#> Tuned MFF fit
#>   Method: fcm | Candidates: 7 | Functions: 3 
#>   Validation-selected function: 1 | RMSE = 1006.745 
#>   Convergence status: converged | Iterations: 10 
#>   Configurations: 4 | Successful: 4
summary(fit)
#> MFF fit summary
#> Configuration:
#>   method      : fcm
#>   metric      : RMSE
#>   clusters    : 3
#>   m           : 2
#>   nstart      : 3
#>   iter_max    : 100
#>   reltol      : 1e-06
#>   standardize : FALSE
#>   reduction   : none
#>   Convergence status: converged | Iterations: 10 
#> Selected function: 1 
#> Selected weights:
#>            lm         lasso         ridge   elastic_net random_forest 
#>   0.003028213   0.001572850   0.085995441   0.001584017   0.003038848 
#>       xgboost      lightgbm 
#>   0.456845200   0.447935430 
#> Validation scores:
#>                 MAE     RMSE     MAPE    SMAPE     MSE    MedAE
#> function_1 485.0458 1006.745 12.90416 12.51245 1013536 178.3300
#> function_2 771.9162 1130.517 43.02233 41.64732 1278068 556.7292
#> function_3 643.9853 1189.650 22.41862 19.08970 1415266 314.0998
#> Search: 4 configurations; 0 failed
selected_configuration(fit)
#>   method      : fcm
#>   metric      : RMSE
#>   clusters    : 3
#>   m           : 2
#>   nstart      : 3
#>   iter_max    : 100
#>   reltol      : 1e-06
#>   standardize : FALSE
#>   reduction   : none
search_results(fit)
#>     m clusters seed success     loss elapsed message
#> 1 1.5        2   20    TRUE 1056.685   0.003        
#> 2 2.0        2   21    TRUE 1042.631   0.001        
#> 3 1.5        3   22    TRUE 1019.745   0.002        
#> 4 2.0        3   23    TRUE 1006.745   0.001

Test outcomes are used only after fitting. Changing them cannot change weights or the selected configuration. The following compares MFF with equal averaging and the candidate chosen solely on validation RMSE.

pv <- validation_predictions(pool)
pt <- test_predictions(pool)
best <- which.min(evaluate(pv, validation_observed(pool))[, "RMSE"])
comparisons <- cbind(
  MFF = predict(fit, pool),
  Mean = rowMeans(pt),
  Validation_best = pt[, best]
)
evaluate(comparisons, test_observed(pool))
#>                      MAE      RMSE     MAPE    SMAPE       MSE    MedAE
#> MFF             483.6754  911.1981 13.17991 12.71691  830282.0 188.7307
#> Mean            616.3507 1032.5433 24.42174 25.44028 1066145.6 370.0113
#> Validation_best 479.6820  924.4773 12.19211 11.72586  854658.2 177.1585

The heatmap exposes all candidate weights; prediction defaults to the function selected on validation data. type = "all" means the functions in the chosen configuration, not every configuration in the tuning grid.

plot(fit, type = "weight_heatmap")

plot(fit, type = "observed_predicted", newdata = pool)

Bootstrap candidates

Explicit split rows make the training stage auditable. Bootstrap resampling uses training rows only. Validation/test rows are identical for each candidate.

split <- mff_split(train = 1:15, validation = 16:23, test = 24:31)
boot_pool <- boot_train(Volume ~ Girth + Height, datasets::trees, split,
                        times = 8, control = mff_control(seed = 44))
summary(boot_pool)
#> MFF prediction pool summary
#>   Task: regression | Pool type: bootstrap 
#>   Candidates: 8 | Validation rows: 8 | Test rows: 8 
#> Candidates:
#>    candidate algorithm seed
#>  bootstrap_1        lm   45
#>  bootstrap_2        lm   46
#>  bootstrap_3        lm   47
#>  bootstrap_4        lm   48
#>  bootstrap_5        lm   49
#>  bootstrap_6        lm   50
#>  bootstrap_7        lm   51
#>  bootstrap_8        lm   52
boot_fit <- tune_mff(boot_pool, clusters = 2:3, method = "kmeans")
evaluate(predict(boot_fit, boot_pool), test_observed(boot_pool))
#>           MAE     RMSE     MAPE    SMAPE     MSE   MedAE
#> [1,] 12.07627 13.57597 21.17138 24.03179 184.307 11.5067

User-defined learners

Any regression method can be trained by model_train() and boot_train() after wrapping it with mff_learner(). The fit function receives the model matrix built on the training rows and the training response; predict returns one prediction per row. Named lists of learners let one algorithm enter several times with different controls.

ppr_learner <- mff_learner(
  fit = function(x, y, nterms = 2) stats::ppr(x, y, nterms = nterms),
  predict = function(object, newx) stats::predict(object, newx),
  name = "ppr"
)
air <- datasets::airquality[stats::complete.cases(datasets::airquality), ]
set.seed(11)
air_rows <- sample(nrow(air))
air_split <- mff_split(air_rows[1:60], air_rows[61:85], air_rows[86:111])
air_pool <- model_train(Ozone ~ Solar.R + Wind + Temp, air, air_split,
                        learners = list(lm = "lm", ppr_1 = ppr_learner,
                                        ppr_2 = ppr_learner,
                                        ppr_3 = ppr_learner),
                        controls = list(ppr_1 = list(nterms = 1),
                                        ppr_3 = list(nterms = 3)),
                        seed = 11)
candidate_info(air_pool)
#>   candidate algorithm
#> 1        lm        lm
#> 2     ppr_1       ppr
#> 3     ppr_2       ppr
#> 4     ppr_3       ppr
boot_ppr <- boot_train(Ozone ~ Solar.R + Wind + Temp, air, air_split,
                       times = 5, learner = ppr_learner,
                       control = mff_control(seed = 11))
candidate_info(boot_ppr)
#>     candidate algorithm seed
#> 1 bootstrap_1       ppr   12
#> 2 bootstrap_2       ppr   13
#> 3 bootstrap_3       ppr   14
#> 4 bootstrap_4       ppr   15
#> 5 bootstrap_5       ppr   16

Cross-validated candidates

A separate validation set is not required. With resampling = mff_cv(), model_train() predicts every training row from candidates fitted on the other folds, and these out-of-fold predictions serve as validation predictions. Model-matrix encoding and learner tuning are repeated inside each fold using that fold’s training rows only. Test predictions come from candidates refitted on all training rows, and test rows never enter the folds.

set.seed(7)
rows <- sample(nrow(datasets::quakes))
cv_split <- mff_split(train = rows[1:750], validation = NULL,
                      test = rows[751:1000])
cv_pool <- model_train(mag ~ lat + long + depth + stations, datasets::quakes,
                       cv_split, learners = c("lm", "ridge", "random_forest"),
                       resampling = mff_cv(folds = 5), seed = 7)
cv_pool
#> MFF prediction pool
#>   Task: regression | Pool: heterogeneous 
#>   Candidates: 3 | Validation: 750 | Test: 250
table(pool_info(cv_pool)$metadata$resampling$fold_id)
#> 
#>   1   2   3   4   5 
#> 150 150 150 150 150
cv_fit <- tune_mff(cv_pool, clusters = 2, method = "fcm", m = c(1.5, 2))
weights(cv_fit)
#>            lm         ridge random_forest 
#>  5.006733e-01  4.993267e-01  4.811049e-08
evaluate(predict(cv_fit, cv_pool), test_observed(cv_pool))
#>            MAE      RMSE     MAPE    SMAPE        MSE     MedAE
#> [1,] 0.1532053 0.1985064 3.295108 3.314259 0.03940481 0.1200164

mff_cv(fold_id = ) accepts user-defined folds, for example to keep grouped observations together. A factor level that occurs only in a held-out fold stops with an error that names the fold.

Pools from caret and tidymodels

Models tuned in other frameworks already carry out-of-fold predictions. as_mff_pool() converts a named list of caret::train() models (or a caretEnsemble caretList), tidymodels tune_grid() results and stacks data stacks into pools. With newdata, test predictions come from the framework’s own models fitted on all training rows.

set.seed(3)
car_data <- datasets::mtcars
ctrl <- caret::trainControl(method = "cv", savePredictions = "final",
                            index = caret::createFolds(car_data$mpg[1:26], k = 4,
                                                       returnTrain = TRUE))
caret_models <- list(
  linear = caret::train(mpg ~ wt + hp + disp, car_data[1:26, ], method = "lm",
                        trControl = ctrl),
  knn = caret::train(mpg ~ wt + hp + disp, car_data[1:26, ], method = "knn",
                     trControl = ctrl, tuneLength = 2),
  weight_only = caret::train(mpg ~ wt, car_data[1:26, ], method = "lm",
                             trControl = ctrl)
)
#> Zorunlu paket yükleniyor: ggplot2
#> Zorunlu paket yükleniyor: lattice
caret_pool <- as_mff_pool(caret_models, newdata = car_data[27:32, ],
                          test_observed = car_data$mpg[27:32])
candidate_info(caret_pool)
#>     candidate method         tuning
#> 1      linear     lm intercept=TRUE
#> 2         knn    knn            k=5
#> 3 weight_only     lm intercept=TRUE
car_data <- datasets::mtcars
set.seed(4)
folds <- rsample::vfold_cv(car_data[1:26, ], v = 4)
lasso <- workflows::workflow(
  mpg ~ wt + hp + disp,
  parsnip::set_engine(parsnip::linear_reg(penalty = tune::tune(), mixture = 1),
                      "glmnet"))
tuned <- tune::tune_grid(lasso, folds,
                         grid = data.frame(penalty = c(0.01, 0.3, 1, 3)),
                         control = tune::control_grid(save_pred = TRUE,
                                                      save_workflow = TRUE))
tune_pool <- as_mff_pool(tuned, newdata = car_data[27:32, ],
                         test_observed = car_data$mpg[27:32])
candidate_info(tune_pool)
#>         candidate penalty
#> 1 pre0_mod1_post0    0.01
#> 2 pre0_mod2_post0    0.30
#> 3 pre0_mod3_post0    1.00
#> 4 pre0_mod4_post0    3.00

For heterogeneous learner families, model_train() can use an explicit selection such as learners = c("lm", "random_forest"). The requested optional learner packages must be installed. A single "lm" candidate is allowed as a baseline, although fitting an MFF needs at least two candidate columns.

Configurations of a single algorithm

Here polynomial degree is varied outside MFF; each fitted regression produces one candidate column. The candidate table retains its degree for inspection. The same approach applies to external XGBoost, SVM or forecasting settings.

dat <- datasets::trees
degrees <- 1:3
models <- lapply(degrees, function(d) {
  stats::lm(Volume ~ poly(Girth, d), data = dat[1:15, ])
})
make_predictions <- function(rows) {
  out <- vapply(models, function(model) {
    as.numeric(predict(model, newdata = dat[rows, ]))
  }, numeric(length(rows)))
  colnames(out) <- paste0("degree_", degrees)
  out
}
hp <- mff_pool(
  make_predictions(16:23), dat$Volume[16:23],
  make_predictions(24:31), dat$Volume[24:31],
  candidate_info = data.frame(candidate = paste0("degree_", degrees),
                               algorithm = "polynomial_lm", degree = degrees),
  pool_type = "hyperparameter"
)
candidate_info(hp)
#>   candidate     algorithm degree
#> 1  degree_1 polynomial_lm      1
#> 2  degree_2 polynomial_lm      2
#> 3  degree_3 polynomial_lm      3
hp_fit <- tune_mff(hp, clusters = 2:3, method = "fcm", m = c(1.5, 2))
evaluate(predict(hp_fit, hp), test_observed(hp))
#>           MAE     RMSE     MAPE   SMAPE      MSE    MedAE
#> [1,] 10.93218 12.79735 18.77613 21.2117 163.7723 10.63079

Methods and diagnostics

The supported membership methods are fuzzy c-means (fcm), possibilistic fuzzy c-means (pfcm), Gustafson-Kessel (gk) and crisp k-means (kmeans). For every new-interface method, nstart selects the start with the lowest clustering objective. Validation outcomes subsequently select a meta function.

The optional reltol argument controls convergence. For FCM it is the relative objective tolerance. In PFCM it bounds the mean absolute change of the prototype coordinates; in GK the largest change of a membership between iterations. These are different stopping rules. NULL retains the defaults: approximately 1.49e-8 for FCM, 1e-5 for PFCM and 1e-3 for GK. PFCM follows Pal, Pal, Keller and Bezdek (2005) with their protocol (a = b = 1, penalties from a terminal FCM partition, at most 100 iterations). K-means accepts only reltol = NULL; use iter_max there. The effective tolerance is recorded in selected_configuration() and fit_diagnostics(). Increasing tolerance may reduce iterations, but can change the fitted weights and predictions.

GK estimates one fuzzy covariance matrix per cluster. With few candidates and many validation observations these matrices are singular, so MFF implements the modified GK algorithm of Babuska, van der Veen and Kaymak (2002): the ratio between the largest and smallest eigenvalue of each matrix is limited to beta (default 1e15), and gamma (default 0) adds a scaled identity matrix. tune_mff() can search gamma like m. reduction = "pca" retains every nonzero principal component in clustering space. This preserves Euclidean distances but does not guarantee that GK or PFCM results are unchanged. Nonfinite memberships fail explicitly rather than being replaced by uniform weights.

fit_diagnostics(fit)
#> $selected_start
#> [1] 2
#> 
#> $objective
#> [1] 21195166
#> 
#> $iterations
#> [1] 10
#> 
#> $tolerance
#> $tolerance$value
#> [1] 1e-06
#> 
#> $tolerance$parameter
#> [1] "control$reltol"
#> 
#> $tolerance$criterion
#> [1] "relative_objective_change"
#> 
#> 
#> $convergence
#> [1] "converged"
#> 
#> $starts
#>   start objective success message
#> 1     1  24862060    TRUE        
#> 2     2  21195166    TRUE        
#> 3     3  24862052    TRUE        
#> 
#> $preprocessing
#> $preprocessing$method
#> [1] "none"
#> 
#> $preprocessing$original_dimensions
#> [1] 500
#> 
#> $preprocessing$retained_dimensions
#> [1] 500
#> 
#> $preprocessing$distance_preserving
#> [1] TRUE
#> 
#> 
#> $seed
#> [1] 23
weights(fit, type = "all")
#>                function_1   function_2  function_3
#> lm            0.003028213 0.2704951651 0.003489439
#> lasso         0.001572850 0.2719100403 0.001818082
#> ridge         0.085995441 0.1588521724 0.177951374
#> elastic_net   0.001584017 0.2718990153 0.001831337
#> random_forest 0.003038848 0.0007193028 0.692905267
#> xgboost       0.456845200 0.0124026466 0.056735672
#> lightgbm      0.447935430 0.0137216577 0.065268830

The selection metric can be MAE, RMSE, MAPE, SMAPE, MSE or MedAE. MAPE and SMAPE use a small epsilon in the denominator; MAPE can be unstable for responses near zero. See ?evaluate for formulas. Tuning diagnostics record failed configurations and warnings. The convergence element of fit_diagnostics() is "converged" only when the selected start met its stopping rule before iter_max without warnings; "iteration_limit" and "backend_warning" mean convergence is not confirmed.

For reproducible socket parallelism use mff_control(seed = 19, parallel = TRUE, workers = 2). The default is sequential.

mirror server hosted at Truenetwork, Russian Federation.