Introduction

TMoE (t Mixture-of-Experts) provides a flexible and robust modelling framework for heterogenous data with possibly heavy-tailed distributions and corrupted by atypical observations. TMoE consists of a mixture of K t expert regressors network (of degree p) gated by a softmax gating network (of degree q) and is represented by:

The gating network parameters alpha’s of the softmax net.
The experts network parameters: The location parameters (regression coefficients) beta’s, scale parameters sigma’s, and the degree of freedom (robustness) parameters nu’s. TMoE thus generalises mixtures of (normal, t, and) distributions and mixtures of regressions with these distributions. For example, when \(q=0\), we retrieve mixtures of (t-, or normal) regressions, and when both \(p=0\) and \(q=0\), it is a mixture of (t-, or normal) distributions. It also reduces to the standard (normal, t) distribution when we only use a single expert (\(K=1\)).

Model estimation/learning is performed by a dedicated expectation conditional maximization (ECM) algorithm by maximizing the observed data log-likelihood. We provide simulated examples to illustrate the use of the model in model-based clustering of heterogeneous regression data and in fitting non-linear regression functions.

It was written in R Markdown, using the knitr package for production.

See help(package="meteorits") for further details and references provided by citation("meteorits").

Application to a simulated dataset

Generate sample

n <- 500 # Size of the sample
alphak <- matrix(c(0, 8), ncol = 1) # Parameters of the gating network
betak <- matrix(c(0, -2.5, 0, 2.5), ncol = 2) # Regression coefficients of the experts
sigmak <- c(0.5, 0.5) # Standard deviations of the experts
nuk <- c(5, 7) # Degrees of freedom of the experts network t densities
x <- seq.int(from = -1, to = 1, length.out = n) # Inputs (predictors)

# Generate sample of size n
sample <- sampleUnivTMoE(alphak = alphak, betak = betak, sigmak = sigmak, 
                         nuk = nuk, x = x)
y <- sample$y

Set up tMoE model parameters

K <- 2 # Number of regressors/experts
p <- 1 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-5
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

tmoe <- emTMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
               threshold, verbose, verbose_IRLS)
## EM - tMoE: Iteration: 1 | log-likelihood: -529.945288794032
## EM - tMoE: Iteration: 2 | log-likelihood: -527.348101613744
## EM - tMoE: Iteration: 3 | log-likelihood: -526.697258494959
## EM - tMoE: Iteration: 4 | log-likelihood: -526.100920519297
## EM - tMoE: Iteration: 5 | log-likelihood: -525.530707141445
## EM - tMoE: Iteration: 6 | log-likelihood: -525.004975573049
## EM - tMoE: Iteration: 7 | log-likelihood: -524.537304371765
## EM - tMoE: Iteration: 8 | log-likelihood: -524.134230296182
## EM - tMoE: Iteration: 9 | log-likelihood: -523.796173749721
## EM - tMoE: Iteration: 10 | log-likelihood: -523.519134936736
## EM - tMoE: Iteration: 11 | log-likelihood: -523.296473104273
## EM - tMoE: Iteration: 12 | log-likelihood: -523.120395143099
## EM - tMoE: Iteration: 13 | log-likelihood: -522.983018797515
## EM - tMoE: Iteration: 14 | log-likelihood: -522.877027662562
## EM - tMoE: Iteration: 15 | log-likelihood: -522.796003916234
## EM - tMoE: Iteration: 16 | log-likelihood: -522.734538519799
## EM - tMoE: Iteration: 17 | log-likelihood: -522.68820514474
## EM - tMoE: Iteration: 18 | log-likelihood: -522.653461853027
## EM - tMoE: Iteration: 19 | log-likelihood: -522.627523155938
## EM - tMoE: Iteration: 20 | log-likelihood: -522.608228167519
## EM - tMoE: Iteration: 21 | log-likelihood: -522.593918674577
## EM - tMoE: Iteration: 22 | log-likelihood: -522.583333279152
## EM - tMoE: Iteration: 23 | log-likelihood: -522.57551921559
## EM - tMoE: Iteration: 24 | log-likelihood: -522.569760986911
## EM - tMoE: Iteration: 25 | log-likelihood: -522.565523823543

Summary

tmoe$summary()
## -------------------------------------
## Fitted t Mixture-of-Experts model
## -------------------------------------
## 
## tMoE model with K = 2 experts:
## 
##  log-likelihood df       AIC       BIC       ICL
##       -522.5655 10 -532.5655 -553.6386 -553.6456
## 
## Clustering table (Number of observations in each expert):
## 
##   1   2 
## 249 251 
## 
## Regression coefficients:
## 
##     Beta(k = 1) Beta(k = 2)
## 1    0.01321746   0.2258488
## X^1  2.55858529  -2.8607695
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2)
##      0.2821912     0.4560227

Plots

Mean curve

tmoe$plot(what = "meancurve")

Confidence regions

tmoe$plot(what = "confregions")

Clusters

tmoe$plot(what = "clusters")

Log-likelihood

tmoe$plot(what = "loglikelihood")

Application to a real dataset

Load data

library(MASS)
data("mcycle")
x <- mcycle$times
y <- mcycle$accel

Set up tMoE model parameters

K <- 4 # Number of regressors/experts
p <- 2 # Order of the polynomial regression (regressors/experts)
q <- 1 # Order of the logistic regression (gating network)

Set up EM parameters

n_tries <- 1
max_iter <- 1500
threshold <- 1e-5
verbose <- TRUE
verbose_IRLS <- FALSE

Estimation

tmoe <- emTMoE(X = x, Y = y, K, p, q, n_tries, max_iter, 
               threshold, verbose, verbose_IRLS)
## EM - tMoE: Iteration: 1 | log-likelihood: -607.963023404096
## EM - tMoE: Iteration: 2 | log-likelihood: -603.530462450757
## EM - tMoE: Iteration: 3 | log-likelihood: -600.936924880401
## EM - tMoE: Iteration: 4 | log-likelihood: -597.134488483045
## EM - tMoE: Iteration: 5 | log-likelihood: -587.345068256529
## EM - tMoE: Iteration: 6 | log-likelihood: -579.214908026282
## EM - tMoE: Iteration: 7 | log-likelihood: -575.789415000413
## EM - tMoE: Iteration: 8 | log-likelihood: -574.743067640861
## EM - tMoE: Iteration: 9 | log-likelihood: -573.824355322963
## EM - tMoE: Iteration: 10 | log-likelihood: -572.704434066976
## EM - tMoE: Iteration: 11 | log-likelihood: -571.456957238225
## EM - tMoE: Iteration: 12 | log-likelihood: -570.311351216642
## EM - tMoE: Iteration: 13 | log-likelihood: -569.295154539063
## EM - tMoE: Iteration: 14 | log-likelihood: -568.319884936335
## EM - tMoE: Iteration: 15 | log-likelihood: -567.372601882285
## EM - tMoE: Iteration: 16 | log-likelihood: -566.478212031608
## EM - tMoE: Iteration: 17 | log-likelihood: -565.659310717143
## EM - tMoE: Iteration: 18 | log-likelihood: -564.901676479101
## EM - tMoE: Iteration: 19 | log-likelihood: -564.155447286696
## EM - tMoE: Iteration: 20 | log-likelihood: -563.446620149915
## EM - tMoE: Iteration: 21 | log-likelihood: -562.937110761917
## EM - tMoE: Iteration: 22 | log-likelihood: -562.667086966818
## EM - tMoE: Iteration: 23 | log-likelihood: -562.520110670808
## EM - tMoE: Iteration: 24 | log-likelihood: -562.426801842479
## EM - tMoE: Iteration: 25 | log-likelihood: -562.361727752526
## EM - tMoE: Iteration: 26 | log-likelihood: -562.314424482959
## EM - tMoE: Iteration: 27 | log-likelihood: -562.279516472013
## EM - tMoE: Iteration: 28 | log-likelihood: -562.253254369394
## EM - tMoE: Iteration: 29 | log-likelihood: -562.233295182051
## EM - tMoE: Iteration: 30 | log-likelihood: -562.217975445467
## EM - tMoE: Iteration: 31 | log-likelihood: -562.206116173187
## EM - tMoE: Iteration: 32 | log-likelihood: -562.196865909629
## EM - tMoE: Iteration: 33 | log-likelihood: -562.189597689509
## EM - tMoE: Iteration: 34 | log-likelihood: -562.183858578194
## EM - tMoE: Iteration: 35 | log-likelihood: -562.179299411545

Summary

tmoe$summary()
## -------------------------------------
## Fitted t Mixture-of-Experts model
## -------------------------------------
## 
## tMoE model with K = 4 experts:
## 
##  log-likelihood df       AIC       BIC       ICL
##       -562.1793 26 -588.1793 -625.7538 -625.7472
## 
## Clustering table (Number of observations in each expert):
## 
##  1  2  3  4 
## 28 37 32 36 
## 
## Regression coefficients:
## 
##     Beta(k = 1) Beta(k = 2)  Beta(k = 3) Beta(k = 4)
## 1    -1.0422893 1008.728925 -2132.506787 654.7349946
## X^1  -0.1089089 -105.713093   135.481456 -27.8267024
## X^2  -0.0079480    2.481934    -2.112076   0.2888205
## 
## Variances:
## 
##  Sigma2(k = 1) Sigma2(k = 2) Sigma2(k = 3) Sigma2(k = 4)
##       1.596783      440.1084      473.8641       30.5968

Plots

Mean curve

tmoe$plot(what = "meancurve")

Confidence regions

tmoe$plot(what = "confregions")

Clusters

tmoe$plot(what = "clusters")

Log-likelihood

tmoe$plot(what = "loglikelihood")

A-quick-tour-of-tMoE

Introduction

Application to a simulated dataset

Generate sample

Set up tMoE model parameters

Set up EM parameters

Estimation

Summary

Plots

Mean curve

Confidence regions

Clusters

Log-likelihood

Application to a real dataset

Load data

Set up tMoE model parameters

Set up EM parameters

Estimation

Summary

Plots

Mean curve

Confidence regions

Clusters

Log-likelihood