Introduction

Statement of the problem from the customer’s perspective

History of the problem, previous results

Exploratory data analysis

• Head of the data

  • Discuss the characteristics of each feature.

• Barchart of target (0 or 1) vs each feature, by percent (%)

  • Discussion of y vs target variables

• Boxplots of the numeric data (insert plot here)

  • Discussion of boxplots of the numeric data

• Histograms of each numeric column (insert plot here)

  • Discussion of histograms of each numeric column

• Data summary (insert table here)

  • Discussion of the data summary

• Outliers in the data (insert outliers data here)

  • Discussion of outliers in the data

• Correlation of the data (table)

• Correlation plot of the numeric data as circles and colors

• Correlation of the ensemble

• Variance Inflation Factor

• The stories in the exploratory data analysis

24 logistic models (Individual models then ensembles, in alphabetical order)

One paragraph summary about statistical modeling here

• Cubist

  cubist_train_fit <- Cubist::cubist(x = as.data.frame(train), y = train$y)

• Flexible Discriminant Analysis

  fda_train_fit <- MachineShop::fit(as.factor(y) ~ ., data = train01, model = “FDAModel”)

• GAM (Generalized Additive Models) (uses smoothing splines)

  f2 <- stats::as.formula(paste0(“y ~”, paste0(“gam::s(”, names_df, “)”, collapse = “+”)))

  gam_train_fit <- gam(f2, data = train1)

• Generalized Linear Models

  glm_train_fit <- stats::glm(y ~ ., data = train, family = binomial)

• Lasso (uses best model)

  best_lasso_lambda <- lasso_cv$lambda.min

  best_lasso_model <- glmnet(x, y, alpha = 1, lambda = best_lasso_lambda)

• Linear (tuned)

  linear_train_fit <- e1071::tune.rpart(formula = y ~ ., data = train)

• Linear Discriminant Analysis

  lda_train_fit <- MASS::lda(as.factor(y) ~ ., data = train01, model = “LMModel”)

• Penalized Discriminant Analysis

  pda_train_fit <- MachineShop::fit(as.factor(y) ~ ., data = train01, model = “PDAModel”)

• Quadratic Discriminant Analysis

  qda_train_fit <- MASS::qda(as.factor(y) ~ ., data = train01)

• Random Forest

  rf_train_fit <- randomForest(x = train, y = as.factor(y_train), data = df, family = binomial(link = “logit”))

• Ridge

  best_ridge_lambda <- ridge_cv$lambda.min

  best_ridge_model <- glmnet(x, y, alpha = 0, lambda = best_ridge_lambda)

• RPart

  rpart_train_fit <- rpart::rpart(train$y ~ ., data = train)

• SVM (Support Vector Machines) (tuned)

  svm_train_fit <- e1071::tune.svm(x = train, y = train$y, data = train)

• Tree

  tree_train_fit <- tree::tree(train$y ~ ., data = train)

  Ensemble models start here

• Ensemble Gradient Boosted

  ensemble_gb_train_fit <- gbm::gbm(ensemble_train$y_ensemble ~ ., data = ensemble_train, distribution = “gaussian”, n.trees = 100, shrinkage = 0.1, interaction.depth = 10 )

• Ensemble Lasso (uses best model)

  ensemble_best_lasso_lambda <- ensemble_lasso_cv$lambda.min

  ensemble_best_lasso_model <- glmnet(ensemble_x, ensemble_y, alpha = 1, lambda = ensemble_best_lasso_lambda)

• Ensemble Partial Least Squares

  ensemble_pls_train_fit <- MachineShop::fit(as.factor(y) ~ ., data = ensemble_train, model = “PLSModel”)

• Ensemble Penalized Discriminant Analysis

  ensemble_pda_train_fit <- MachineShop::fit(as.factor(y) ~ ., data = ensemble_train, model = “PDAModel”)

• Ensemble Ridge

  x = model.matrix(y ~ ., data = ensemble_train)[, -1]

  y = ensemble_train$y

  ensemble_ridge_train_fit <- glmnet::glmnet(x, y, alpha = 0)

• Ensemble RPart

  ensemble_rpart_train_fit <- MachineShop::fit(as.factor(y) ~ ., data = ensemble_train, model = “RPartModel”)

• Ensemble Support Vector Machines (SVM)

  ensemble_svm_train_fit <- e1071::svm(as.factor(y) ~ ., data = ensemble_train, kernel = “radial”, gamma = 1, cost = 1)

• Ensemble Trees

  ensemble_tree_train_fit <- tree::tree(ensemble_train$y ~ ., data = ensemble_train)

• The stories in the models (fill in here)

Ensembles and individual model plots

• Negative predictive value (fixed scales)

• Negative predictive value (free scales)

• Positive predictive value (fixed scales)

• Positive predictive value (free scales)

• F1 Score (fixed scales)

• F1 Score (free scales)

• False negative rate (fixed scales)

• False negative rate (free scales)

• False positive rate (fixed scales)

• False positive rate (free scales)

• True negative rate (fixed scales)

• True negative rate (free scales)

• True positive rate (fixed scales)

• True positive rate (free scales)

• ROC Curves for each of the 24 models

• Over or under fitting (closer to 1 is better) barchart

• Duration (mean) by model barchart

• Overfitting by model and resample, fixed scales

• Overfitting by model and resample, free scales

• Model accuracy bar chart

• Accuracy by model and resample, including train and holdout by each resample, fixed scales

• Accuracy by model and resample, including train and holdout by each resample, free scales

• Summary report

  • Accuracy (mean)

  • Accuracy (standard deviation)

  • True positive rate (also known as sensitivity)

  • True negative rate (also known as specificity)

  • False positive rate (also known as Type I error)

  • False negative rate (also known as Type II error)

  • Positive predictive value

  • Negative predictive value

  • F1 score

  • Area under the curve (AUC)

  • Overfitting (mean)

  • Overfitting (standard deviation)

  • Duration (mean)

  • Duration (standard deviation)

• Function call

• Warnings or errors

• The stories in the plots

Strongest evidence based results:

• Most accurate models with error ranges

• Strongest predictor with error ranges

• The stories of the strongest evidenced based data

Five strongest evidence based recommendations

Conclusions

References