Evaluating a Text Classifier

Why evaluation design matters

Text models can appear accurate because related documents occur in both training and test data, preprocessing used the entire dataset, or accuracy hides poor minority-class performance. Evaluation must mirror the intended use of the classifier.

library(textclassificationtutorial)

text <- c(
  "analyze customer data", "build statistical models",
  "create predictive analytics", "report business metrics",
  "provide patient care", "support clinical treatment",
  "coordinate nursing care", "assist hospital patients",
  "analyze experimental results", "develop data dashboard",
  "monitor patient health", "coordinate clinical team"
)
label <- rep(c("data", "care"), each = 4)
label <- c(label, "data", "data", "care", "care")

Stratified folds

Stratification distributes each class across folds. The smallest class must have at least k observations.

folds <- stratified_folds(label, k = 3, seed = 2026)
folds
#> $Repeat01_Fold01
#> [1] 2 6 8 9
#> 
#> $Repeat01_Fold02
#> [1]  1  4  7 12
#> 
#> $Repeat01_Fold03
#> [1]  3  5 10 11
#> 
#> attr(,"class")
#> [1] "text_folds" "list"      
#> attr(,"k")
#> [1] 3
#> attr(,"repeats")
#> [1] 1

For nested documents—sentences within vacancies, employees within teams, or posts within authors—split on the higher-level entity instead. Ordinary stratification does not prevent grouped leakage.

Cross-validation loop

The vocabulary must be learned from the training documents. Test documents are then aligned to that training vocabulary by predict.text_nb().

fold_results <- lapply(folds, function(test_index) {
  train_index <- setdiff(seq_along(text), test_index)

  train_clean <- preprocess_text(text[train_index])
  test_clean <- preprocess_text(text[test_index])

  train_dtm <- document_term_matrix(train_clean)
  test_dtm <- document_term_matrix(test_clean)

  model <- fit_naive_bayes(train_dtm, label[train_index])
  estimate <- predict(model, test_dtm)

  classification_metrics(
    truth = label[test_index],
    estimate = estimate,
    positive = "data"
  )
})

results <- do.call(rbind, fold_results)
results
#>                 n true_positive false_positive true_negative false_negative
#> Repeat01_Fold01 4             2              1             1              0
#> Repeat01_Fold02 4             2              0             2              0
#> Repeat01_Fold03 4             2              1             1              0
#>                 accuracy balanced_accuracy precision recall specificity  f1
#> Repeat01_Fold01     0.75              0.75 0.6666667      1         0.5 0.8
#> Repeat01_Fold02     1.00              1.00 1.0000000      1         1.0 1.0
#> Repeat01_Fold03     0.75              0.75 0.6666667      1         0.5 0.8
colMeans(results[c(
  "accuracy", "balanced_accuracy", "precision", "recall", "f1"
)], na.rm = TRUE)
#>          accuracy balanced_accuracy         precision            recall 
#>         0.8333333         0.8333333         0.7777778         1.0000000 
#>                f1 
#>         0.8666667

Interpret the metrics

Precision asks what proportion of predicted positive documents is truly positive.
Recall asks what proportion of true positive documents was found.
Specificity is recall for the negative class.
Balanced accuracy averages recall and specificity.
F1 is the harmonic mean of precision and recall.

Choose metrics before examining results. In rare-category classification, ordinary accuracy can be high even when the classifier never detects the class of interest.

From tutorial to research

For a substantive study, add:

a held-out final test set;
grouped or temporal resampling where the data structure requires it;
training-fold-only tuning and feature selection;
confidence intervals across appropriate sampling units;
human review of false positives and false negatives;
documentation of annotation quality and inter-rater agreement;
fairness and drift checks for the deployment context.