Text classification assigns documents to predefined categories. In organizational research, documents might be vacancy sentences, employee comments, reports, or interview excerpts. This tutorial develops a transparent workflow from raw HTML to predictions.
The package includes the nursing-vacancy page used by the original tutorial.
html_file <- system.file(
"extdata", "sample_nursing_vacancy.html",
package = "textclassificationtutorial"
)
vacancy_text <- extract_html_text(html_file)
substr(vacancy_text, 1, 200)
#> [1] "Als einer der führenden privaten Träger im Bereich der stationären Pflege bietet CASA REHA Ihnen zukunfts- und krisensichere Arbeitsplätze. Über 6500 Mitarbeiter versorgen rund um die Uhr unsere Bewoh"To process a folder, use extract_html_dir(). The result
keeps a document ID, the source path, and extracted text together.
CSS and XPath selection are available when xml2 is
installed:
The unit of analysis should follow the research question. Here, each sentence is treated as one document.
sentences <- split_sentences(vacancy_text)
head(sentences)
#> [1] "Als einer der führenden privaten Träger im Bereich der stationären Pflege bietet CASA REHA Ihnen zukunfts- und krisensichere Arbeitsplätze."
#> [2] "Über 6500 Mitarbeiter versorgen rund um die Uhr unsere Bewohner und leben die CASA REHA-Philosophie – \"von Mensch zu Mensch\"."
#> [3] "Bereits zum zweiten Mal in Folge wurden wir als einer der besten Arbeitgeber Deutschlands im Bereich \"Gesundheit & Soziales\" ausgezeichnet."
#> [4] "Jede unserer fast 70 Einrichtungen hat einen einzigartigen Stil, der die Besonderheiten des Standorts widerspiegelt."
#> [5] "Allen gemein sind eine familiäre Atmosphäre, ein modernes Arbeitsumfeld und hohe Qualitätsstandards."
#> [6] " "Preprocessing choices are analytical decisions, not housekeeping. Removing numbers may discard years of experience, and removing stopwords may discard meaningful negation. Make each choice explicit.
german_stopwords <- c(
"der", "die", "das", "den", "dem", "des", "und", "oder", "mit",
"für", "von", "zu", "im", "in", "auf", "ein", "eine"
)
clean <- preprocess_text(
sentences,
lowercase = TRUE,
remove_punctuation = TRUE,
remove_numbers = TRUE,
stopwords = german_stopwords,
min_token_length = 2
)
clean <- clean[nzchar(clean)]
head(clean)
#> [1] "als einer führenden privaten träger bereich stationären pflege bietet casa reha ihnen zukunfts krisensichere arbeitsplätze"
#> [2] "über mitarbeiter versorgen rund um uhr unsere bewohner leben casa reha philosophie mensch mensch"
#> [3] "bereits zum zweiten mal folge wurden wir als einer besten arbeitgeber deutschlands bereich gesundheit soziales ausgezeichnet"
#> [4] "jede unserer fast einrichtungen hat einen einzigartigen stil besonderheiten standorts widerspiegelt"
#> [5] "allen gemein sind familiäre atmosphäre modernes arbeitsumfeld hohe qualitätsstandards"
#> [6] "seniorenpflegeheim rosenpark hemmingen bieten wir bewohnern zuhause"dtm <- document_term_matrix(
clean,
min_doc_freq = 2,
max_doc_prop = 0.95
)
dtm
#> <text_dtm> 41 documents x 45 terms
#> als an andreas auch baumert bei bereich betreuung bewohner bewohnern
#> doc1 1 0 0 0 0 0 1 0 0 0
#> doc2 0 0 0 0 0 0 0 0 1 0
#> doc3 1 0 0 0 0 0 1 0 0 0
#> doc4 0 0 0 0 0 0 0 0 0 0
#> doc5 0 0 0 0 0 0 0 0 0 0
#> doc6 0 0 0 0 0 0 0 0 0 1
#> doc7 1 0 0 1 0 0 0 0 0 0
#> doc8 0 0 0 0 0 0 0 0 0 0
#> doc9 1 0 0 0 0 0 0 0 0 0
#> doc10 0 0 0 0 0 0 0 0 0 0
#> doc11 0 1 0 0 0 0 0 1 1 0
#> doc12 0 0 0 0 0 0 0 0 0 0
#> doc13 0 0 0 0 0 0 0 0 0 0
#> doc14 0 0 0 0 0 0 0 1 1 0
#> doc15 0 0 0 0 0 0 0 0 0 0
#> doc16 0 1 0 0 0 0 0 0 0 0
#> doc17 0 0 0 0 0 0 0 0 0 0
#> doc18 0 0 0 0 0 0 0 0 0 0
#> doc19 0 0 0 0 0 0 0 0 0 0
#> doc20 0 0 0 0 0 0 0 0 0 0
#> doc21 0 0 0 0 0 0 0 0 0 1
#> doc22 0 1 0 1 0 1 0 0 0 0
#> doc23 0 0 0 0 0 0 0 0 0 0
#> doc24 1 0 0 0 0 0 1 0 0 0
#> doc25 0 0 0 0 0 0 0 0 0 0
#> doc26 0 0 0 0 0 0 0 0 0 0
#> doc27 0 0 0 0 0 0 0 0 0 0
#> doc28 0 0 0 0 0 0 0 0 0 0
#> doc29 0 0 0 0 0 0 0 0 0 0
#> doc30 0 0 0 0 0 0 0 0 0 0
#> doc31 0 0 0 0 0 0 0 0 0 0
#> doc32 0 1 0 0 0 1 0 0 0 0
#> doc33 0 0 0 0 0 0 0 0 0 0
#> doc34 0 0 0 0 0 1 0 0 0 0
#> doc35 0 0 0 0 0 0 0 0 0 0
#> doc36 0 0 0 0 0 0 0 0 0 0
#> doc37 0 0 0 0 0 0 0 0 0 0
#> doc38 0 0 0 0 0 0 0 0 0 0
#> doc39 0 0 1 0 1 0 0 0 0 0
#> doc40 0 0 0 0 0 0 0 0 0 0
#> doc41 0 0 1 0 1 0 0 0 0 0
#> bieten casa dann durch einen einer einrichtung freuen gerne hemmingen
#> doc1 0 1 0 0 0 1 0 0 0 0
#> doc2 0 1 0 0 0 0 0 0 0 0
#> doc3 0 0 0 0 0 1 0 0 0 0
#> doc4 0 0 0 0 1 0 0 0 0 0
#> doc5 0 0 0 0 0 0 0 0 0 0
#> doc6 1 0 0 0 0 0 0 0 0 1
#> doc7 0 0 0 0 0 0 1 1 0 0
#> doc8 0 0 0 0 0 0 0 0 0 0
#> doc9 0 0 0 0 0 0 0 0 0 0
#> doc10 0 0 0 0 0 0 0 0 0 0
#> doc11 0 0 0 0 0 0 0 0 0 0
#> doc12 0 0 0 0 0 0 0 0 0 0
#> doc13 0 0 0 0 0 0 0 0 0 0
#> doc14 0 0 0 1 0 0 0 0 0 0
#> doc15 0 0 0 0 0 0 0 0 0 0
#> doc16 0 0 0 0 0 0 0 0 0 0
#> doc17 0 0 0 0 0 0 0 0 0 0
#> doc18 0 0 0 0 0 0 0 0 0 0
#> doc19 0 0 0 0 0 0 0 0 0 0
#> doc20 0 0 0 0 0 0 0 0 0 0
#> doc21 0 0 0 0 0 0 0 0 0 0
#> doc22 0 0 0 0 0 0 0 0 1 0
#> doc23 0 0 0 1 0 0 0 0 0 0
#> doc24 0 0 0 0 0 0 0 0 0 0
#> doc25 0 0 0 0 0 1 1 0 0 0
#> doc26 0 0 0 0 0 0 0 0 0 0
#> doc27 0 0 0 0 0 0 0 0 0 0
#> doc28 0 0 0 0 0 0 0 0 0 0
#> doc29 0 0 0 0 0 0 0 0 0 0
#> doc30 1 0 0 0 0 0 0 0 0 0
#> doc31 1 0 0 0 0 0 0 0 0 0
#> doc32 0 0 0 0 0 0 0 0 0 0
#> doc33 0 0 0 0 1 0 0 0 0 0
#> doc34 0 2 1 0 0 0 0 0 0 0
#> doc35 0 0 0 0 0 0 0 1 0 0
#> doc36 0 0 0 0 0 0 0 0 0 0
#> doc37 0 0 0 0 0 0 0 0 0 1
#> doc38 0 0 0 0 0 0 0 0 0 0
#> doc39 0 0 0 0 0 0 0 0 0 0
#> doc40 0 0 0 0 0 0 0 0 0 0
#> doc41 0 0 1 0 0 0 0 0 1 0
#> hohe ihnen ihre ihrer mensch mitarbeiter neben pflege philosophie reha
#> doc1 0 1 0 0 0 0 0 1 0 1
#> doc2 0 0 0 0 2 1 0 0 1 1
#> doc3 0 0 0 0 0 0 0 0 0 0
#> doc4 0 0 0 0 0 0 0 0 0 0
#> doc5 1 0 0 0 0 0 0 0 0 0
#> doc6 0 0 0 0 0 0 0 0 0 0
#> doc7 0 0 0 0 0 1 0 0 0 0
#> doc8 0 0 0 0 0 0 0 0 0 0
#> doc9 0 0 0 0 0 0 0 0 0 0
#> doc10 0 0 1 0 0 0 0 0 0 0
#> doc11 0 0 0 0 0 0 0 1 0 0
#> doc12 0 0 0 0 2 0 0 0 1 0
#> doc13 0 0 1 0 0 0 0 0 0 0
#> doc14 0 0 0 0 0 0 0 1 0 0
#> doc15 0 0 0 0 0 0 0 0 0 0
#> doc16 0 0 0 0 0 0 0 1 0 0
#> doc17 0 0 0 0 0 0 0 0 0 0
#> doc18 0 0 0 0 0 0 0 0 0 0
#> doc19 0 0 0 0 0 0 0 0 0 0
#> doc20 0 0 0 0 0 0 0 0 0 0
#> doc21 0 0 0 1 0 0 0 0 0 0
#> doc22 0 0 0 0 0 0 0 0 0 0
#> doc23 0 0 0 1 0 0 1 0 0 0
#> doc24 0 0 0 0 0 0 0 0 0 0
#> doc25 0 0 0 0 0 0 0 0 0 0
#> doc26 0 0 0 0 0 0 0 0 0 0
#> doc27 1 0 0 0 0 0 0 0 0 0
#> doc28 0 0 0 0 0 0 0 0 0 0
#> doc29 0 0 0 0 0 0 0 0 0 0
#> doc30 0 0 0 0 0 0 0 0 0 0
#> doc31 0 1 0 0 0 0 1 0 0 0
#> doc32 0 0 1 0 0 0 0 0 0 0
#> doc33 0 0 0 0 0 0 0 0 0 0
#> doc34 0 0 0 0 0 0 0 0 0 2
#> doc35 0 0 0 0 0 0 0 0 0 0
#> doc36 0 0 0 0 0 0 0 0 0 0
#> doc37 0 0 0 0 0 0 0 0 0 0
#> doc38 0 0 0 0 0 0 0 0 0 0
#> doc39 0 0 0 0 0 0 0 0 0 0
#> doc40 0 0 0 0 0 0 0 0 0 0
#> doc41 0 1 0 0 0 0 0 0 0 0
#> rosenpark seniorenpflegeheim sich sie sind sowie sozialkonzept
#> doc1 0 0 0 0 0 0 0
#> doc2 0 0 0 0 0 0 0
#> doc3 0 0 0 0 0 0 0
#> doc4 0 0 0 0 0 0 0
#> doc5 0 0 0 0 1 0 0
#> doc6 1 1 0 0 0 0 0
#> doc7 0 0 1 1 0 0 1
#> doc8 0 0 0 0 0 0 0
#> doc9 0 0 0 0 0 0 0
#> doc10 0 0 0 0 1 0 0
#> doc11 0 0 1 1 0 0 0
#> doc12 0 0 1 1 0 0 0
#> doc13 0 0 0 0 1 0 0
#> doc14 0 0 0 0 0 0 0
#> doc15 0 0 0 0 0 0 0
#> doc16 0 0 0 0 0 0 0
#> doc17 0 0 0 0 0 1 0
#> doc18 0 0 0 0 0 0 0
#> doc19 0 0 0 0 0 0 0
#> doc20 0 0 0 1 1 0 0
#> doc21 0 0 0 1 0 0 0
#> doc22 0 0 0 1 1 0 0
#> doc23 0 0 1 1 0 0 0
#> doc24 0 0 0 0 0 0 0
#> doc25 0 0 0 0 0 0 0
#> doc26 0 0 0 0 0 0 0
#> doc27 0 0 0 0 0 0 0
#> doc28 0 0 0 0 0 0 0
#> doc29 0 0 0 0 0 0 0
#> doc30 0 0 0 0 0 0 0
#> doc31 0 0 0 0 0 1 0
#> doc32 0 0 2 1 0 0 0
#> doc33 0 0 0 1 0 0 0
#> doc34 0 0 1 1 0 0 0
#> doc35 0 0 0 1 0 0 0
#> doc36 1 1 0 0 0 0 1
#> doc37 0 0 0 0 0 0 0
#> doc38 0 0 0 0 0 0 0
#> doc39 0 0 0 0 0 0 0
#> doc40 0 0 0 1 0 0 0
#> doc41 0 0 0 0 0 0 0
#> stationären täglich uns unsere unserer unter wir über
#> doc1 1 0 0 0 0 0 0 0
#> doc2 0 0 0 1 0 0 0 1
#> doc3 0 0 0 0 0 0 1 0
#> doc4 0 0 0 0 1 0 0 0
#> doc5 0 0 0 0 0 0 0 0
#> doc6 0 0 0 0 0 0 1 0
#> doc7 0 0 0 1 0 0 0 0
#> doc8 0 0 0 0 0 0 0 0
#> doc9 0 0 0 0 0 0 0 0
#> doc10 0 0 0 0 0 0 0 0
#> doc11 0 1 0 2 0 0 0 0
#> doc12 0 0 0 0 1 0 0 0
#> doc13 0 0 0 0 0 0 0 0
#> doc14 0 0 0 0 1 0 0 0
#> doc15 0 0 0 0 0 0 0 0
#> doc16 0 0 0 0 0 0 0 0
#> doc17 0 0 0 0 0 0 0 0
#> doc18 0 0 0 0 0 0 0 0
#> doc19 0 0 0 0 0 0 0 0
#> doc20 0 0 0 0 0 0 0 0
#> doc21 0 1 0 0 0 0 0 0
#> doc22 0 0 0 0 0 0 0 0
#> doc23 0 0 0 0 0 0 0 0
#> doc24 0 0 0 0 0 0 0 0
#> doc25 1 0 0 0 0 0 0 0
#> doc26 0 0 0 0 0 0 0 0
#> doc27 0 0 0 0 0 0 0 0
#> doc28 0 0 0 0 0 0 0 0
#> doc29 0 0 0 0 0 0 0 0
#> doc30 0 0 0 0 0 0 1 0
#> doc31 0 0 0 0 0 1 1 0
#> doc32 0 0 1 0 0 0 1 0
#> doc33 0 0 0 0 0 0 0 0
#> doc34 0 0 0 0 0 1 0 1
#> doc35 0 0 1 0 0 0 1 0
#> doc36 0 0 0 0 0 0 0 0
#> doc37 0 0 0 0 0 0 0 0
#> doc38 0 0 0 0 0 0 0 0
#> doc39 0 0 0 0 0 0 0 0
#> doc40 0 0 0 0 0 0 0 0
#> doc41 0 0 0 0 0 0 0 0
#> attr(,"binary")
#> [1] FALSERows represent documents, columns represent terms, and cells contain
counts. Use binary = TRUE when presence is more appropriate
than frequency.
TF-IDF increases the weight of terms that are frequent in a particular document but uncommon across the collection.
weighted <- tf_idf(dtm)
keywords <- extract_keywords(dtm, n = 3)
head(keywords, 12)
#> document rank term weight
#> 1 doc1 1 stationären 0.4548822
#> 2 doc1 2 bereich 0.4189219
#> 3 doc1 3 casa 0.4189219
#> 4 doc2 1 mensch 0.8086794
#> 5 doc2 2 mitarbeiter 0.4043397
#> 6 doc2 3 philosophie 0.4043397
#> 7 doc3 1 bereich 0.8378438
#> 8 doc3 2 einer 0.8378438
#> 9 doc3 3 als 0.7364775
#> 10 doc4 1 einen 1.8195287
#> 11 doc4 2 unserer 1.6756876
#> 12 doc5 1 hohe 1.8195287Cosine similarity compares the orientation of two feature vectors while reducing the influence of document length.
For a compact illustration, use synthetic documents with known labels.
training_text <- c(
"analyze data statistical model",
"build predictive model data",
"create dashboard analyze metrics",
"provide nursing care patient",
"support patient clinical care",
"coordinate nurse patient treatment"
)
training_labels <- c("data", "data", "data", "care", "care", "care")
training_dtm <- document_term_matrix(training_text)
model <- fit_naive_bayes(training_dtm, training_labels, laplace = 1)
model
#> <text_nb> Multinomial Naive Bayes
#> Classes: data, care
#> Terms: 18
predicted <- predict(model, training_dtm)
classification_metrics(training_labels, predicted, positive = "data")
#> n true_positive false_positive true_negative false_negative accuracy
#> 1 6 3 0 3 0 1
#> balanced_accuracy precision recall specificity f1
#> 1 1 1 1 1 1This training-set result demonstrates mechanics, not generalization. The next vignette shows out-of-sample evaluation.