creating a tcorpus from full-text

The create_tcorpus function creates a tcorpus from full-text input. The full text can be provided as a single character vector or as a data.frame in which the text is given in one of the columns. We recommend using the data.frame approach, because this automatically imports all other columns as document meta.

As an example we have provided the sotu_texts demo data. This is a data.frame in which each rows represents a paragraph of the state of the union speeches from Bush and Obama

colnames(sotu_texts)

## [1] "id"        "date"      "party"     "text"      "president"

We can pass sotu_texts to the create_tcorpus function. Here we also need to specify which column contain the text (text_columns), and which column contains the document id (doc_column). Note that multiple text columns can be given.

tc = create_tcorpus(sotu_texts, doc_column = 'id', text_columns = 'text')

Printing tc shows the number of tokens (i.e. words) in the tcorpus, the number of documents, and the columns in the tokens and meta data.tables.

tc

## tCorpus containing 90827 tokens
## grouped by documents (n = 1090)
## contains:
##   - 3 columns in $tokens:    doc_id, token_id, token
##   - 4 columns in $meta:      doc_id, date, party, president

We can also look at the tokens and meta data directly (for changing this data, please read the Managing the tcorpus section below).

## first 6 rows (head) of tokens and meta data.tables
head(tc$tokens)   

## Key: <doc_id, token_id>
##       doc_id token_id      token
##       <fctr>    <int>     <fctr>
## 1: 111541965        1         It
## 2: 111541965        2         is
## 3: 111541965        3        our
## 4: 111541965        4 unfinished
## 5: 111541965        5       task
## 6: 111541965        6         to

head(tc$meta)

## Key: <doc_id>
##       doc_id       date     party    president
##       <char>     <Date>    <fctr>       <char>
## 1: 111541965 2013-02-12 Democrats Barack Obama
## 2: 111541995 2013-02-12 Democrats Barack Obama
## 3: 111542001 2013-02-12 Democrats Barack Obama
## 4: 111542006 2013-02-12 Democrats Barack Obama
## 5: 111542013 2013-02-12 Democrats Barack Obama
## 6: 111542018 2013-02-12 Democrats Barack Obama

Importing a tokenlist

If you already have a tokenlist it can be imported as a tcorpus with the tokens_to_tcorpus function. The tokenlist has to be formatted as a data.frame. As an example we provide the corenlp_tokens demo data, which is the output of the Stanford CoreNLP parser.

This type of data.frame can be passed to the tokens_to_tcorpus function. The names of the columns that describe the token positions (document_id, sentence and token_id) also need to be specified.

tc = tokens_to_tcorpus(corenlp_tokens, doc_col = 'doc_id',
                       sentence_col = 'sentence', token_id_col = 'id')
tc

## tCorpus containing 36 tokens
## grouped by documents (n = 1) and sentences (n = 6)
## contains:
##   - 13 columns in $tokens:   doc_id, sentence, token_id, token, lemma, CharacterOffsetBegin, CharacterOffsetEnd, POS, NER, Speaker, parent, relation, pos1
##   - 1 column in $meta:   doc_id

To include document meta data, a separate data.frame with meta data has to be passed to the meta argument. This data.frame needs to have a column with the document id, using the name specified in doc_col.

Adding, removing and mutating columns

To add, remove or mutate columns in the tc$tokens or tc$meta data.tables, the tc$set() and tc$set_meta() methods can be used (also see ?tCorpus_data. These methods change the tCorpus by reference (for more details about R6 methods and changing by reference, see the section Using the tCorpus R6 methods.).

The set method (?set) has two mandatory arguments: columns and value. The column argument takes the name of the column to add or mutate, and the value argument takes the value to assign to the column.

tc = create_tcorpus('This is an example')

tc$set('new_column', "any value")
tc$tokens

## Key: <doc_id, token_id>
##    doc_id token_id   token new_column
##    <fctr>    <int>  <fctr>     <fctr>
## 1:      1        1    This  any value
## 2:      1        2      is  any value
## 3:      1        3      an  any value
## 4:      1        4 example  any value

The value can also be an expression, which will be evaluated in the $tokens data.table. Here we overwrite the new_column with the values of the token column in uppercase.

tc$set('new_column', toupper(token))

Optionally, the subset argument can be used to only mutate values for the rows where the subset expression evaluates to TRUE. For example, the following operation changes the new_column values to lowercase if token_id <= 2.

tc$set('new_column', tolower(token), token_id <= 2)

Set cannot be used to change a column name, so for this we have the $set_name() method

tc$set_name('new_column', 'token2')

To remove columns, you can use the delete_columns method.

tc$delete_columns('token2')

To add, remove and mutate columns in the $meta data.table, the set_meta, set_meta_name and delete_meta_columns methods can be used.

Subsetting a tCorpus

The generic subset function can be used to subset the tCorpus on both the tokens and meta data. To subset on tokens, the subset argument can be used.

tc = create_tcorpus(sotu_texts, doc_col='id', split_sentences = T)
tc2 = subset(tc, sentence==1)

## number of tokens in full corpus
tc$n

## [1] 90827

## number of tokens in first sentence
tc2$n

## [1] 23233

To subset on meta, the subset_meta argument can be used.

tc2 = subset(tc, subset_meta = president == 'Barack Obama')

## number of tokens in president Obama documents
tc2$n

## [1] 45956

As an alternative, subsetting can also be performed with the $subset() R6 method. This will modify the tCorpus by reference, which is faster and more memory efficient.

tc$subset(sentence == 1, subset_meta = president == 'Barack Obama')
tc$n

## [1] 12274

Now tc itself has been subsetted, and only contains tokens of the first sentence of Obama documents.

Deduplication

A common problem in text analysis is that there can be duplicate or near duplicate documents in a corpus. For example, a corpus of news articles might have multiple versions of the same news article if it has been updated. It is often appropriate to delete such duplicates. The tCorpus has a flexible deduplicate method that can delete duplicates based on document similarity.

Document similarity is calculated using a similarity measure for the weighted frequencies of words (or any selected feature) in documents. By default, this is the cosine similarity of token frequencies with a tf-idf weighting scheme. To ease computation a minimum document frequency (default is 2) and maximum document frequency percentage (default is 50%) is used.

It is often possible to limit comparisons by including meta data about date and categories. For example, duplicate articles in newspapers should occur within a limited time frame and within the same newspaper. The meta_cols argument can be used to only compare within documents with identical meta values, and the date_col can be used to only compare documents within a the time interval specified in hour_window. Also, by using date_col, it can be specified whether the duplicate with the first or last date should be removed.

d = data.frame(doc_id = paste('doc', 1:3),
               text = c('I am a text','I am also text', 'Ich bin ein Berliner'),
               date = as.POSIXct(c('2010-01-01','2010-01-02','2010-01-03')))

# document 1 and 2 are duplicates, and the first is deleted
tc = create_tcorpus(d)
tc$deduplicate(feature='token', date_col = 'date', similarity = 0.75, verbose = F)
tc$meta$doc_id     

## [1] "doc 1" "doc 3"

# document 1 and 2 are duplicates, and the last is deleted
tc = create_tcorpus(d)
tc$deduplicate(feature='token', date_col = 'date', similarity = 0.75, keep = 'last', verbose = F)
tc$meta$doc_id

## [1] "doc 2" "doc 3"

Basic preprocessing

The basic preprocessing techniques can be performed on a tcorpus with the preprocess method. The main arguments are:

• column: the name of the column to use as input. Default is “token”
• new_column: the name of the column in which to store the output. Default is “feature”
• lowercase: make text lowercase. Defaults to TRUE
• remove_punctuation: remove punctuation. Defaults to TRUE
• use_stemming: apply stemming. Defaults to FALSE
• remove_stopwords: removes stopwords (e.g., the, it, is). Defaults to FALSE
• language: the language used for stemming and stopword removal. Defaults to “english”

Lowercasing and removing punctuation is so common that its the default. In the following example we also apply stemming and stopword removal. Note that since we work with English data we do not need to set the language, but if other languages are used the language argument needs to be used. Also note that we do not specify column and new_column. A tcorpus created with create_tcorpus always has a “token” column with the raw token text.

tc = create_tcorpus(sotu_texts, doc_column = 'id', text_columns = 'text')

tc$preprocess(use_stemming = T, remove_stopwords=T)
tc$tokens

## Key: <doc_id, token_id>
##           doc_id token_id      token  feature
##           <fctr>    <int>     <fctr>   <fctr>
##     1: 111541965        1         It     <NA>
##     2: 111541965        2         is     <NA>
##     3: 111541965        3        our     <NA>
##     4: 111541965        4 unfinished unfinish
##     5: 111541965        5       task     task
##    ---                                       
## 90823: 111552781        3   continue  continu
## 90824: 111552781        4         to     <NA>
## 90825: 111552781        5      bless    bless
## 90826: 111552781        6    America  america
## 90827: 111552781        7          .     <NA>

The tokens data.table now has the new column “feature” which contains the preprocessed token.

Advanced preprocessing with UDPipe

There are several packages for using NLP pipelines in R, such as spacyr for SpaCy, coreNLP for Stanford CoreNLP, and udpipe for UDPipe. Most of these rely on external dependencies (Python for SpaCy, Java for CoreNLP), but the UDPipe pipeline runs in C++ which plays nicely with R. We have built a wrapper for the udpipe package which provides bindings for using UDPipe directly from within R. Unlike basic preprocessing, this feature has to be used in the create_tcorpus function, because it requires the raw text input.

To use a UDPipe model in create_tcorpus, you simply need to use the udpipe_model argument. The value for this argument is the language/name of the model you want to use (if you don’t know the name, just give the language and you’ll get suggestions). The first time using a language, R will automatically download the model. By default the udpipe model is stored in the current working directory, in a folder called udpipe_models, but you can set the path with the udpipe_model_path parameter.

tc = create_tcorpus("This is a quick example", udpipe_model='english-ewt')
tc$tokens   ## output not shown in this vignette

By default, UDPipe will not perform dependency parsing. You can activate this part of the pipeline by using the use_parser = TRUE argument, but it takes longer to compute and not all language have good dependency parser performance.

The feats column contains a bunch of nested features, such as Person (first, second or third) and Number (singular or plural). If you don’t need this, its good to remove the colum (tc$delete_colums('feats')). If you want to use (some of) this information, you can use the tc$feats_to_columns method to cast these nested features to columns. Here we specifically select the Tense, Person and Number features. Note that the default behavior is to also delete the feats column.

tc$feats_to_columns(c('Tense','Number','Person'))

tc = create_tcorpus(sotu_texts, doc_column = 'id', udpipe_model='english-ewt', updipe_cores=4)

Filtering tokens

In basic preprocessing we saw that NA values were introduced where stopwords were filtered out. If you want to filter out more tokens, you can use the tc$feature_subset() method. This works as a regular subset, but the difference is that the corpus is kept intact. That is, the rows of the tokens that are filtered out are not deleted.

As a (silly) example, we can specifically filter out the token with token_id 5. For this we take the subset of tokens that is not 5.

## for sake of clarity, recreate the corpus and preprocess it
tc = create_tcorpus(sotu_texts, doc_column = 'id', text_columns = 'text')
tc$preprocess(use_stemming = T, remove_stopwords=T)

tc$feature_subset('feature', !token_id == 5)

This sets all values in the ‘feature’ column with token_id 5 to NA. This is just an example that is easy to run. A more usefull application is to use this to subset on part-of-speech tags.

It is often usefull to filter out tokens based on frequency. For instance, to only include words in a model that are not too rare to be interesting and not too common to be informative. The feature subset function has four arguments to do so: min_freq, max_freq, min_docfreq and max_docfreq. Here freq indicates how often a term occured in the corpus, and docfreq indicates the number of unique documents in which a term occured. For example, the following code filters out all tokens that occur less than 10 times.

tc$feature_subset('feature', min_freq=10)

For the sake of convenience, the min/max functions are also integrated in the preprocess method. So it would also have been possible to use tc$preprocess(use_stemming = T, remove_stopwords=T, min_freq=10). Also, if you want to inspect the freq and docfreq of terms, you could use the feature_stats(tc, 'feature') function.

Creating a DTM or DFM

The data in a tCorpus can easily be converted to a document-term matrix (dtm), or to the document-feature matrix (dfm) class of the quanteda package.

In general, we recommmend using the quanteda dfm class, which can be created from a tCorpus by using the get_dfm function. Alternatively, the get_dtm function can be used to create a regular sparse matrix (dgTMatrix from the Matrix package) or the DocumentTermMatrix class from the tm package is also supported. Here we only demonstrate get_dfm.

Here we first preprocess the tokens, creating the feature column, and then create a dfm with these features.

tc = create_tcorpus(sotu_texts, doc_col='id', split_sentences=T)

tc$preprocess(column = 'token', new_column = 'feature',
              remove_stopwords=T, use_stemming=T, min_docfreq = 5)

dfm = get_dfm(tc, 'feature')
dfm

## Document-feature matrix of: 1,090 documents, 1,422 features (97.82% sparse) and 3 docvars.
##            features
## docs        1 10 100 11 11th 12 15 18 19 1990s
##   111541965 0  0   0  0    0  0  0  0  0     0
##   111541995 0  0   0  0    0  0  0  0  0     0
##   111542001 0  0   0  0    0  0  0  0  0     0
##   111542006 0  0   0  0    0  0  0  0  0     0
##   111542013 0  0   0  0    0  0  0  0  0     0
##   111542018 0  0   0  0    0  0  0  0  0     0
## [ reached max_ndoc ... 1,084 more documents, reached max_nfeat ... 1,412 more features ]

The get_dfm function has several usefull parameters. We won’t show each in detail, but list some applications.

##  Use sentences as rows instead of documents
dfm_sent = get_dfm(tc, 'feature', context_level = 'sentence')

##  Use a weighting scheme. 
dfm_weighted = get_dfm(tc, 'feature', weight = 'tfidf')

##  Only use a subset of the tCorpus
dfm_obama = get_dfm(tc, 'feature', subset_meta = president == "Barack Obama")

By preprocessing the tokens within the tCorpus, the features in the dfm can be linked to the original tokens in the tCorpus. In the next section, Keeping the full corpus intact, we show how this can be used to annotate the tCorpus with results from bag-of-words style text analyses.

Why keep the full corpus intact?

In the tCorpus the original tokens are still intact after preprocessing. This is not very memory efficient, so why do we do it?

Keeping the corpus intact in this way has the benefit that the results of an analysis, as performed with the preprocessed tokens, can be linked to the full corpus. For example, here we show a quick and dirty example of annotating a tCorpus based on a topic model.

## create tcorpus and preprocess tokens
tc = create_tcorpus(sotu_texts, doc_column = 'id', text_columns = 'text')
tc$preprocess(use_stemming = T, remove_stopwords=T, min_docfreq = 5)

## fit lda model, using the create_feature argument to store the topic assignments
m = tc$lda_fit('feature', create_feature = 'topic', K = 5, alpha = 0.001)

The lda_fit R6 method is simply a wrapper for the LDA function in the topicmodels package. However, next to returning the topic model (m), it also adds a column to tc$tokens with the topic assignments.

head(tc$tokens, 10)

## Key: <doc_id, token_id>
##        doc_id token_id      token  feature topic
##        <fctr>    <int>     <fctr>   <fctr> <num>
##  1: 111541965        1         It     <NA>    NA
##  2: 111541965        2         is     <NA>    NA
##  3: 111541965        3        our     <NA>    NA
##  4: 111541965        4 unfinished unfinish     2
##  5: 111541965        5       task     task     2
##  6: 111541965        6         to     <NA>    NA
##  7: 111541965        7    restore   restor     2
##  8: 111541965        8        the     <NA>    NA
##  9: 111541965        9      basic    basic     2
## 10: 111541965       10    bargain     <NA>    NA

This is just an example (with a poorly trained topic model) but it demonstrates the purpose. In the tokens we now see that certain words (unfinished, restore) have topic assignments. These topic assignments are based on the topic model trained with the preprocessed versions of the tokens (unfinish, restor). Thus, we can relate the results of a text analysis technique (in this case topic modeling, but the same would work with word scaling, dictionaries, etc.) to the original text. This is important, because in the end we apply these techniques to make inferences about the text, and this approach allows us to validate the results and/or perform additional analyses on the original texts.

We can now also reconstruct the full text with the topic words coloured to visualize the topic assignments. The browse_text function creates an HTML browser of the full texts, and here we use the category = 'topic' argument to indicate that the tokens$topic column should be used to colour the words. If you use the view = TRUE argument, you’ll also directly see the browser in the viewer panel.

url = browse_texts(tc, category='topic', view=T)

By default, browse_texts only uses the first 500 documents to prevent making huge HTML files. The number of documents can be changed (n = 100), and the selection can be set to random (select = ‘random’), optionally with a seed (seed = 1) to create a reproducible sample for validation purposes.

The browse_texts function can also be used to visualize types of annotations (highlight values between 0 and 1, scale values between -1 and 1, categories such as topics or search results), or to only create a text browser.

search_features()

To demonstrate the search_features() function, we first make a tcorpus of the SOTU speeches. Note that we use split_sentences = T, so that we can also view results at the sentence level.

tc = create_tcorpus(sotu_texts, doc_column = 'id', text_columns = 'text', split_sentences=T)

The search_features() function takes the tcorpus as the first argument. The query should be a character vector with one or multiple queries. Here we look for two queries, terror* and war*.

hits = search_features(tc, query = c('terror*','war*'))

The first time a query search is performed the token column is indexed to enable fast binary search(the tCorpus remembers the index).

The output of search_features is a featureHits object.

hits

## 284 hits (in 175 documents)

summary(hits)

##      code hits documents
## 1 query_1  184       126
## 2 query_2  100        80

The regular output shows the total number of hits, and the summary shows the hits per query. Note, however, that the code is now query_1 and query_2 because we didn’t label our queries. There are two ways to label queries in corpustools. One is to provide the code labels with the code argument. The other is to prefix the label in the query string with a hashtag. Here we use the latter.

hits = search_features(tc, query = c('Terror# terror*','War# war*'))
summary(hits)

##     code hits documents
## 1 Terror  184       126
## 2    War  100        80

It is usefull to know that the featureHits object has a hits and queries slot. The queries slot contains the query input for provenance, and the hits slot shows the tokens that are matched. Note that queries can match multiple words. The hit_id column indicates the unique matches.

hits$queries

##     code    query
## 1 Terror  terror*
## 2    War     war*

head(hits$hits)

## Key: <doc_id, token_id>
##      code    feature    doc_id sentence token_id hit_id
##    <char>     <fctr>    <fctr>   <lgcl>    <int>  <int>
## 1: Terror terrorists 111542025       NA       99      1
## 2: Terror terrorists 111542114       NA       84      2
## 3:    War        war 111542119       NA        5      1
## 4: Terror  terrorism 111542119       NA       34      3
## 5:    War        war 111542189       NA       50      2
## 6:    War        war 111542284       NA       54      3

queries = data.frame(label = c('War','Terrorism','Economy','Education'),
                     query = c('war* OR army OR bomb*', 'terroris*', 'econom* OR bank*', 'educat* OR school*'))
hits = search_features(tc, query=queries$query, code=queries$label)

count_tcorpus(tc, hits=hits)

## Key: <group, N, V>
##     group     N     V Economy Education Terrorism   War
##    <char> <int> <int>   <num>     <num>     <num> <num>
## 1:  total  1090 90827     182       126        95    97

count_tcorpus(tc, hits=hits, meta_cols = 'president')  ## break by meta variables

## Key: <president, N, V>
##         president     N     V Economy Education Terrorism   War
##            <char> <int> <int>   <num>     <num>     <num> <num>
## 1:   Barack Obama   554 45956     115        80        14    38
## 2: George W. Bush   536 44871      67        46        81    59

library(ggplot2) 

date_hits = count_tcorpus(tc, hits, meta_cols='date', wide = F)
  
ggplot(date_hits, aes(x=date, y=count, group=code)) + 
  geom_line(aes(linetype=code))
  
pres_hits = count_tcorpus(tc, hits, meta_cols='president', wide = F)
  
ggplot(pres_hits, aes(president, count)) + 
  geom_col(aes(fill=code), width=.5, position = "dodge")

g = semnet(hits, measure = 'con_prob')

igraph::get.adjacency(g, attr = 'weight')

## Warning: `get.adjacency()` was deprecated in igraph 2.0.0.
## ℹ Please use `as_adjacency_matrix()` instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.

## 4 x 4 sparse Matrix of class "dgCMatrix"
##           Economy Education Terrorism    War
## Economy    .         0.1429    0.0385 0.0385
## Education  0.2063    .         0.0238 0.0635
## Terrorism  0.0737    0.0316    .      0.2842
## War        0.0722    0.0825    0.2784 .

plot(hits)

url = browse_hits(tc, hits, view=T)

get_kwic(tc, query = 'freedom* AND america*', n = 2)

##      doc_id code hit_id            feature
## 1 111542576           1 America -> freedom
## 2 111542742           2 America -> freedom
##                                                                                                                                                                                                                                 kwic
## 1 ...misguided idealism. In reality, the future security of <America> depends on it. On September the 11th, 2001 [...] and join the fight against terror. Every step toward <freedom> in the world makes our country safer, so we...
## 2                                               ...In Afghanistan, <America>, our 25 NATO allies, and 15 partner nations are helping the Afghan people defend their <freedom> and rebuild their country. Thanks to the courage of...

## example query that matches first words in corpus
q = 'Example# unfinished OR restore OR "basic bargain"' 
tc$code_features(q)
head(tc$tokens, 10)

## Key: <doc_id, token_id>
##        doc_id sentence token_id      token    code
##        <fctr>    <int>    <int>     <fctr>  <char>
##  1: 111541965        1        1         It    <NA>
##  2: 111541965        1        2         is    <NA>
##  3: 111541965        1        3        our    <NA>
##  4: 111541965        1        4 unfinished Example
##  5: 111541965        1        5       task    <NA>
##  6: 111541965        1        6         to    <NA>
##  7: 111541965        1        7    restore Example
##  8: 111541965        1        8        the    <NA>
##  9: 111541965        1        9      basic Example
## 10: 111541965        1       10    bargain Example

search_contexts()

The search_contexts function is very similar to search_features, but it only looks for whether a query occurs in a document or sentence.

tc = create_tcorpus(sotu_texts, doc_column = 'id', text_columns = 'text', split_sentences=T)

con = search_contexts(tc, 'war')
con

## 68 documents

tc = create_tcorpus(sotu_texts, doc_column = 'id', text_columns = 'text', split_sentences=T)
tc_war = subset_query(tc, 'war')

tc$subset_query('war')

search_dictionary

The search_dictionary function, and the related code_dictionary method, are alternative to search_features and search_context for working with larger but less complex dictionaries. For example, dictionaries with names of people, organizations or countries, or scores for certain emotions. Most of these dictionaries do not use Boolean operators such as AND and more complicated patterns such as word proximity.

The following code shows an example (not shown in this vignette, because it depends on quanteda). Here we use one of the dictionaries provided by quanteda. The quanteda dictionary class (dictionary2) can be used as input for the search dictionary functions.

library(quanteda)
dict = quanteda::data_dictionary_LSD2015

Given a tCorpus, the dictionary can be used (by default on the “token” column) with the search_dictionary function.

tc = create_tcorpus(sotu_texts, doc_column = 'id', text_columns = 'text', split_sentences=T)
hits = search_dictionary(tc, dict)

The output of this function is the same as search_features, so we can do anything with it as discussed above. For example, we can count and plot the data.

library(ggplot2)
agg_hits = count_tcorpus(tc, 'date', hits, wide = F)
ggplot(agg_hits, aes(x=date, y=count, group=code)) + 
  geom_line(aes(linetype=code))

Like code_features, the code_dictionary R6 method can be used to annotate the tokens data with the results. For the LSD2015 dictionary this would add a column with the labels “positive” and “negative”, and the negated terms “neg_positive” and “neg_negative”.

Instead of using these labels, we can also convert them to scores on a scale from -1 (negative) to 1 (positive). While it is possible to create the numeric scores for sentiment within the tCorpus, we’ll show another option here. Instead of using the quanteda dictionary, we can use a data.frame as a dictionary as long as it has a column named “string” that contains the dictionary pattern. All other columns in the data.frame are then added to $tokens. With the melt_quanteda_dict function we can also convert a quanteda dictionary to this type of data.frame. The code is then as follows:

dict = melt_quanteda_dict(dict)
dict$sentiment = ifelse(dict$code %in% c('positive','neg_negative'), 1, -1)
tc$code_dictionary(dict)
tc$tokens

The browse_texts function can also color words based on a scale value between -1 and 1. This is a nice way to inspect and validate the results of a sentiment dictionary.

browse_texts(tc, scale='sentiment')

We can use agg_tcorpus to aggregate the results. This is a convenient wrapper for data.table aggregation for tokens data. Especially important is the .id argument, which allows us to specify an id column. In the dictionary results, there are matches that span multiple rows (e.g., “not good”), and we only want to count these values once. By using the code_id column that code_dictionary added to indicate unique matches, we only aggregate unique non-NA rows in tokens.

agg_tcorpus(tc, sent = mean(sentiment), N = length(sentiment), .id = 'code_id')

The by argument also lets us group by columns in the tokens or meta data.

agg_tcorpus(tc, sent = mean(sentiment), .id = 'code_id', by='president')
agg_tcorpus(tc, sent = mean(sentiment), .id = 'code_id', by=c('token','president'))

Semantic networks based on co-occurence

Semantic networks can be created based on the co-occurence of features in contexts (document, sentences) or word windows. For this we use the semnet and semnet_window functions.

It is recommended to first apply preprocessing to filter out terms below a given document frequency, because the number of comparisons and the size of the output matrix/network increases exponentially with vocabulary size.

tc = create_tcorpus(sotu_texts, doc_col='id')
tc$preprocess(min_docfreq = 20, remove_stopwords = T, remove_numbers = T)

g = semnet_window(tc, 'feature')

We can view the network with the plot_semnet function. However, this network will be a mess if we do not apply some filter on nodes (i.e. features) and edges (i.e. similarity scores). One way is to take only the most frequent words, and use a hard threshold for similarity.

A nice alternative is to use backbone extraction using the backbone_filter function. We then give an alpha score (similar to a p-value, but see paper reference in function documentation for details) to filter edges. In addition, the max_vertices argument can be used to specify the maximum number of nodes/features. Instead of looking for the most frequent features, this approach keeps lowering the alpha and deleting isolates until the max is reached. As a result, the filter focuses on features with strong co-occurences.

gb = backbone_filter(g, alpha = 0.001, max_vertices = 100)

## Used cutoff edge-weight 2.13863525362624e-05 to keep number of vertices under 100

## (For the edges the original weight is still used)

plot_semnet(gb)

Corpus comparisons

We can compare feature frequencies between two corpora (compare_corpus), or between subsets of a corpus (compare_subset). Here we compare the subset of speeches from Obama with other speeches (in this demo data that’s only George Bush)

comp = compare_subset(tc, feature='feature',
                      subset_meta_x = president == "Barack Obama")

The output comp is a data.frame with for each feature how often it occured in the subset (Obama) and the rest of the corpus (Bush), including (smoothed) probability scores, the ratio of probabilities, and a chi2 value.

For a quick peek, we can plot the most strongly overrepresented (ratio > 1) and underrepresented (ratio < 1). By default, the plot method for the output of a vocabulary comparison function(such as compare_subset) plots the log ratio on the x axis, and uses the chi2 score to determine the size of words (chi2 can be larger for more frequent terms, but also relies on how strongly a term is over/under-represented).

plot(comp)

For alternative settings, see ?plot.vocabularyComparison.

Feature associations

Similar to a corpus comparison, we can compare how frequently words occur close to the results of a query to the overal frequency of these words. This gives a nice indication of the context in which words queries tend to occur.

The feature_associations function either requires a query or hits (results of search_features). With the feature argument we indiate for what feature column we want to compare the values. Important to consider is that this is not necessarily the column on which we want to perform the query. For example, we might want to compare frequencies of stemmed, lowercase words, or topics from a topic model, but we want to perform the query on the raw tokens.

By default, the query is always performed on the ‘token’ column. In the following example we thus query on the tokens, but look at the frequencies of the preprocessed features. We query for “terror*“.

tc = create_tcorpus(sotu_texts, doc_col='id')
tc$preprocess(use_stemming = T, remove_stopwords = T, min_docfreq = 10)

fa = feature_associations(tc, 'feature', query = 'terror*')

The output fa is almost identical to the output of a corpus comparison, but by default feature_associtaions only returns the top features with the strongest overrepresentation (underrepresentation is not really interesting here).

head(fa)

##     feature freq freq_NOT ratio  chi2
## 837     war   25       61  7.35 103.1
## 74   attack   16       30  9.57  84.4
## 540  offens    8        5 28.41  74.8
## 843  weapon   20       54  6.64  74.2
## 115    camp    7        4 30.97  66.7
## 308   fight   19       57  5.98  62.8

We can also plot the output as a wordcloud.

plot(fa, col=c('lightblue','navyblue'))

For more details see ?plot.featureAssociations.

Being carefull with shallow copies.

Here we show two examples of possible mistakes in using corpustools that you might make if you are not familiar with modifying by reference. The following section shows how to do this properly.

Firstly, you can assign the output of a tCorpus R6 method, but this will only be a shallow copy of the tCorpus. The original corpus will still be modified. In the following example, we create a subset of tc and assign it to tc2, but we see that tc itself is still modified.

tc = create_tcorpus('this is an example')
tc2 = tc$subset(token_id < 3)

## tc and tc2 are both subsets with token_id < 3
identical(tc$tokens, tc2$tokens)

## [1] TRUE

The second thing to be aware of is that you do not copy a tCorpus by assigning it to a different name. In the following example, tc2 is a shallow copy of tc, and by modifying tc2 we also modify tc.

tc = create_tcorpus('this is an example')
tc2 = tc
tc2$subset(token_id < 3)

## tc and tc2 are both subsets with token_id < 3
identical(tc$tokens, tc2$tokens)

## [1] TRUE

Copying a tCorpus

The tCorpus R6 methods modify the tCorpus by reference. If you do not want this, you have several options. Firstly, for methods where it makes sense, we provide identical regular functions that do not modify the tCorpus by reference. For example, instead of the subset R6 method you could use the subset function is classic R style.

tc2 = subset(tc, token_id < 2)

Alternatively, these R6 methods also have a copy argument, which can be set to TRUE to force a deep copy.

tc2 = tc$subset(token_id < 2, copy=T)

Finally, you could always make a deep copy of the entire tCorpus. This is not the same as simply assigning to the tCorpus to a new name.

tc2 = tc         ## shallow copy
tc3 = tc$copy()  ## deep copy

The difference is that in the shallow copy, if you change tc2 by reference (e.g., by using the subset R6 method), it wil also change tc!! With a deep copy, this does not happen because the entire tCorpus is copied and stored seperately in memory.

Of course, making deep copies is ‘expensive’ in speed and memory. Our recommended approach is therefore to not overwrite or remove data from the corpus, but rather add columns or make values NA. The tCorpus was designed to keep all information in one big corpus, and manages to do this fairly efficiently by properly using the data.table package to prevent copying where possible.