---
title: "Accessing the Wordbank database"
author: "Mika Braginsky"
date: "`r Sys.Date()`"
output:
  rmarkdown::html_vignette:
    toc: true
vignette: >
  %\VignetteIndexEntry{Accessing the Wordbank database}
  %\VignetteEncoding{UTF-8}
  %\VignetteEngine{knitr::rmarkdown}
---

```{r, include=FALSE}
library(wordbankr)
library(dplyr)
library(ggplot2)
knitr::opts_chunk$set(message = FALSE, warning = FALSE, cache = FALSE)
theme_set(theme_minimal())

# CRAN machines must not execute network code (and vignette builds there
# must be fast): data chunks only evaluate when NOT_CRAN=true, e.g. locally
# or on CI jobs that opt in
knitr::opts_chunk$set(eval = identical(Sys.getenv("NOT_CRAN"), "true"))
```

The `wordbankr` package allows you to access data in the [Wordbank database](https://wordbank.stanford.edu/) from `R`. This vignette shows some examples of how to use the data loading functions and what the resulting data look like.

The data are hosted as a versioned dataset on [Redivis](https://stanford.redivis.com/datasets/627v-9ewzpdvz0). The dataset is public, but downloading from Redivis requires a (free) account: the first time you call a `wordbankr` function in an interactive session, a browser window opens asking you to authorize access, and your credentials are cached after that. For scripts and servers, set the `REDIVIS_API_TOKEN` environment variable instead. If Wordbank cannot be reached, functions print a message and return `NULL` rather than raising an error. See [Data versions and reproducibility] below for how to pin an analysis to a specific release of the data.

There are three different data views that you can pull out of Wordbank: by-administration, by-item, and administration-by-item. Additionally, you can get metadata about the datasets and instruments underlying the data. Advanced functionality let's you get estimates of words' age of acquisition and word mappings across languages.


## Administrations

The `get_administration_data()` function gives by-administration information, either for a specific language and/or form or for all instruments.
```{r}
get_administration_data(language = "English (American)", form = "WS")
get_administration_data()
```


## Items

The `get_item_data()` function gives by-item information, either for a specific language and/or form or for all instruments.
```{r}
get_item_data(language = "Italian", form = "WG")
get_item_data()
```


## Administrations x Items

If you are only looking at total vocabulary size, `admins` is all you need, since it has both productive and receptive vocabulary sizes calculated. If you are looking at specific items or subsets of items, you need to load instrument data, using the `get_instrument_data()` function. Pass it an instrument language and form, along with a list of items you want to extract (by `item_id`).
```{r}
get_instrument_data(
  language = "English (American)",
  form = "WS",
  items = c("item_26", "item_46")
)
```

By default `get_instrument_table()` returns a data frame with columns of the administration's `data_id`, the item's `num_item_id` (numerical `item_id`), and the corresponding value. To include administration information, you can set the `administrations` argument to `TRUE`, or pass the result of `get_administration_data()` as `administrations` (that way you can prevent the administration data from being loaded multiple times). Similarly, you can set the `iteminfo` argument to `TRUE`, or pass it result of `get_item_data()`.

Loading the data is fast if you need only a handful of items, but the time scales about linearly with the number of items, and can get quite slow if you need many or all of them. So, it's a good idea to filter down to only the items you need before calling `get_instrument_data()`.

As an example, let's say we want to look at the production of animal words on English Words & Sentences over age. First we get the items we want:
```{r, fig.width=6, fig.height=4}
items <- get_item_data(language = "English (American)", form = "WS")
animals <- if (!is.null(items)) items %>% filter(category == "animals")
```

Then we get the instrument data for those items:
```{r}
animal_data <- if (!is.null(animals)) {
  get_instrument_data(language = "English (American)",
                      form = "WS",
                      items = animals$item_id,
                      administration_info = TRUE,
                      item_info = TRUE)
}
```

Finally, we calculate how many animals words each child produces and the median number of animals of each age bin:
```{r, fig.width=6, fig.height=4}
if (!is.null(animal_data)) {
  animal_summary <- animal_data %>%
    group_by(age, data_id) %>%
    summarise(num_animals = sum(produces, na.rm = TRUE)) %>%
    group_by(age) %>%
    summarise(median_num_animals = median(num_animals, na.rm = TRUE))
  
  ggplot(animal_summary, aes(x = age, y = median_num_animals)) +
    geom_point() +
    labs(x = "Age (months)", y = "Median animal words producing")
}
```


## Metadata

### Instruments

The `get_instruments()` function gives information on all the CDI instruments in Wordbank.
```{r}
get_instruments()
```


### Datasets

The `get_datasets()` function gives information on all the datasets in Wordbank, either for a specific language and/or form or for all instruments. If the `admin_data` argument is set to `TRUE`, the results will also include the number of administrations in the database from that dataset.
```{r}
get_datasets(form = "WG")
get_datasets(language = "Spanish (Mexican)", admin_data = TRUE)
```


## Advanced functionality: Age of acquisition

The `fit_aoa()` function computes estimates of items' age of acquisition (AoA). It needs to be provided with a data frame returned by `get_instrument_data()` -- one row per administration x item combination, and minimally the columns `age` and `num_item_id`. It returns a data frame with one row per item and an `aoa` column with the estimate, preserving and item-level columns in the input data. The AoA is estimated by computing the proportion of administrations for which the child understands/produces (`measure`) each word, smoothing the proportion using `method`, and taking the age at which the smoothed value is greater than `proportion`.
```{r}
if (!is.null(animal_data)) {
  fit_aoa(animal_data)
  fit_aoa(animal_data, method = "glmrob", proportion = 1/3)
}
```


## Advanced functionality: Cross-linguistic data

One of the item-level fields is `uni_lemma` ("universal lemma"), which is intended to be an approximate semantic mapping between words across the languages in Wordbank. The function `get_crossling_items()` simply gives all the available `uni_lemma` values.
```{r}
get_crossling_items()
```

The function `get_crossling_data()` takes a vector of `uni_lemmas` and returns a data frame of summary statistics for each item mapped to that uni_lemma in any language (on `WG` forms). Each row is combination of item and age, and the columns indicate the number of children (`n_children`), means (`comprehension`, `production`), standard deviations (`comprehension_sd`, `production_sd`), and item-level fields.
```{r, eval=FALSE}
get_crossling_data(uni_lemmas = c("hat", "nose")) %>%
  select(language, uni_lemma, item_definition, age, n_children, comprehension,
         production, comprehension_sd, production_sd) %>%
  arrange(uni_lemma)
```


## Data versions and reproducibility

Wordbank is updated as researchers contribute new datasets and as errors are corrected. Each update is published as a new version of the Redivis dataset (`v1.5`, `v2.0`, ...). Released versions are immutable and stay available permanently.

Every data function takes a `version` argument. The default, `"current"`, is the most recent release, so results can change when Wordbank is updated. Pin the version to make an analysis reproducible:
```{r}
instruments_v2 <- get_instruments(version = "v2.0")
```

Every result has a `dataset_version` column recording the release it came from. With the default `version = "current"`, this is resolved to the actual version tag, so you can always tell which release you were given:
```{r}
if (!is.null(instruments_v2)) unique(instruments_v2$dataset_version)
```

The `wb_dataset()` function returns a reference to the Redivis dataset itself, which you can use to see which versions exist:
```{r, eval=FALSE}
versions <- wb_dataset()$list_versions()
sapply(versions, function(v) v$properties$tag)
```

Version numbers follow the data rather than the package. A minor bump (`v1.4` to `v1.5`) adds or corrects data without changing the structure of what `wordbankr` returns. A major bump (`v1.x` to `v2.0`) changes the structure of the underlying tables; `wordbankr` absorbs these changes so that its output stays the same, and any differences that do reach users are listed in the package's NEWS.

When you report an analysis, cite both the package version (`packageVersion("wordbankr")`) and the dataset version you used.