---
title: "Get started with tbl.now"
output: rmarkdown::html_vignette
bibliography: references.bib
link-citations: true
vignette: >
  %\VignetteIndexEntry{Get started with tbl.now}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  message = FALSE,
  warning = FALSE,
  # `html_vignette` renders at 96 dpi, which for the wide multi-panel figures
  # below produces PNGs wider than the 700px the vignette CSS will ever show
  # and inflates the installed size of `doc/` for pixels nobody sees. 72 dpi
  # keeps every figure at essentially its displayed width.
  dpi = 72
)
options(tibble.width = Inf, width = 120, pillar.width = 120)
options(pillar.print_max = 6, pillar.print_min = 6,
        tibble.print_min = 6, dplyr.print_min = 6,
        tibble.print_max = 6, dplyr.print_max = 6)
```

## Why nowcasting needs a tbl.now

A case that happened yersterday is not necessarily in yesterday's data report. It might have been reported today or tomorrow or weeks later. Until that happens, until it gets reported, the most recent estimates of every epidemic curve _underestimate_ for no epidemiological reason. **Nowcasting** estimates what those recent dates look like by taking into account both the overall epidemic (epidemic process) and the delay distribution (reporting-delay process). 

To do so, one needs to carry **two** time indices at once:

1. The `event_date` describing when the event happened, and

2. The `report_date` describing when it was reported. 

Traditional tidyverse time-series classes
([tsibble](https://tsibble.tidyverts.org/),
[timetk](https://business-science.github.io/timetk/)) assume a single date
[@wang2020new; @timetk]. `tbl.now` is an extension of the [tibble](https://tibble.tidyverse.org/)
[@tidyverse; @wickham2014tidy] that is aware of which columns represent the event
date as well as the report date, the units (days, weeks, etc), the type of data (linelist vs counts), the now of the nowcast, as well as other important columns. The main advantage of `tbl.now` is that, being a `tibble`, it can work within any `dplyr` pipeline.

## The nowcasting workflow

Everything in the package depends on the `tbl.now` object. One has to declare it once and then everything follows:

```{r workflow-diagram, echo = FALSE, fig.width = 7, fig.height = 6, out.width = "100%", fig.alt = "Flow diagram read top to bottom. Clean your data leads to tbl_now(), which leads to diagnose() and summary(), which lead to autoplot(). A dashed arrow loops back from diagnose() to the cleaning step, labelled fix what it finds. Below autoplot() the path forks in two: left to tbl_now_to_*(), labelled hand it to another package, and right to run_nowcast(engine()), which leads on to nowcast_backtest() and score_nowcast()."}
library(ggplot2)
pal <- tbl.now::tbl_now_palette()

boxes <- data.frame(
  x     = c(5.0, 5.0, 5.0, 5.0, 2.2, 7.8, 7.8),
  y     = c(9.3, 7.7, 6.1, 4.5, 2.6, 2.6, 0.9),
  hw    = c(1.9, 1.3, 1.9, 1.3, 2.0, 2.1, 2.1),
  hh    = c(0.55, 0.4, 0.55, 0.4, 0.4, 0.4, 0.55),
  label = c(
    "Clean your data\n(dplyr, as usual)", 
    "State the `tbl_now()`",
    "Diagnose data problems\n`diagnose()`",
    "Describe\n(`summary()`, `autoplot()`, ...)",
    "tbl_now_to_*()", 
    "Nowcast\n`run_nowcast(engine())`",
    "Evaluate\n`nowcast_backtest()`\n`score_nowcast()`"
  ),
  fill = c(
    pal[["reporting_light"]], pal[["reporting_light"]], pal[["reporting_light"]],
    pal[["reporting_light"]], pal[["reporting_light"]], pal[["reporting_light"]],
    pal[["reporting_light"]]
  ),
  ink = c(
    pal[["ink"]], pal[["ink"]], pal[["ink"]],
    pal[["ink"]], pal[["ink"]], pal[["ink"]],
    pal[["ink"]]
  )
)

arrows <- data.frame(
  x    = c(5.0, 5.0, 5.0, 4.6, 5.4, 7.8),
  y    = c(8.75, 7.30, 5.55, 4.10, 4.10, 2.20),
  xend = c(5.0, 5.0, 5.0, 2.6, 7.4, 7.8),
  yend = c(8.25, 6.65, 4.90, 3.10, 3.10, 1.45)
)

ggplot() +
  geom_segment(
    data = arrows, aes(x = x, y = y, xend = xend, yend = yend),
    colour = pal[["guide_strong"]], linewidth = 0.45,
    arrow = arrow(length = unit(0.18, "cm"), type = "closed")
  ) +
  geom_curve(
    aes(x = 3.05, y = 6.1, xend = 3.05, yend = 9.0),
    colour = pal[["guide_strong"]], linewidth = 0.4, linetype = "22",
    curvature = -0.55, ncp = 12,
    arrow = arrow(length = unit(0.15, "cm"), type = "closed")
  ) +
  annotate("text", x = 1.35, y = 7.6, angle = 90, label = "fix what it finds",
           colour = pal[["ink"]], size = 3, fontface = "bold") +
  annotate("text", x = 2.2, y = 1.7, label = "hand it to another package",
           colour = pal[["ink"]], size = 3, fontface = "bold") +
  geom_label(
    data = boxes, aes(x = x, y = y, label = label),
    colour = boxes$ink, fill = boxes$fill, 
  ) +
  scale_x_continuous(limits = c(-0.1, 10.1)) +
  scale_y_continuous(limits = c(0.1, 10.0)) +
  theme_void()
```

This vignette follows the nowcasting workflow at speed. We provide links and a deeper tutorial in each section. Our main goal here is for you to understand what `tbl.now` is about and how we implement the nowcasting workflow. 

## The data

For this tutorial, we will use `denguedat`, a weekly dengue line-list that comes with the
package:

```{r setup, message = FALSE}
library(dplyr)
library(tbl.now)

data(denguedat)
```

The dataset is a linelist with `onset_week` representing when symptoms appear, `report_week`, when they reached the surveillance system, and the `gender` of each individual (one individual = one line):  

```{r, eval = FALSE}
denguedat
```

```{r, echo = FALSE}
tibble(denguedat)
```

The data has 20 years of dengue. For this analysis, we will focus on the 2005 season and assume we were just seeing data from that year and we are sitting at the first day of October 2005.

## 1. Create the `tbl_now`

To create a `tbl_now` one has to use the `tbl_now()` function and specify which column is the event, which is the report, and which columns are the strata (if applicable):

```{r declare, message = TRUE}
#For this example, we filter the data to keep only those cases that happened 
#on 2005 and were reported before October 2005
denguedat <- denguedat |>
  filter(onset_week >= as.Date("2005-01-01") & report_week <= as.Date("2005-10-01")) 

#We then create the tbl_now object
dengue <- denguedat |>
  tbl_now(
    event_date  = onset_week,  
    report_date = report_week
  )

dengue
```

`tbl_now()`  tells you what it had to infer (data is a `linelist`) on a
`weekly` grid, that `now` is the last report week in the data corresponding
to `r  get_now(dengue)`. In addition, it added the `.delay`, `.event_num` and `.report_num` columns which
is used downstream for nowcasts. 

This object is still a tibble. You can use `filter()`, `mutate()`, `rename()`, etcetera 
and the object keeps its attributes. See
[*More on the `tbl_now` object*](https://rodrigozepeda.github.io/tbl.now/articles/more-on-tbl-now.html)
for the full list of attributes and how each dplyr verb treats them.

## 2. Diagnose it

Before modelling anything, ask what is wrong with the data. `diagnose()` creates a report to show
you any potential problems with the data (for example the strata size, the estimated truncation, etc):

```{r diagnose}
diagnose(dengue)
```

Deeper diagnostics are covered in 
[*Diagnosing a `tbl_now`*](https://rodrigozepeda.github.io/tbl.now/articles/diagnosing-a-tbl-now.html)
and
[*Identifying reporting batches*](https://rodrigozepeda.github.io/tbl.now/articles/batches.html).

## 3. Describe it 

### Summarise the data

The `summary()` function describes your data. It returns the summaries for multiple data components: 

```{r summary}
summary(dengue) 
```


Here, among other things, we can see that half the cases are reported within a week of onset but the tail runs much longer. More information on summary is available [in the diagnosing article](https://rodrigozepeda.github.io/tbl.now/articles/diagnosing-a-tbl-now.html)

## Visualize the data

The `autoplot()` function lays the object as a grid showing how the epidemic and the 
delay processes behave:

```{r autoplot, fig.width = 9, fig.height = 7, out.width = "100%", fig.alt = "A grid of diagnostic panels for the dengue data, with the epidemic process in the left column and the reporting process in the right column."}
autoplot(dengue)
```

Every panel is also a function of its own and there are additional functions
such as `plot_reporting_triangle()`, `plot_reporting_hexamap()`, `plot_delay_drift()` 
which you can use to better characterize your data. See them in the [*Diagnosing a `tbl_now`*](https://rodrigozepeda.github.io/tbl.now/articles/diagnosing-a-tbl-now.html) vignette

## 4. Then: hand it over, or fit it here

### Hand it over

If you already use another nowcasting package, `tbl.now` probably speaks its dialect.
The `tbl_now_to_*()` converters go out, and `tbl_now_from_*()` (or
`as_tbl_now()`) come back. For example one can transform the `tbl.now` to a
reporting triangle:

```{r converters, warning = FALSE}
#Transform to a baselinenowcast reporting triangle
triangle <- tbl_now_to_baselinenowcast(dengue)

#This is now a reporting triangle:
triangle[(nrow(triangle) - 5):nrow(triangle), 1:7]
```

That is now a reporting triangle: its rows are onset weeks, columns are reporting delays
in weeks. It can then be used within the \pkg{baselinenowcast} framework:

```{r, eval=FALSE}
library(baselinenowcast)

#and nowcast within the framework
baselinenowcast(triangle)
```

There are converters for
[epinowcast](https://package.epinowcast.org/),
[EpiNow2](https://epiforecasts.io/EpiNow2/),
[baselinenowcast](https://baselinenowcast.epinowcast.org/),
[NobBS](https://CRAN.R-project.org/package=NobBS),
[surveillance](https://CRAN.R-project.org/package=surveillance),
[epidist](https://epidist.epinowcast.org/), `tsibble` and `data.table` --
see
[*Nowcasting with different models*](https://rodrigozepeda.github.io/tbl.now/articles/nowcasting-models.html).

### Or fit it here

The function `run_nowcast()` takes the object and an `engine()`, and returns the same
`tbl_nowcast` result whichever engine you picked. There are engines for each of the packages
mentioned above (except for `epidist`). Here we will use `example_engine()` -- a deliberately
simple nowcast that runs very fast for examples. However we recommend chainging it in real life for `engine_baselinenowcast()`, `engine_epinowcast()`, `engine_diseasenowcasting()` or another one.

```{r nowcast, fig.width = 9, fig.height = 4, out.width = "100%", fig.alt = "Nowcast of dengue cases by onset week and gender: observed counts as bars with the predicted median and interval overlaid on the most recent weeks."}
#Change to engine_diseasenowcasting() if possible
fit <- run_nowcast(dengue, example_engine())

#Visualize the results
autoplot(fit)
```

The result prints the estimate, interval at the last event date it covers. You can use 
`tidy()` to get a table of quantiles (or `as_tibble()` for all predictions):

```{r, eval = FALSE}
tidy(fit)
```

```{r, echo = FALSE}
tidy(fit) |> tail(5)
```

## 5. Evaluate your nowcast

The `nowcast_backtest()` function re-fits at a series of past `now` dates (each time ignoring everything that had not been reported by then)  and scores the predictions against what eventually arrived:

```{r backtest}
backtest <- nowcast_backtest(
  dengue,
  example_engine(),
  now_dates = as.Date(c("2005-08-07", "2005-09-04")), #Evaluate at these two dates
  verbose = FALSE
)

backtest
```

We can see the metrics which include the weighted interval score (lower is
better) and how often the 90% interval actually contained the truth. 


## 6. And create ensembles

Several nowcasts can be combined into one with `nowcast_ensemble()` which is in
general a more robust model. That is a topic of its own, and it has its own article:
[*Ensemble nowcasting*](https://rodrigozepeda.github.io/tbl.now/articles/ensemble-nowcasting.html).

## Where to go next

That is the whole workflow. If you want to see it done on real, messy
surveillance data from beginning to end, [the tutorial is the place to start](https://rodrigozepeda.github.io/tbl.now/articles/example.html).

```{r learning-more, echo=FALSE, results="asis"}
cat(
  knitr::knit_child(
    system.file("fragments", "learning-more.Rmd", package = "tbl.now"),
    quiet = TRUE
  ),
  sep = "\n"
)
```

## References