---
title: "Getting started with lineager"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Getting started with lineager}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
library(lineager)
```

You build an analysis dataset. Along the way rows disappear — filtered
out, dropped in joins, excluded by criteria. Later someone asks:
*"Which records were removed, why, and what happened to row 42 between
source and analysis?"*

Without `lineager`, that answer requires manual reconstruction.
With `lineager`, it is a single function call.

`lineager` tags every row of every dataset with a unique lineage
identifier that survives filters, joins, and derivations. Every row
removal requires a documented reason. At any point, `lg_trace()` returns
any row's complete journey across the pipeline. `lg_report()` compiles
everything into a structured provenance document.

## 1. Start a session

All `lineager` state lives in a session store reset by `lg_start()`.
Call it once at the top of your analysis.

```{r start}
lg_start(study_id = "PROJECT-001", analysis_id = "primary")
```

The optional `study_id` and `analysis_id` appear in reports. They can be
any strings — or omitted entirely.

## 2. Tag your source datasets

`lg_tag()` is the entry point. It assigns a `lineage_id` (lineage ID) to
every row at position 1. This ID persists through all operations.

```{r tag-basic}
patients <- data.frame(
  USUBJID = c("P001", "P002", "P003", "P004", "P005", "P006"),
  age = c(34L, 19L, 52L, 28L, 61L, 44L),
  group = c("A", "B", "A", "B", "A", "B"),
  eligible = c(TRUE, FALSE, TRUE, TRUE, FALSE, TRUE),
  stringsAsFactors = FALSE
)

tagged <- lg_tag(patients,
  dataset_id = "PATIENTS",
  label = "Patient registry"
)

tagged
```

The lineage ID format embeds the dataset ID and a zero-padded sequence.
When a `USUBJID` column is present (CDISC datasets), it is also embedded
for human readability:

```
PATIENTS_000001   ← non-CDISC: dataset + sequence
DM_0001_01-042    ← CDISC: dataset + sequence + USUBJID
```

### Tagging multiple datasets

Tag all source datasets before any transformations:

```{r tag-multiple}
labs <- data.frame(
  USUBJID = c("P001", "P001", "P003", "P004", "P006"),
  test = c("ALT", "AST", "ALT", "ALT", "ALT"),
  value = c(28.4, 31.2, 45.1, 22.8, 38.6),
  stringsAsFactors = FALSE
)

labs_tagged <- lg_tag(labs, dataset_id = "LABS", label = "Laboratory results")

cat("Patients tagged:", nrow(tagged), "rows\n")
cat("Labs tagged:    ", nrow(labs_tagged), "rows\n")
```

## 3. Derive new variables

`lg_derive()` works like `dplyr::mutate()` but requires a `description`
argument that is recorded in the operation log.

```{r derive-basic}
derived <- lg_derive(tagged,
  age_group = ifelse(age >= 40L, ">=40", "<40"),
  adult = age >= 18L,
  description = "age_group: >=40 vs <40 from age; adult: age >= 18"
)

derived[, c("USUBJID", "age", "age_group", "adult")]
```

The `lineage_id` column is preserved unchanged through derivations:

```{r lid-check}
all(derived[["lineage_id"]] == tagged[["lineage_id"]])
```

Chain derivations naturally:

```{r derive-chain}
derived2 <- lg_derive(derived,
  label = paste0(USUBJID, " (", group, ")"),
  description = "Display label combining USUBJID and group"
)

derived2[, c("lineage_id", "USUBJID", "group", "label")]
```

Each call adds one `DERIVE` operation to the session log.

## 4. Join datasets with lineage tracking

`lg_join()` performs a tracked join. It preserves `lineage_id` from the
left dataset and adds `lineage_id_y` to record which rows from the right
dataset contributed — enabling bilateral tracing.

```{r join-left}
joined <- lg_join(tagged, labs_tagged,
  by          = "USUBJID",
  type        = "left",
  description = "Merge ALT lab values from LABS onto PATIENTS"
)

joined[, c("lineage_id", "USUBJID", "eligible", "test", "value", "lineage_id_y")]
```

The `lineage_id_y` column shows which LABS row contributed to each PATIENTS
row. Rows with no matching lab record have `NA` in `lineage_id_y`.

Supported join types:

```{r join-types, eval = FALSE}
lg_join(x, y, by = "USUBJID", type = "left") # all rows of x
lg_join(x, y, by = "USUBJID", type = "inner") # only matching rows
lg_join(x, y, by = "USUBJID", type = "full") # all rows of both
lg_join(x, y, by = "USUBJID", type = "right") # all rows of y
```

`"left"` and `"full"` never drop rows of `x`, so `description` stays
optional for them. `"inner"` and `"right"` can drop unmatched `x` rows —
the moment a drop actually happens, `lg_join()` requires a `description`
(used as the exclusion reason for those dropped rows) and errors if one
isn't supplied. If no rows end up dropped, `description` remains optional
even for `"inner"`/`"right"`.

## 5. Filter with mandatory exclusion reasons

`lg_filter()` works like `dplyr::filter()` but the `reason` argument is
**mandatory with no default**. Every row removal must be documented.

Excluded rows and their IDs are captured in the session exclusion
registry automatically.

```{r filter-basic}
eligible_only <- lg_filter(tagged,
  eligible == TRUE,
  reason = "Not eligible for analysis (eligible != TRUE)"
)

cat("Before:", nrow(tagged), "\n")
cat("After: ", nrow(eligible_only), "\n")
```

Optional arguments enrich the exclusion record:

| Argument | Purpose | Example |
|---|---|---|
| `reason` | Why these rows are excluded (required) | `"Under age threshold"` |
| `reason_code` | Short controlled-vocabulary code | `"UNDERAGE"` |
| `population` | Which population/cohort this relates to | `"ANALYSIS_SET"` |

```{r filter-enriched}
# reason_code and population enrich the exclusion record
step1 <- lg_filter(tagged,
  eligible == TRUE,
  reason = "Screening criteria not met (eligible != TRUE)",
  reason_code = "SCREEN_FAIL",
  population = "ELIGIBLE_SET"
)

step2 <- lg_filter(step1,
  age >= 18L,
  reason = "Under minimum age threshold (age < 18)",
  reason_code = "UNDERAGE",
  population = "ADULT_SET"
)

cat("Enrolled: ", nrow(tagged), "\n")
cat("Eligible: ", nrow(step1), "\n")
cat("Adult:    ", nrow(step2), "\n")
```

## 6. The session operation log

Every `lg_derive()`, `lg_join()`, and `lg_filter()` call adds an entry
to the session operation log. Retrieve it with `lg_operations()`:

```{r operations}
ops <- lg_operations()
ops[, c("op_id", "op_type", "description", "rows_in", "rows_out")]
```

The operation log is the backbone of the provenance report — it shows
the complete sequence of transformations applied to the data.

`lg_operations()` returns the log for the **whole session** — every
dataset, every operation. When you only want the history behind one
specific object, `lg_history()` returns just the operations that produced
it, in order:

```{r history}
lg_history(step2)
```

`step2` above went through two `lg_filter()` calls (eligibility, then
age), so `lg_history(step2)` returns both of those operation records —
useful when you have several derived objects in scope and want to check
what actually produced a particular one, without filtering the full
session log by hand.

## 7. End the session

```{r end}
lg_end()
```

`lg_end()` marks the session inactive and prints a summary. The store
remains in memory and is still queryable via `lg_trace()`,
`lg_exclusions()`, etc. until `lg_start()` is called again.

## 8. Complete minimal workflow

```{r complete}
lg_start(study_id = "DEMO")

# Source data
raw <- data.frame(
  id = sprintf("P%03d", 1:8),
  value = c(12.4, NA, 8.1, 15.2, 9.8, NA, 11.3, 7.4),
  group = rep(c("treatment", "control"), 4),
  include = c(TRUE, TRUE, FALSE, TRUE, TRUE, TRUE, FALSE, TRUE),
  stringsAsFactors = FALSE
)

# Tag, derive, filter
ds <- lg_tag(raw, dataset_id = "RAW", label = "Raw analysis dataset")

ds <- lg_derive(ds,
  log_value = log(value),
  value_cat = ifelse(!is.na(value) & value >= 10, "high", "low/missing"),
  description = "Log-transform value; categorise as high (>=10) vs low/missing"
)

ds_clean <- ds |>
  lg_filter(include == TRUE,
    reason = "Excluded by study protocol (include != TRUE)"
  ) |>
  lg_filter(!is.na(value),
    reason = "Missing primary endpoint value"
  )

cat("Rows after cleaning:", nrow(ds_clean), "\n")

# Visualise the pipeline
lin <- lg_lineage()
print(lin)

lg_end()
```

## 9. Visualise the pipeline

`lg_lineage()` builds a complete lineage graph of the pipeline —
every dataset, operation, and exclusion branch — and returns a Graphviz
DOT string. Render it inline or export to a file.

```{r lineage-demo}
lg_start()
raw <- lg_tag(
  data.frame(
    USUBJID = sprintf("P%02d", 1:6),
    group = rep(c("A", "B"), 3L),
    flag = c(TRUE, TRUE, FALSE, TRUE, FALSE, TRUE),
    stringsAsFactors = FALSE
  ),
  dataset_id = "RAW"
)
raw <- lg_derive(raw,
  group_n = ifelse(group == "A", 1L, 2L),
  description = "Numeric group code"
)
lg_filter(raw, flag == TRUE, reason = "Flag not set")

lin <- lg_lineage()
print(lin)
lg_end()
```

```{r lineage-plot, eval = FALSE}
# Render inline (requires DiagrammeR)
lg_plot(lin)

# Export DOT file for Graphviz / online renderers
lg_plot(lin, output = "outputs/pipeline.dot")
```

**Node colour legend:**

| Colour | Shape | Meaning |
|---|---|---|
| Blue | Box | Source dataset from `lg_tag()` |
| Yellow | Ellipse | `lg_derive()` operation |
| Green | Diamond | `lg_join()` operation |
| Orange | Ellipse | `lg_filter()` operation |
| Red label | — | Rows excluded at that filter step |
| White | Box | Dataset state after each operation |

The next two vignettes cover exclusion tracking and reporting in depth:

- `vignette("exclusion-tracking")` — `lg_filter()`, `lg_exclusions()`,
  `lg_disposition()`, and `lg_trace()` with detailed examples
- `vignette("populations-and-reporting")` — `lg_population()`,
  `lg_spec()`, `lg_report()`, and `lg_lineage()` / `lg_plot()` for
  structured documentation and pipeline visualisation