---
title: "Pipeline views"
output:
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 4
description: >
  How to filter pipelines with `pip_view()` and the `[` operator, compose
  views, and run only a subset of steps.
vignette: >
  %\VignetteIndexEntry{Pipeline views}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r knitr-setup, include = FALSE}
knitr::opts_chunk$set(
    comment = "#",
    prompt = FALSE,
    tidy = FALSE,
    cache = FALSE,
    collapse = TRUE
)

old <- options(width = 100L)
```

Pipelines can get long, and often you want to focus on a subset of steps:
a topic, a stage, or the steps that produce the outputs you care about.
*Views* are {pipeflow}'s way of working on a subset of steps without
copying anything. A view references the underlying pipeline, so every
operation applied to a view (running it, updating parameters, tagging,
locking, ...) writes through to the original pipeline, restricted to the
steps covered by the view.

This vignette shows how to create and combine views, how to select steps
with the `[` operator, how views behave when the pipeline changes, and how
to run only part of a pipeline.

## Setup

Again, we use a very simplified example of a pipeline.

```{r setup-pipeline}
library(pipeflow)

pip <- pip_new("my-pip") |>
    pip_add("load", \(n = 5) seq_len(n), tags = c("io", "daily")) |>
    pip_add("clean", \(x = ~load) x * 2, tags = c("io", "core")) |>
    pip_add("fit", \(x = ~clean) sum(x), tags = c("model", "core")) |>
    pip_add("report", \(x = ~fit) paste("result:", x), tags = "report")
```

As you see above, besides `step` and `fun`, `pip_add()` also allows to set
tags. We will use these tags as meta information to filter certain steps by
topic and/or output type. Let's do a first run before we move on.

```{r first-run}
(pip_run(pip, lgr = NULL))
```

## Creating views

`pip_view()` returns a *view* of the pipeline that contains only the
steps matching the given filters:

```{r create-view}
pip_view(pip, tags = "core")
```

Filters can be combined. By default, steps must match *all* filters
(logical AND), while the values within a single filter are treated as
alternatives (OR):

```{r combine-filters}
pip_view(pip, tags = "core", state = "done")

pip_view(pip, step = c("clean", "fit"))
```

`join = "union"` keeps steps that match *any* filter:

```{r union-filters}
pip_view(pip, tags = "report", step = "clean", join = "union")
```

With `fixed = FALSE`, filter values are interpreted as regular
expressions:

```{r regex-filters}
pip_view(pip, step = "^f", fixed = FALSE)
```

The available filters are `step`, `params`, `state`, `exec`, `tags`, and
`depends`. For example, to find all steps that depend on `load` and are
still `new`:

```{r filter-depends}
pip_reset(pip) # reset to initial state

pip_view(pip, depends = "load", state = "new")
```

## Selecting steps with `[`

The extract operator `[` provides a data.table-like way of selecting
steps. It returns a view by default:

```{r select-steps}
pip_run(pip, lgr = NULL)
pip[c("load", "fit")]

pip[1:3]
```

Boolean filters are evaluated in the context of the step table, so the
same columns as in `pip_view()` are available as variables:

```{r boolean-filters}
pip[tags %like% "core"]

pip[step %in% c("clean", "fit") & state == "done"]
```

Note that `%like%` matches the tags of each step as a single string, such
as `'c("model", "core")'`, or `"character(0)"` for a step without tags. A
pattern can therefore match more steps than intended, so to select steps
by their tags, `pip_view(tags = )` is the safer choice.

Negative indices select all steps except the excluded ones:

```{r negative-indices}
pip[-2]
```

`p[]` returns a copy of the pipeline, and using two indices
(`p[i, j]`) extracts the given rows and columns as a `data.table`:

```{r copy-and-columns}
pip2 <- pip[]

pip[, c("step", "tags")]

pip[c("load", "fit"), "out"]
```

While `[` returns a view by default, `view = FALSE` builds a new,
self-contained pipeline containing the selected steps together with all
their upstream dependencies:

```{r extract-pipeline}
pip[c("fit", "report"), view = FALSE]
```

The printed message tells you how many steps were pulled in as upstream
dependencies.

## Composing views

Views can be nested: applying `pip_view()` (or `[`) to a view narrows
the view further.

```{r compose-views}
v1 <- pip_view(pip, tags = "io") # load, clean
v1

v2 <- v1 |> pip_view(tags = "core") # clean only
v2
```

## The `view` meta field

Under the hood, a view stores the steps it covers in the `view` meta field
of the pipeline object (`NULL` for a full pipeline). Reading the field
returns the names of the covered steps, so in principle you can also
manipulate the view directly by assigning step names to the meta field^[
    Direct manipulation of the view usually is not needed and probably
    mostly useful for debugging.
].

```{r view-meta-field}
pip[["view"]] # NULL — not a view

w <- pip
w[["view"]] <- c("load", "clean")
w

w[["view"]] <- NULL # back to the full pipeline
w
```

## When the pipeline changes

A view keeps covering its steps when the pipeline changes later, no matter
whether the change is made through the view, another view, or the pipeline
itself. Renamed steps stay in the view under their new name, and removed
steps drop out of it. New steps are not added to an existing view, even if
they match the view's filters.

```{r view-after-changes}
q <- pip_clone(pip)
v <- pip_view(q, tags = "core")
v

# A new "core" step is not added to the existing view
pip_add(q, "check", \(x = ~clean) all(x > 0), tags = "core", after = "clean")

# A renamed step stays in the view under its new name
pip_rename(q, from = "fit", to = "model")
v

# Removed steps drop out of the view
pip_remove(q, "model", force = TRUE)
v
```

## Running views

Running a view executes the covered steps together with any upstream
dependencies that are not up to date. The run log marks steps that belong
to the view as `[view]` and steps that were pulled in as dependencies as
`[upstream]`:

```{r run-view}
pip_reset(pip)
pip_run(pip_view(pip, step = "report"))
```

Afterwards, the original pipeline is up to date for the covered steps.

Pipeline views are not only useful to inspect or run certain parts
of the pipeline but also to filter and collect the final output of
your analysis run. For more details on this see the next vignette
[Collect and group output](v04-collect-output.html).

```{r restore-options, include = FALSE}
options(old)
```