Controller groups

Each controller object only supports only one type of worker configuration which you set in advance. However, different controllers may have different types of workers, and crew supports controller groups to coordinate among these different worker types. With third-party launcher subclasses from other packages, this mechanism will allow you to e.g. send some tasks to GPU-capable or high-memory workers while other tasks go to low-spec workers.

We demonstrate with a controller of fully persistent workers which always stay running and a controller of semi-persistent workers which terminate after completing four tasks. We create controller objects with names.

library(crew)
persistent <- crew_controller_local(name = "persistent")
transient <- crew_controller_local(name = "semi-persistent", tasks_max = 4L)

crew uses a different TCP port for each controller you run, so please do not create hundreds of controllers. Please see the subsection on ports in the README.

We put these controller objects into a new controller group object.

group <- crew_controller_group(persistent, transient)

This controller group has a global connect() method to initialize both controllers.

group$start()

You can choose which worker pool to receive tasks.

group$push(name = "my task", command = sqrt(4), controller = "semi-persistent")

The controller group also supports global methods for wait(), pop(), and terminate(). These methods operate on all controllers at once by default, but the controllers argument allows you to select a subset of controllers to act on. Below in pop() the launcher column of the output indicates which controller ran the task.

group$wait(controllers = "semi-persistent")
group$pop()
#> # A tibble: 1 × 11
#>   name    command result seconds   seed error trace warnings launcher worker
#>   <chr>   <chr>   <list>   <dbl>  <int> <chr> <chr> <chr>    <chr>     <int>
#> 1 my task NA      <dbl>        0 6.30e8 NA    NA    NA       semi-pe…      1
#> # ℹ 1 more variable: instance <chr>

The map() method provides functional programming, and the controller argument lets you choose the controller to submit the tasks.

group$map(
  command = a + b + c + d,
  iterate = list(
    a = c(1, 3),
    b = c(2, 4)
  ),
  data = list(c = 5),
  globals = list(d = 6),
  controller = "persistent"
)
#> # A tibble: 2 × 11
#>   name  command result    seconds      seed error trace warnings
#>   <chr> <chr>   <list>      <dbl>     <int> <chr> <chr> <chr>
#> 1 1     NA      <dbl [1]>       0 870591596 NA    NA    NA
#> 2 2     NA      <dbl [1]>       0 870591595 NA    NA    NA
#> # ℹ 3 more variables: launcher <chr>, worker <int>,
#> #   instance <chr>

The controller group has a summary() method which aggregates the summaries of one or more controllers.

group$summary()
#> # A tibble: 2 × 6
#>   controller      worker tasks seconds errors warnings
#>   <chr>            <int> <int>   <dbl>  <int>    <int>
#> 1 persistent           1     2       0      0        0
#> 2 semi-persistent      1     1       0      0        0

When you are finished, please call terminate() with no arguments to terminate all controllers in the controller group.

group$terminate()