Using Julia inside targets

This vignette is the reference for running Julia scripts as targets steps with JuliaCall: every tar_target_jl() argument, how to choose the Julia install and project, and the common use cases. It mirrors vignette("python"): the shape of a Julia step is identical to a Python one, only the bridge and the environment arguments differ. For a gentler tour start with vignette("get_started").

Code blocks are illustrative and not executed when the vignette builds.

The two constructors

Both return a single targets target and forward every targets::tar_target_raw() argument (pattern, format, iteration, deployment, resources, cue, …).

These constructors depend on the JuliaCall R package. As a result, any limitation of JuliaCall is inherited by the constructors. Reading JuliaCall’s own documentation is recommended.

Arguments

Scripts, data, and output

These behave exactly as for Python (see vignette("python")):

Argument Meaning
script Path to the Julia script to run (required). Runs in the Main module.
pre_script Optional R script run before the Julia script. Assign a named list to_jl to push objects into Main.
post_script Optional R script run after the Julia script. jl_get() and jl_call() are available; its last expression is the value (object mode) or file paths (file mode).
inputs Named vector mapping in-step names to upstream targets, e.g. c(df = "prepared").
output "object" (default) or "file".
retrieve Julia variable name(s) to return when there is no post-script (object mode).
files Paths to return when there is no post-script (file mode).

As with Python, any of script / pre_script / post_script may be a literal path or a tar_target_path("name") reference to track the file.

Choosing the Julia install / project

Argument Meaning
julia_version A version string (e.g. "1.11"), resolved to a juliaup-managed install. Used when julia_home is not given.
julia_home The directory containing the julia executable. Defaults to getOption("tarpolyglot.julia_home"); when unset and no julia_version, JuliaCall discovers Julia on PATH.
julia_project A Julia project environment (folder with Project.toml / Manifest.toml) to Pkg.activate(). When NULL, Julia’s default global environment (@v#.#) is used.
julia_packages Character vector of packages to using before running the script.

The three-script model

 upstream targets ─► pre_script (R) ─► script (.jl) ─► post_script (R) ─► target value
                     builds `to_jl`     computes         reads jl_get("name") /
                                        `result`         jl_call(fn, ...)
  1. script: the Julia file, run in Main.
  2. pre_script: the inputs are already bound by name. Assign a named list to_jl; each element is julia_assign()ed as a variable in Main.
  3. post_script: JuliaCall has no py-style proxy, so you read variables back through jl_get("name") (a shortcut for JuliaCall::julia_eval("name")) and call Julia functions with jl_call (an alias of JuliaCall::julia_call()).

Object output

jl/stats.jl:

# `x` was pushed from R by the pre-script.
seq = isa(x, AbstractVector) ? x : [x]
result = Dict("sum" => sum(seq), "n" => length(seq), "mean" => sum(seq) / length(seq))

R/pre_push.R:

to_jl <- list(x = x)   # `x` came from inputs = c(x = "prepared_x")

R/post_result.R:

res <- jl_get("result")
data.frame(sum = res$sum, n = res$n, mean = res$mean)   # last expression = value

_targets.R:

library(targets)
library(tarpolyglot)

list(
  tar_target(prepared_x, c(1, 2, 3, 4)),

  # (a) return a Julia variable directly with `retrieve` (no post-script)
  tar_target_jl(
    name = jl_direct,
    script = "jl/stats.jl",
    inputs = c(x = "prepared_x"),
    pre_script = "R/pre_push.R",
    retrieve = "result"
  ),

  # (b) reshape the result in a post-script
  tar_target_jl(
    name = jl_prepost,
    script = "jl/stats.jl",
    inputs = c(x = "prepared_x"),
    pre_script = "R/pre_push.R",
    post_script = "R/post_result.R"
  )
)

File output

Set output = "file" when the script writes to disk. The target format becomes "file" automatically, and the step has to report the path(s) it wrote. There are two ways to do that, covered in turn below.

Returning the path from the post-script

Use this when the script chooses the path, for example a temporary file whose name is not known in advance.

jl/write.jl: writes x to a file, keeps the path in out_path for the post-script to read back:

out_path = tempname() * ".csv"
open(out_path, "w") do io
    println(io, join(x, ","))
end

R/post_files.R:

jl_get("out_path")
tar_target_jl(
  name = jl_file,
  script = "jl/write.jl",
  inputs = c(x = "prepared_x"),
  pre_script = "R/pre_push.R",
  post_script = "R/post_files.R",    # returns jl_get("out_path")
  output = "file"
)

Declaring the path up front with files

Use this when the path is already known as you write _targets.R. The script writes to that fixed location, files declares it, and no post-script is needed at all.

jl/write_fixed.jl:

mkpath("out")
open("out/values.csv", "w") do io
    println(io, join(x, ","))
end
tar_target_jl(
  name = jl_fixed,
  script = "jl/write_fixed.jl",
  inputs = c(x = "prepared_x"),
  pre_script = "R/pre_push.R",
  output = "file",
  files = "out/values.csv"           # no post-script needed
)

files and post_script are alternatives, not complements. When both are given the post-script wins and files is silently ignored:

Supplied Result
files only the paths in files are returned
post_script only the value of the post-script’s last expression is returned
both the post-script wins, files is ignored
neither error: output = "file" needs a post_script returning paths, or files

Dynamic branching (iris example)

jl/fit.jl: df (a per-species subset of iris, auto-converted to a Julia DataFrame) was pushed from R by the pre-script:

result = sum(df[!, "Petal.Length"]) / nrow(df)

R/pre_push_jl.R:

to_jl <- list(df = df)   # `df` came from inputs = c(df = "iris_groups")
list(
  tar_target(iris_groups, split(iris, iris$Species), iteration = "list"),
  tar_target_jl(
    name = fit_by_group,
    script = "jl/fit.jl",
    inputs = c(df = "iris_groups"),
    pre_script = "R/pre_push_jl.R",
    retrieve = "result",
    pattern = map(iris_groups),      # one branch per species
    iteration = "list"
  )
)

Run the branches in parallel by setting a crew controller (see vignette("get_started"), the “Parallelism and isolation with crew” section, which also covers polyglot_controller(log = polyglot_log(...)) for per-step stdout/stderr log files, including a caveat specific to branching there, since every branch of a pattern shares one log file).

Parallel computing (Distributed, not Threads)

Julia’s native Threads.@threads multi-threading does not work through JuliaCall: Threads.nthreads() reports 1 from a run_jl_step() session regardless of the JULIA_NUM_THREADS environment variable, confirmed set both to a fixed number and to "auto" before JuliaCall::julia_setup() runs. This is a limitation of how JuliaCall embeds Julia, not something tar_target_jl() or deployment can work around.

Distributed, Julia’s process-based parallelism, works correctly instead, confirmed both in the main session and inside a real crew worker:

# jl/parallel_sqrt.jl
using Distributed
addprocs(n_workers)
@everywhere sqrt_one(v) = sqrt(v)
result = pmap(sqrt_one, x)
tar_target_jl(
  name = jl_parallel,
  script = "jl/parallel_sqrt.jl",
  inputs = c(x = "big_x", n_workers = "n_workers"),
  pre_script = tar_code({ to_jl <- list(x = x, n_workers = n_workers) }),
  retrieve = "result"
)

addprocs() spawns genuine separate julia processes, so the same oversubscription concern from vignette("get_started") applies: a crew worker has access to every core on the machine, not a restricted slice of it, so several workers each calling addprocs() at once can oversubscribe the machine. Put the parallel-heavy step on deployment = "main" if you have more than one, or cap n_workers per step, for the same reason described there.

Choosing the Julia install and project

The use cases below are about the installation-selection arguments, not what each script computes.

Use case 1: Julia on PATH (default)

Set nothing; JuliaCall discovers the julia on PATH and uses the global environment. jl/probe.jl:

result = Sys.BINDIR
tar_target_jl(
  name = probe,
  script = "jl/probe.jl",
  retrieve = "result"
)

Use case 2: a specific juliaup version

Reuses jl/fit.jl from “Dynamic branching” above as a stand-in script (without inputs here, so df is undefined, the point of this example is julia_version, not the computation):

tar_target_jl(
  name = fit,
  script = "jl/fit.jl",
  julia_version = "1.11",            # resolved via juliaup
  retrieve = "result"
)

Use case 3: an explicit Julia home

If discovery fails (e.g. a fresh install not yet on PATH, or the Windows juliaup shim), point at the bin directory. You can set it once globally:

options(tarpolyglot.julia_home = "C:/Users/me/.julia/juliaup/.../bin")

or per step, again reusing jl/fit.jl as a stand-in:

tar_target_jl(
  name = fit,
  script = "jl/fit.jl",
  julia_home = "C:/Users/me/.julia/juliaup/.../bin",
  retrieve = "result"
)

Tracking scripts as dependencies

Reuses jl/fit.jl from “Dynamic branching” above as a stand-in script (again without inputs = c(df = ...), so df is undefined here, the point of this example is tar_target_path(), not the computation):

list(
  tar_target(fit_jl, "jl/fit.jl", format = "file"),

  tar_target_jl(
    name = fit,
    script = tar_target_path("fit_jl"),   # re-runs when jl/fit.jl changes
    inputs = c(x = "data"),
    retrieve = "result"
  )
)

Conversion caveats

See ?tar_target_jl and ?run_jl_step for the full argument reference.