Resources and objects

A resource is one loadable unit, such as a directory or file. Opening a resource produces one or more named R objects. Resource names, file names, and object names are separate concepts and need not match. Catalog and conflict resolution work on the recorded mapping from resources to objects, and an unknown mapping stays unknown until it is inspected.

The loader

An adapter contributes one or more loaders, built with data_loader(). Each loader has a namespaced identifier such as "tinydisk::tinymat" and up to three callbacks:

Only open may read data. Discovery and validation work from file names and small metadata and must not restore the dataset. This is what lets projectData build a catalog of a large collection without touching it.

When each operation runs

 install data package
        |
        v
 register collection ---- discovery (only for loaders that provide it,
        |                  and only if the resource is not in a manifest)
        |                 validation (if the loader is already registered)
        v
 catalog / data_query()   metadata only, nothing opened
        |
        v
 first access: data_get(), or a bare name in the attached environment
        |
        +-- loader missing? resolve it from the manifest's AdapterPackage
        +-- validation again, against the loader about to open
        +-- open()  -> objects cached under the resource
        v
 later access: cached; no further validation or opening

Discovery answers “what resources does this package contain?” It returns descriptors, which are plain lists with these fields:

Validation answers “is this descriptor well formed, and are the files it needs present?” It runs when a collection is registered, if the loader is already registered, and again before every uncached opening attempt against the loader selected for that attempt. This means a collection can be registered while an optional adapter is missing, and an adapter that arrives later is still validated before it opens anything. A rejection marks the resource failed without calling open; fix the problem and call data_retry().

Opening restores the resource when someone first asks for an object from it. The result is cached, so later access does not repeat validation or opening.

Manifests and discovery are alternatives

There are two ways a resource becomes known:

• Discovery. The adapter scans the package and reports what it finds, using a file-naming convention you define.
• A manifest. The data package author lists the resource in inst/projectData/manifest.dcf (section 5). The manifest describes the resource completely, so no adapter needs to discover it, and a loader without a discover callback works fine.

They can be combined. If both name the same resource, the manifest entry wins. Details are in the advanced section.

The format

A tinymat is a directory named <name>.tinymat/ with two files: dim.txt ("nrow ncol") and values.bin (column-major doubles). An opened tinymat is a small object holding the directory path and dimensions. It reads values from disk only when you subset it with [. That makes it a miniature version of a disk-backed matrix.

The loader declaration

projectData::data_loader(
  "tinydisk::tinymat",
  storage = "disk-backed",
  needs_work_dir = TRUE,
  discover = function(root, context) { ... },
  validate = function(descriptor, root, context) { ... },
  open = function(descriptor, paths, context) { ... }
)

The id has the form package::name, and the projectData:: prefix is reserved for built-in loaders. Using your package name as the prefix keeps ids stable and avoids collisions with other adapters.

storage = "disk-backed" tells projectData that the returned object reads from files lazily. The catalog and the first-access message then avoid suggesting that data are in memory. needs_work_dir = TRUE asks for a writable directory to be provided at opening, explained below.

Discovery

discover = function(root, context) {
  dirs <- list.files(file.path(root, "extdata"), pattern = "\\.tinymat$")
  lapply(sub("\\.tinymat$", "", dirs), function(nm) {
    list(resource = nm, source = paste0("extdata/", nm, ".tinymat"), objects = nm)
  })
}

root is the installed package root, where data/ and the contents of inst/ sit side by side. The callback uses an explicit convention (the .tinymat suffix), and it only lists names. It does not read dim.txt. projectData does not assume that every file under extdata/ is a public data object, so a convention like this is how an adapter avoids claiming unrelated files. Here the object is named after the resource, but that is a choice of this adapter, not a requirement.

Validation

validate = function(descriptor, root, context) {
  src <- file.path(root, descriptor$source)
  ok <- file.exists(file.path(src, "dim.txt")) && file.exists(file.path(src, "values.bin"))
  if (ok) TRUE else "tinymat directory is missing dim.txt or values.bin"
}

The validator checks that the files the format requires exist, and returns either TRUE or a message explaining why not. An error raised inside validate also rejects the descriptor. It never opens the data, so it is cheap enough to run before every opening attempt, and it turns a damaged installation into a clear message instead of a failure deep inside the format’s reader.

Opening

open = function(descriptor, paths, context) {
  src <- paths$source
  if (identical(descriptor$options$work_copy, "true")) {
    dest <- file.path(paths$work_dir, basename(src))
    file.copy(src, paths$work_dir, recursive = TRUE)
    src <- dest
  }
  stats::setNames(list(tinymat_open(src)), descriptor$objects)
}

projectData resolves the paths for you, so open never builds a library location itself:

• paths$root is the installed package root, and paths$source is the descriptor’s source resolved to an absolute path (NA if there is none).
• paths$work_dir is a directory that is unique to this opening, provided only when the loader declared needs_work_dir = TRUE (otherwise NULL). Installed files are read-only, so a loader that must write or copy anything does it here.
• context carries collection, package, version, library, root, and resource, which is useful for messages and diagnostics.
• descriptor$options carries the strings the data package author supplied. The example uses work_copy = "true" to demonstrate a private copy.

open must return a named list, with one element for each object the resource produces, so projectData can cache them together. Here, tinymat_open() reads only dim.txt and records the path. It returns a restored R object, but the values stay on disk until the user subsets them. Restoring an object is not the same as materializing its data; for a disk-backed format the first is cheap and the second is not. The catalog state ready therefore means only that the loader returned.

An adapter package is not a data package

An adapter package contains code for one format and has no data. A data package contains data and can say which adapter it needs, but contains no loader code. Many data packages can use one adapter, and the adapter can change without rebuilding them as long as the file format stays compatible.

Packaging the adapter

An adapter package exports a function that returns a data_loader(), or a list of them. By default projectData looks for an export named projectData_loaders:

# in the adapter package, e.g. tinydiskAdapter
#' @export
projectData_loaders <- function() tinydisk_loader()

Use a stable id with your package name as prefix ("tinydisk::tinymat"), because data packages record it. Put optional heavy dependencies in Suggests or Imports of the adapter, not of projectData or of the data packages.

Users register an adapter in one call:

data_loader_adapter("tinydiskAdapter")

data_loader_adapter() loads the namespace without attaching it, calls the export with no arguments, checks the whole declaration, and registers everything or nothing. It never calls discover, validate or open. It fails with a projectData_loader_unavailable error if the package or export is missing.

Declaring resources in a manifest

A data package author can describe resources in inst/projectData/manifest.dcf. A manifest holds data only: relative paths and loader references, never code.

ManifestVersion: 1

Resource: counts
Loader: tinydisk::tinymat
Source: extdata/counts.tinymat
Objects: counts
Options: work_copy=true
AdapterPackage: tinydiskAdapter
AdapterExport: projectData_loaders

Resource and Loader are required. Objects is a comma-separated list of names the resource produces, Options is key=value pairs separated by ;, and Title, HelpTopic, and RecordedClass are optional recorded metadata. AdapterPackage (and optionally AdapterExport, default projectData_loaders) tell projectData where to find the loader. If the loader is not registered when the object is first accessed, projectData loads that namespace (without attaching it) and registers the loader.

The same fields are described by the descriptor lists above, so a manifest is a serialization of descriptors. Because it holds no code, projectData can validate it without running anything.

You can write the file by hand, but projectData can also draft and write it. data_manifest_draft() asks the registered adapters what they would claim under a root, using discovery only. data_manifest_write() validates and writes the descriptors, then reads the text back to confirm it reproduces them. data_manifest_validate() checks an existing file.

pkg <- tempfile("pkg-")
dir.create(file.path(pkg, "data"), recursive = TRUE)
dir.create(file.path(pkg, "extdata"), recursive = TRUE)
example_obj <- 1:3
save(example_obj, file = file.path(pkg, "data", "example.rda"))
tinymat_write(matrix(as.double(1:6), 2, 3),
              file.path(pkg, "extdata", "counts.tinymat"))

drafts <- data_manifest_draft(pkg, layout = "standalone")
vapply(drafts, function(d) d$source, "")
#> [1] "extdata/counts.tinymat"

drafts <- lapply(drafts, function(d) {
  d$title <- "Counts"
  d$options <- list(work_copy = "true")
  d$adapter <- list(package = "tinydiskAdapter")
  d
})
manifest <- file.path(pkg, "projectData", "manifest.dcf")
data_manifest_write(drafts, manifest, root = pkg, layout = "standalone")
writeLines(readLines(manifest))
#> ManifestVersion: 1
#> 
#> Resource: counts
#> Loader: tinydisk::tinymat
#> Source: extdata/counts.tinymat
#> Objects: counts
#> Title: Counts
#> Options: work_copy=true
#> AdapterPackage: tinydiskAdapter

data_manifest_write(list(resource = "counts", loader = "tinydisk::tinymat",
                         source = "../elsewhere", objects = "counts"),
                    file.path(pkg, "other.dcf"), root = pkg, layout = "standalone")
#> Error: resource "counts": Source must be a relative path without '..' components or absolute/backslash forms; got "../elsewhere"

When the adapter is not installed

Data packages can use an adapter without making it a hard dependency. If the adapter named by a manifest is not installed, only the resources that need it are affected. The next example installs a package with an ordinary dataset and a tinymat whose manifest names an adapter that does not exist. The loader is removed first so projectData has to resolve it from the manifest.

data_loader_remove("tinydisk::tinymat")
fixture_package(lib, "mixedData", "1.0.0", lazy = FALSE,
  datasets = list(plain = list(x = 1:3)),
  files = list("extdata/m.tinymat" = function(path) {
    tinymat_write(matrix(1, 2, 2), path)
  }),
  manifest = c("ManifestVersion: 1", "",
               "Resource: m", "Loader: tinydisk::tinymat",
               "Source: extdata/m.tinymat", "Objects: m",
               "AdapterPackage: noSuchAdapter"))
data_attach("mixedData", lib.loc = lib)
data_get("x")
#> projectData: opening resource 'plain' of collection 'mixedData' (mixedData 1.0.0) with object(s) x [111 bytes on disk: data/plain.rda]. Loader projectData::dataset; opened from the installed package (projectData does not install packages).
#> [1] 1 2 3

data_get("m")
#> Error: Adapter package 'noSuchAdapter' is not installed. projectData never installs packages automatically; install it and retry. (needed for loader 'tinydisk::tinymat', resource 'm' of collection 'mixedData'). Other objects are unaffected.

data_catalog()[, c("exposed_name", "origin", "state")]
#> <projectData catalog>: 2 row(s); metadata only, no data opened.
#>  exposed_name   origin              state
#>             x    index              ready
#>             m manifest unavailable-loader
data_detach()

The ordinary dataset opened normally and the catalog marks the other resource unavailable-loader. Installing the adapter and retrying fixes it without re-registering the collection.

Resource identity and caching

Each resource has one record per collection. Aliases (suffix names), the unqualified name, and data_get(collection = ) all point to that record, so a successful opening happens once and is shared. Reentrant opening of a resource that is already being opened is rejected rather than started twice. If opening fails, the resource is marked failed, nothing is cached, and data_retry() allows another attempt. Resources already ready are never validated or opened again. Replacing a loader with data_loader_add(replace = TRUE) affects only resources opened afterwards, and removing one with data_loader_remove() marks unopened resources unavailable-loader and leaves ready objects alone.

Discovery precedence

Discovery uses metadata only. For each resource name, in this order:

1. A manifest entry replaces every other candidate for that resource. A manifest can therefore hand a standard dataset to a specialized loader, or choose between competing adapters. Declaring one resource twice in a manifest is an error, even if the records agree.
2. A resource in the standard package data inventory (.rda and script datasets in data/) belongs to the built-in reader. An adapter claim on one is ignored with a projectData_ignored_claim warning naming the claimant, unless a manifest says otherwise. Adapters that routinely scan extdata/ can make this a message or silent with data_config(ignored_claims = "message") or "silent". The rule is about resource identity rather than file type: an adapter can still claim a directory resource that contains .rda files internally.
3. Otherwise, several adapters claiming one resource is ambiguous and fails with projectData_ambiguous_discovery, naming the claimants. The outcome does not depend on the order in which the loaders were registered.

Two distinct resources that expose the same object name are a separate error (projectData_duplicate_object), and a manifest does not merge them. The check is repeated on the full proposed mapping in data_inspect(). Identical names in separate collections are supported.

Registration-time and first-access checks

At registration, validation runs only for loaders that are already registered. Registration therefore succeeds when an optional adapter is missing. At first access, validation runs again, against whichever loader is selected to open. A replacement loader is never covered by an earlier loader’s validation. Validators should inspect paths and small metadata only.

Roots, layouts, and relative paths

Source paths in a manifest, and the paths your discover() and validate() callbacks work with, are relative to the installed package root, where data/ and the contents of inst/ sit side by side (data/example.rda, extdata/counts.tinymat). A source tree is laid out differently (inst/extdata/...), so authoring accepts three kinds of root:

• a package source tree (a directory with a DESCRIPTION), the recommended entry point. projectData builds a temporary view of the installed shape from links to data/ and each top-level entry of inst/, passes it to your callbacks, and removes it afterwards. Your files are not modified, the package is not installed, and no dataset is loaded, copied, or run. Standard datasets are recognized by file name, so their object names stay unknown.
• an installed package directory, used as is.
• a standalone resource directory, used as is. This is the earlier convention of passing inst/. It works for extdata/ resources but cannot see data/.

The context your callbacks receive describes which one applies:

context$collection is NA while authoring and in a standalone data_manifest_validate(), and context$resource is NA during discovery. A callback that stores the root it received during authoring must not rely on it afterwards, because the source-tree view is temporary.

Symlinks and filesystem limits

Authoring against a source tree needs a file system and R session that can create symbolic links. If a link cannot be created, authoring fails with an error that names the entry. It does not fall back to copying, because that could duplicate a very large collection, and existing manifests are left unchanged. The temporary view is removed as it was made (links as links) without following or deleting their targets. If something cannot be removed a projectData_cleanup_warning names it.

In a verified source tree, a top-level entry of inst/ may itself be a symbolic link, for example inst/extdata -> /shared/data/project. A source such as extdata/example_resource is accepted if the final path exists, and is always written as the package-relative path, never the physical target. This does not apply to data/, installed packages, or standalone directories, which keep strict containment, and it does not make the package portable. Check that R CMD build places regular files, not links, in the tarball.

Optional dependencies and failures

• Heavy dependencies belong in the adapter, loaded inside open with their namespace (requireNamespace() or pkg::), not attached. A missing dependency should produce a clear error from open, and then only that resource fails.
• validate should not need those dependencies. If a check needs them, return TRUE and let open report the problem.
• An unavailable adapter package gives a projectData_loader_unavailable error for the resources that need it; unrelated resources are unaffected.
• data_retry() clears a failed state for another attempt. data_inspect() inspects a resource on request to learn its true object names.

Safety checklist

• Do not read data in discover or validate.
• Resolve locations from paths; never hard-code a library location.
• Declare storage = "disk-backed" when objects read lazily from files.
• Know whether the object references installed files or a working copy, and document which.
• Discover by an explicit format convention, or let manifests list resources.