# Remote and Cloud Stores

Zarr was designed for data that sits in object storage or on a local disk. This vignette covers how pizzarr reaches them — over HTTPS, and over the S3 API, including S3-compatible services that are not Amazon.

The worked example throughout is gridMET, a daily gridded meteorology dataset for the continental United States, republished by the USGS on an Open Storage Network pod. Its catalog entry advertises two addresses for the same data:

    s3://mdmf/gdp/gridMET.zarr/    endpoint https://usgs.osn.mghpcc.org/

That pairing — a bucket URL plus a non-Amazon endpoint — is a case that requires extra documentation, so it is illustrated in this vignette.

## Coming from xarray or fsspec

In Python you would hand that catalog entry to `xarray` and let `fsspec` sort out the connection:

    import xarray as xr

    ds = xr.open_dataset(
        "s3://mdmf/gdp/gridMET.zarr/",
        engine="zarr",
        backend_kwargs={"storage_options": {
            "anon": True,
            "client_kwargs": {"endpoint_url": "https://usgs.osn.mghpcc.org/"},
        }},
    )

However, pizzarr has no `storage_options` argument. Connection settings come from environment variables that the Rust `object_store` client reads when it first opens a store. The reason is mechanical rather than principled: store handles are cached by URL on the Rust side and shared across calls, so there is no per-call place to pass options. The translation is:

fsspec / xarray | pizzarr
--- | ---
`"anon": True` | the default — unsigned unless credentials are set
`"client_kwargs": {"endpoint_url": "https://host"}` | `AWS_ENDPOINT=https://host`
`"key"`, `"secret"` | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
`"token"` | `AWS_SESSION_TOKEN`
`"client_kwargs": {"region_name": "us-west-2"}` | `AWS_REGION=us-west-2`
`"use_ssl": False` | `AWS_ALLOW_HTTP=true`
`"config_kwargs": {"s3": {"addressing_style": "path"}}` | `AWS_VIRTUAL_HOSTED_STYLE_REQUEST=false`
`gcsfs` `"token"` | `GOOGLE_APPLICATION_CREDENTIALS`
`gcsfs` `"token": "anon"` | `GOOGLE_SKIP_SIGNATURE=true`
`gcsfs` `"endpoint_url"` | `GOOGLE_BASE_URL`

Set them with `Sys.setenv()` for the session, or in `.Renviron` to persist them. Because the handle is cached, changing a variable after a store is open does nothing until you call `zarrs_close_store()` on that URL.

## Two routes to the same data

An S3 bucket read anonymously is, underneath, an HTTPS server that returns objects at predictable paths. pizzarr can take either view of it, and which one you want depends less on the data than on which build you installed (CRAN or r-universe).

The HTTPS route addresses objects directly. That is, an `s3://bucket/key` address served by endpoint `https://host` is reachable at `https://host/bucket/key`, so gridMET becomes:

    https://usgs.osn.mghpcc.org/mdmf/gdp/gridMET.zarr

This https route works on both distribution tiers — the pure-R CRAN build and the r-universe build with the zarrs backend. It needs no credentials and no configuration, and it is the only route the CRAN build has, so it is the one to reach for unless something stops you.

The S3 route uses the `s3://` URL and the real S3 API by way of `object_store`. It requires the r-universe build with the `s3` feature compiled in, and it is what you need for buckets that are not anonymously readable over plain HTTPS, or that require signed requests. Check what you have:

    pizzarr_compiled_features()
    #> zarrs backend not available (pure R install).
    #> See ?pizzarr_upgrade for the r-universe install.
    #> character(0)

## Reading over HTTPS

`HttpStore` takes the URL, and `zarr_open()` gives back a group:

    url <- "https://usgs.osn.mghpcc.org/mdmf/gdp/gridMET.zarr"

    store <- HttpStore$new(url)
    g <- zarr_open(store)
    g
    #> <ZarrGroup> /
    #>   Store type  : HttpStore
    #>   Zarr format : 2
    #>   Read-only   : TRUE
    #>   No. members : 12

HTTP servers generally cannot be listed the way a directory can, so a remote store is only browsable if it publishes consolidated metadata — a `.zmetadata` key holding every array’s metadata in one document. gridMET does, and `listdir()` reports its contents:

    store$listdir()
    #>  [1] "crs"
    #>  [2] "lat"
    #>  [3] "lon"
    #>  [4] "max_air_temperature"
    #>  [5] "max_relative_humidity"
    #>  [6] "min_air_temperature"
    #>  [7] "min_relative_humidity"
    #>  [8] "precipitation_amount"
    #>  [9] "specific_humidity"
    #> [10] "surface_downwelling_shortwave_flux_in_air"
    #> [11] "time"
    #> [12] "wind_speed"

Without consolidated metadata you have to know the array names in advance. This is why so many published Zarr stores are consolidated, and why the flag appears in the gridMET catalog entry alongside the connection settings.

gridMET’s arrays are Blosc-compressed, which the pure-R build handles only if the `blosc` package is installed. The chunks below need it:

    lat <- g$get_item("lat")
    lat
    #> <ZarrArray> /lat
    #>   Shape       : (585)
    #>   Chunks      : (585)
    #>   Data type   : <f8
    #>   Fill value  : NaN
    #>   Order       : C
    #>   Read-only   : TRUE
    #>   Compressor  : BloscCodec
    #>   Store type  : HttpStore
    #>   Zarr format : 2

    lat$get_item(list(slice(1, 5)))$data
    #> [1] 49.40000 49.35833 49.31667 49.27500 49.23333

Reaching the data variables is the same call. Their shape is worth a look before reading anything:

    tmmx <- g$get_item("max_air_temperature")
    tmmx
    #> <ZarrArray> /max_air_temperature
    #>   Shape       : (17369, 585, 1386)
    #>   Chunks      : (2190, 150, 150)
    #>   Data type   : <i2
    #>   Fill value  :
    #>   Order       : C
    #>   Read-only   : TRUE
    #>   Compressor  : BloscCodec
    #>   Store type  : HttpStore
    #>   Zarr format : 2

That is 17369 days by 585 rows by 1386 columns, in chunks of 2190 by 150 by 150. A chunk is the smallest unit the store will return, so asking for a single value still transfers the roughly 98 MB block that contains it — compressed in flight, but decompressed in full. Slicing a remote array cheaply means slicing along chunk boundaries and keeping the request small in the dimensions that are chunked finely.

## Reading over the S3 API

The `s3://` route needs the endpoint, and nothing else for a public bucket:

    Sys.setenv(AWS_ENDPOINT = "https://usgs.osn.mghpcc.org")

    s3_url <- "s3://mdmf/gdp/gridMET.zarr"

    zarrs_open_array_metadata(s3_url, "lat")

Reads go through `zarrs_get_subset()`, which takes zero-based, stop-exclusive ranges — one per dimension — and returns the data along with its shape:

    zarrs_get_subset(s3_url, "lat", list(c(0L, 5L)), NULL)

Blosc decompression happens in Rust here, so this path does not need the `blosc` R package. When you are done with a store, drop its cached handle:

    zarrs_close_store(s3_url)

`S3Store` and `GcsStore` exist to mark a URL as cloud-backed so that dispatch picks the right backend. They are not full stores — they carry a URL and nothing else, so calling `get_item()` or `listdir()` on one raises an error naming `zarrs_get_subset()`, `HttpStore`, and this vignette. The functions above are the working entry point for `s3://` data.

## Alternate endpoints

The endpoint variable is the whole trick for MinIO, Ceph, Wasabi, Cloudflare R2, and Open Storage Network pods. A self-hosted MinIO over plain HTTP with path-style addressing needs three settings:

    Sys.setenv(
      AWS_ENDPOINT = "http://localhost:9000",
      AWS_ALLOW_HTTP = "true",
      AWS_VIRTUAL_HOSTED_STYLE_REQUEST = "false",
      AWS_ACCESS_KEY_ID = "minioadmin",
      AWS_SECRET_ACCESS_KEY = "minioadmin"
    )

    zarrs_get_subset("s3://my-bucket/data.zarr", "temperature",
                     list(c(0L, 10L)), NULL)

Most managed services need only `AWS_ENDPOINT`; region is often unnecessary outside AWS itself. Requests are signed as soon as any credential variable is present and unsigned when none are, which is what makes public buckets work with no setup. `AWS_SKIP_SIGNATURE` overrides that decision in either direction — set it to `"true"` to force anonymous access when stale credentials are lying around in your environment, or `"false"` to insist on signing.

## Google Cloud Storage

GCS works the same way, with `GOOGLE_` variables in place of `AWS_` ones, and it needs the `gcs` feature compiled in. The one difference that matters: GCS does not infer anonymous access from the absence of credentials the way S3 does. A `gs://` read against a world-readable bucket still tries to authenticate — via `GOOGLE_APPLICATION_CREDENTIALS`, `GOOGLE_SERVICE_ACCOUNT`, or the GCE metadata server — and fails if none are available. Ask for anonymous access explicitly:

    Sys.setenv(GOOGLE_SKIP_SIGNATURE = "true")

    gs_url <- "gs://pangeo-data/ECCO_basins.zarr"

    zarrs_open_array_metadata(gs_url, "basin_mask")$shape

`GOOGLE_BASE_URL` overrides the endpoint, which is what you want for a local emulator such as fake-gcs-server.

The same data is also reachable over plain HTTPS at `https://storage.googleapis.com/bucket/path`, which needs no configuration and works on the CRAN tier as well:

    z <- zarr_open(HttpStore$new(
      "https://storage.googleapis.com/pangeo-data/ECCO_basins.zarr"
    ))

## What is not supported

Cloud stores are read-only. Writes to `s3://` and `gs://` fall back to the R-native path, which has no cloud implementation, so they fail — build locally and upload with another tool. Azure (`az://`) is not implemented at all. Requester-pays buckets have no way to signal the requester, and there is no per-store credential object: configuration is process-global, so two buckets needing different credentials in one session have to be read in sequence with `zarrs_close_store()` and an environment change between them.