The NestedScale object

This vignette takes one object apart. The example is a small fuel classification with every awkward feature: a code reused at two frames, a frame that cross-cuts another, an atom with no code at one frame, and attributes stored at three different levels.

lt <- data.frame(
  group   = c("Fossil", "Fossil", "Fossil", "Fossil", "Fossil",
              "Bio", "Bio", "Bio", "Power"),
  phase   = c("solid", "solid", "liquid", "liquid", "gas",
              "solid", "liquid", "gas", NA),
  carrier = c("Coal", "Coal", "Oil", "Oil", "Gas",
              "Biomass", "Biomass", "Biomass", "Electricity"),
  fuel    = c("Anthracite", "Lignite", "Diesel", "Gasoline", "NaturalGas",
              "Wood", "Biodiesel", "Biogas", "Electricity")
)
at <- data.frame(
  frame     = c("fuel", "fuel", "carrier", "group", "fuel", "carrier"),
  unit      = c("Anthracite", "Lignite", "Oil", "Bio", "Anthracite", "Gas"),
  attribute = c("lhv", "lhv", "lhv", "lhv", "ef_co2", "ef_co2"),
  kind      = c(rep("property", 4), "factor", "factor"),
  value     = c(26.7, 11.9, 42.5, 15.0, 98300, 56100),
  unit_of_measure = c(rep("GJ/t", 4), "kg/TJ", "kg/TJ"),
  source    = "example"
)
al <- data.frame(
  frame  = c("fuel", "carrier", "fuel", "carrier", "carrier"),
  unit   = c("Anthracite", "Coal", "Electricity", "Electricity", "Electricity"),
  alias  = c("hard coal", "coal", "elc", "elc", "power"),
  source = "example"
)
x <- scale_from_leaftable(
  lt, frames = c("group", "phase", "carrier", "fuel"), key = "fuel",
  attributes = at, aliases = al, name = "fuels"
)
x
#> NestedScale: fuels
#> Frames (4, coarsest first):
#>   - group (3)
#>     - phase (3)  [1 unit unassigned]
#>       - carrier (5)
#>         - fuel (9)
#> Atoms: 9
#> Attributes: lhv, ef_co2 (sources: example) -- 1 at group, 2 at carrier, 3 at fuel
#> Aliases: 5 (example)

The object

A NestedScale is an S7 object with seven properties:

S7::prop_names(x)
#> [1] "leaftable"  "frames"     "members"    "key"        "meta"      
#> [6] "attributes" "aliases"

property	type	holds
`leaftable`	`data.frame`	one row per atom: one column per frame, the key column, weight columns
`frames`	`character`	the hierarchy, ordered coarsest first
`members`	named `list`	the ordered code vocabulary at each frame
`key`	`character`	the name of the atom column (`"unit"` by default)
`meta`	`list`	`name`, `desc`, `weights`, `default_weight`, and whatever the builder adds
`attributes`	`data.frame`	values keyed by `(frame, unit)`: `attribute`, `kind`, `value`, `lower`, `upper`, `unit_of_measure`, `source`, `note`
`aliases`	`data.frame`	alternative labels keyed by `(frame, unit)`: `alias`, `source`

Everything else – parent/child tables, ancestry, shares, resolved attributes – is derived on demand; nothing is cached on the object. The two tables are empty unless the builder fills them; a scale saved by an earlier version, which had no such properties, loads and validates unchanged.

The leaftable

scale_leaftable(x)
#>    group  phase     carrier        fuel
#> 1 Fossil  solid        Coal  Anthracite
#> 2 Fossil  solid        Coal     Lignite
#> 3 Fossil liquid         Oil      Diesel
#> 4 Fossil liquid         Oil    Gasoline
#> 5 Fossil    gas         Gas  NaturalGas
#> 6    Bio  solid     Biomass        Wood
#> 7    Bio liquid     Biomass   Biodiesel
#> 8    Bio    gas     Biomass      Biogas
#> 9  Power   <NA> Electricity Electricity

Each row is one atom. The key column (fuel here) is unique and complete. Frame columns may hold NA: Electricity has no physical state, so it does not participate at the phase frame, and na_action= decides what recasts do about it. The key may share its name with the finest frame, as here and in the shipped carriers: there the frame column is the key column.

Frames, members, ranks

scale_frames(x)
#> [1] "group"   "phase"   "carrier" "fuel"
scale_atom_level(x)              # the finest frame, or the key
#> [1] "fuel"
scale_units(x, "carrier")        # the members at one frame
#> [1] "Coal"        "Oil"         "Gas"         "Biomass"     "Electricity"
scale_units(x)                   # the atoms
#> [1] "Anthracite"  "Lignite"     "Diesel"      "Gasoline"    "NaturalGas" 
#> [6] "Wood"        "Biodiesel"   "Biogas"      "Electricity"
scale_rank(x, "carrier")
#> [1] 3

Codes are not unique across frames: Electricity is a group and a carrier here, and Nuclear, Heat, Storage, Other are each a group and a carrier in carriers. Every function that takes a code therefore takes its frame as well.

Navigation is derived from the leaftable and routed through the atoms, so a frame that cross-cuts another reports overlap rather than invented containment:

scale_nests(x, "carrier", "fuel")    # a tree
#> [1] TRUE
scale_nests(x, "phase", "carrier")   # Biomass is solid, liquid and gas
#> [1] FALSE
#> attr(,"offenders")
#> [1] "Biomass"
scale_parents(x, "carrier", "Biomass", target = "phase")
#> [1] "solid"  "liquid" "gas"
scale_ancestors(x, "fuel", "Biogas")
#>     frame    unit
#> 1   group     Bio
#> 2   phase     gas
#> 3 carrier Biomass
scale_descendants(x, "group", "Fossil")
#>      frame       unit
#> 1    phase      solid
#> 2    phase     liquid
#> 3    phase        gas
#> 4  carrier       Coal
#> 5  carrier        Oil
#> 6  carrier        Gas
#> 7     fuel Anthracite
#> 8     fuel    Lignite
#> 9     fuel     Diesel
#> 10    fuel   Gasoline
#> 11    fuel NaturalGas
head(scale_ancestry(x), 4)
#>   parent_frame parent child_frame   child
#> 1        group    Bio       phase     gas
#> 2        group    Bio       phase  liquid
#> 3        group    Bio       phase   solid
#> 4        group    Bio     carrier Biomass

The attribute table

Physical attributes live in one long table, and a row may sit on any member, not only on an atom:

scale_attributes(x)
#>     frame       unit attribute     kind   value lower upper unit_of_measure
#> 1    fuel Anthracite       lhv property    26.7    NA    NA            GJ/t
#> 2    fuel    Lignite       lhv property    11.9    NA    NA            GJ/t
#> 3 carrier        Oil       lhv property    42.5    NA    NA            GJ/t
#> 4   group        Bio       lhv property    15.0    NA    NA            GJ/t
#> 5    fuel Anthracite    ef_co2   factor 98300.0    NA    NA           kg/TJ
#> 6 carrier        Gas    ef_co2   factor 56100.0    NA    NA           kg/TJ
#>    source note
#> 1 example <NA>
#> 2 example <NA>
#> 3 example <NA>
#> 4 example <NA>
#> 5 example <NA>
#> 6 example <NA>

Reading with a frame resolves one value per member, walking to the nearest coarser frame when a member has none of its own, and says where the value came from:

scale_attributes(x, "lhv", frame = "fuel")[
  c("unit", "value", "unit_of_measure", "origin_frame", "origin_unit")]
#>          unit value unit_of_measure origin_frame origin_unit
#> 1  Anthracite  26.7            GJ/t         fuel  Anthracite
#> 2     Lignite  11.9            GJ/t         fuel     Lignite
#> 3      Diesel  42.5            GJ/t      carrier         Oil
#> 4    Gasoline  42.5            GJ/t      carrier         Oil
#> 5  NaturalGas    NA            <NA>         <NA>        <NA>
#> 6        Wood  15.0            GJ/t        group         Bio
#> 7   Biodiesel  15.0            GJ/t        group         Bio
#> 8      Biogas  15.0            GJ/t        group         Bio
#> 9 Electricity    NA            <NA>         <NA>        <NA>

Attributes are not weights. A weight says how big an atom is and drives aggregation; an attribute is a per-member conversion factor and is consumed only by convert_scale(). The constructor promotes numeric leaftable columns to weights, so a calorific value must be passed as attributes=, never as a leaftable column. Every attribute name carries a kind: property (an invertible ratio) or factor (one-way).

SCALE_ATTRIBUTE_KINDS
#> [1] "property" "factor"

vignette("attributes") has the reasoning and the full account of inheritance, sources and conversion.

The alias table

scale_aliases(x)
#>     frame        unit     alias  source
#> 1    fuel  Anthracite hard coal example
#> 2 carrier        Coal      coal example
#> 3    fuel Electricity       elc example
#> 4 carrier Electricity       elc example
#> 5 carrier Electricity     power example
scale_match(x, c("hard coal", "elc", "Lignite"))
#>       label frame       unit    method                            candidates
#> 1 hard coal  fuel Anthracite     alias                                  <NA>
#> 2       elc  <NA>       <NA> ambiguous fuel/Electricity; carrier/Electricity
#> 3   Lignite  fuel    Lignite     exact                                  <NA>

One alias may point at several members (elc above); matching reports that as ambiguous unless frame= narrows it. Matching is exact and case-insensitive, with no fuzzy stage.

Weights and meta

scale_weights(x)                        # none declared here
#> character(0)
S7::prop(x, "meta")[c("name", "desc")]
#> $name
#> [1] "fuels"
#> 
#> $desc
#> [1] ""
carriers <- read_scale("carriers")
scale_weights(carriers)
#> character(0)
S7::prop(carriers, "meta")[c("name", "source", "palette_version")]
#> $name
#> [1] "carriers"
#> 
#> $source
#> [1] "energypal carriers.yml"
#> 
#> $palette_version
#> [1] "0.1.1"
head(S7::prop(carriers, "meta")$labels, 4)
#>     frame        unit label_short   label_long
#> 1   group FossilFuels      Fossil Fossil Fuels
#> 2 carrier  FossilCoal        Coal         Coal
#> 3 carrier   FossilOil         Oil          Oil
#> 4 carrier   FossilGas         Gas  Natural Gas

meta$name is load-bearing: join_scale() prefixes the columns it attaches with "<name>.", scale_map_between() names its label columns after the objects, and write_scale() needs a name for the definition. The shipped carriers keeps energypal’s short and long labels in meta$labels; colours are not stored.

What the validator guarantees

scale_from_leaftable() and every operation that rebuilds a scale enforce:

the key column is present, unique and complete; every frame is a leaftable column and members are exactly its non-NA values, in a stable order;
weights are numeric and finite;
every attribute and alias row names a (frame, unit) that is a member; value is finite; lower <= value <= upper where given; kind is one of SCALE_ATTRIBUTE_KINDS and one attribute name has one kind; unit_of_measure and source are non-empty; no duplicate (frame, unit, attribute, source) rows and no duplicate alias rows.

The order of frames is a convention, coarsest first; the direction-dependent rules (share, residuals, reconciliation) follow it.

Subsetting keeps the tables consistent: filter_scale() and [ keep the attribute and alias rows of every surviving member at every frame, and prune_scale() keeps the rows at the frames that survive the collapse.

fossil <- x["group", "Fossil"]
scale_attributes(fossil)[c("frame", "unit", "attribute", "value")]
#>     frame       unit attribute   value
#> 1    fuel Anthracite       lhv    26.7
#> 2    fuel    Lignite       lhv    11.9
#> 3 carrier        Oil       lhv    42.5
#> 4    fuel Anthracite    ef_co2 98300.0
#> 5 carrier        Gas    ef_co2 56100.0

prune_scale() needs the frames above the new atom level to nest, so the cross-cutting phase has to go first:

tree <- scale_from_leaftable(
  lt[c("group", "carrier", "fuel")], frames = c("group", "carrier", "fuel"),
  key = "fuel", attributes = at, aliases = al, name = "tree"
)
scale_aliases(prune_scale(tree, "carrier"))
#>     frame        unit alias  source
#> 1 carrier        Coal  coal example
#> 2 carrier Electricity   elc example
#> 3 carrier Electricity power example

Definition files

A scale is stored as a definition, not as an R object: a JSON header with the three tables inline, or, for large scales, a <name>.scale/ bundle directory holding the header and the tables as Arrow IPC or CSV files. The Python binding reads and writes the same files.

path <- file.path(tempdir(), "fuels.json")
write_scale(x, path)
cat(head(readLines(path), 12), sep = "\n")
#> {
#>   "schema_version": 1,
#>   "nestedscales": "0.2.0",
#>   "class": "nestedscales::NestedScale",
#>   "name": "fuels",
#>   "key": "fuel",
#>   "frames": ["group", "phase", "carrier", "fuel"],
#>   "members": {
#>     "group": ["Fossil", "Bio", "Power"],
#>     "phase": ["solid", "liquid", "gas"],
#>     "carrier": ["Coal", "Oil", "Gas", "Biomass", "Electricity"],
#>     "fuel": ["Anthracite", "Lignite", "Diesel", "Gasoline", "NaturalGas", "Wood", "Biodiesel", "Biogas", "Electricity"]
back <- read_scale(path)
identical(scale_frames(back), scale_frames(x))
#> [1] TRUE
identical(scale_attributes(back), scale_attributes(x))
#> [1] TRUE

write_scale(format = "yaml") writes the same header as YAML, and read_scale() accepts either. Installed packages register directories of definitions; scale_catalog() lists them and read_scale(id) opens one by its id:

scale_catalog()[c("id", "name", "kind", "frames", "n_atoms", "package")]
#>         id     name        kind             frames n_atoms      package
#> 1 carriers carriers NestedScale group/carrier/fuel      71 nestedscales

The registry

One package-level registry changes behaviour explicitly. Rules map value-column names to their aggregation rule; there is no fallback, and an unregistered column without rule= is an error:

SCALE_RULES
#> [1] "sum"           "weighted_mean" "mean"          "copy"         
#> [5] "sd"            "share"         "logshare"
register_scale_rule("tj", "sum")
register_scale_rule("price", "weighted_mean", weight = "tj")
get_scale_rule("price")
#> $rule
#> [1] "weighted_mean"
#> 
#> $weight
#> [1] "tj"
#> 
#> $scope
#> NULL
list_scale_rules()
#>   param          rule weight scope
#> 1 price weighted_mean     tj  <NA>
#> 2    tj           sum   <NA>  <NA>
clear_scale_rules()

A rule may be scoped to one dimension with scope=, so a model’s tj column aggregates the same way over fuels, regions and months without being registered three times – or differently, when it should.