Skip to contents

What makes two computations the same

Source: vignettes/identity.Rmd
identity.Rmd

A shared cache is only as good as its notion of “the same computation”. This vignette shows what tarshare puts into a computation key, what it cannot see, and how to declare the rest. Every example prints computing when the command really runs, so a missing line means a hit.

library(targets)
library(tarshare)

root <- tempfile("tarshare-identity-")
dir.create(root)
cache_path <- file.path(root, "cache")
cache <- shared_cache(cache_path)

What is in the key

The key is a SHA-256 of:

  • the command’s code as an abstract syntax tree, with source references stripped;
  • the values of the variables and the code of the functions that the command refers to, found by static analysis with codetools and followed recursively through your own functions;
  • the versions of the packages involved: the namespaces of functions the command uses, pkg::fun references, their hard dependencies, and anything named in packages;
  • the format, the declared context and the seed;
  • the R version and platform, and the tarshare version.

Not in the key: the target name, the store, the working directory, the time, and the retention owner.

Formatting does not matter, symbols do 

x <- c(3, 5, 8, 13)
summarise <- function(v) { cat("computing\n"); c(mean(v), stats::sd(v)) }
shared_run(summarise(x), cache)
#> tarshare compute bc09afc16a95
#> computing
#> [1] 7.250000 4.349329

Rewrite the helper with different layout and a comment. The abstract syntax tree is unchanged, so the second call is a hit:

summarise <- function(v) {
  # The same code, laid out differently.
  cat("computing\n")
  c(mean(v), stats::sd(v))
}
shared_run(summarise(x), cache)
#> tarshare hit bc09afc16a95
#> [1] 7.250000 4.349329

Refer to the same data through a different name and the command’s code is different, so the key is different:

y <- x
shared_run(summarise(y), cache)
#> tarshare compute 6b13b5032c4b
#> computing
#> [1] 7.250000 4.349329

tarshare compares code, not meaning. It does not try to prove that two different expressions compute the same thing.

Referenced values and helpers are followed

Change one input value and the key changes:

x[1] <- 4
shared_run(summarise(x), cache)
#> tarshare compute 317f5df17c82
#> computing
#> [1] 7.500000 4.041452

Helpers called by the helper are followed too. summarise() has not changed here, but a function it calls has:

centre <- function(v) mean(v)
summarise <- function(v) { cat("computing\n"); c(centre(v), stats::sd(v)) }
shared_run(summarise(x), cache)
#> tarshare compute 97298bfdf1ca
#> computing
#> [1] 7.500000 4.041452

centre <- function(v) stats::median(v)
shared_run(summarise(x), cache)
#> tarshare compute f56ee1449f32
#> computing
#> [1] 6.500000 4.041452

Packages

A function from an installed package is identified by its package and version rather than by its code, and the same goes for pkg::fun references. If your command chooses a package at run time, for example through a method registered by another package, name it in packages so its version joins the key. shared_context() does the same for method providers.

Random numbers

Random draws inside a shared command come from a stream seeded from the key, so a hit returns exactly the draws that were stored, and two sessions that compute the same key draw the same numbers:

shared_run(stats::rnorm(3), cache)
#> tarshare compute dfcc083d350e
#> [1] -0.3424294 -0.4487965  0.1880137
shared_run(stats::rnorm(3), cache)
#> tarshare hit dfcc083d350e
#> [1] -0.3424294 -0.4487965  0.1880137

seed is part of the key. Use it to ask for an independent replicate of an otherwise identical computation:

shared_run(stats::rnorm(3), cache, seed = 1)
#> tarshare compute 3c67c29f24cc
#> [1] 0.8164275 1.1001383 0.9863871

The caller’s random number generator is left as it was:

set.seed(42)
before <- stats::runif(1)
set.seed(42)
invisible(shared_run(stats::rnorm(5), cache))
#> tarshare compute 58d200935feb
after <- stats::runif(1)
identical(before, after)
#> [1] TRUE

What the key cannot see

Static analysis records the code that reads an option, but not the option’s value. The second call below is a hit, and the result is stale:

options(analysis.scale = 2)
scale_values <- function(v) { cat("computing\n"); v * getOption("analysis.scale") }
shared_run(scale_values(x), cache)
#> tarshare compute 520446500bc7
#> computing
#> [1]  8 10 16 26

options(analysis.scale = 10)
shared_run(scale_values(x), cache)
#> tarshare hit 520446500bc7
#> [1]  8 10 16 26

The same applies to environment variables, the locale, executables found on the path, files opened by name, network and database state, and S3 or S4 methods chosen by dispatch.

Pass values in, or declare them

The simplest fix is to resolve the state outside the command and pass it in as a value. A referenced variable is part of the key:

scale_by <- function(v, scale) { cat("computing\n"); v * scale }
scale <- getOption("analysis.scale")
shared_run(scale_by(x, scale), cache)
#> tarshare compute 8924014b317d
#> computing
#> [1]  40  50  80 130

When the state has to stay inside the command, declare it in context. shared_context() snapshots the option’s value, and the snapshot is part of the key:

shared_run(scale_values(x), cache, context = shared_context(options = "analysis.scale"))
#> tarshare compute d9bed6633290
#> computing
#> [1]  40  50  80 130

shared_context() returns a plain list that you can inspect. Besides options it takes named immutable values, file contents, non-secret environment variables, provider packages and the locale:

model_file <- file.path(root, "model.rds")
saveRDS(list(coefficients = c(1.5, -0.2)), model_file)
Sys.setenv(ANALYSIS_VARIANT = "reference")

execution <- shared_context(
  image = "sha256:3f1a...",
  files = list(model = model_file),
  options = "analysis.scale",
  env = "ANALYSIS_VARIANT",
  locale = TRUE
)
str(execution, max.level = 2)
#> List of 6
#>  $ values  :List of 1
#>   ..$ image: chr "sha256:3f1a..."
#>  $ files   :List of 1
#>   ..$ model: 'tarshare_file' chr "/tmp/RtmpOAoBKK/tarshare-identity-1c5b2e080669/model.rds"
#>   .. ..- attr(*, "tarshare_id")= chr "d4cc48a6cd9dd88abbe96935aa2085d8bbffc098877c6ec5c2b63980d9a54d68"
#>  $ options :List of 1
#>   ..$ analysis.scale: num 10
#>  $ env     :List of 1
#>   ..$ ANALYSIS_VARIANT: chr "reference"
#>  $ packages: list()
#>  $ locale  : chr "LC_CTYPE=C.UTF-8;LC_NUMERIC=C;LC_TIME=C.UTF-8;LC_COLLATE=C.UTF-8;LC_MONETARY=C.UTF-8;LC_MESSAGES=C.UTF-8;LC_PAP"| __truncated__

A declaration is an assertion by its owner. A version label such as "model 2.1" does not identify a replaced file, so declare the contents or an immutable id instead. Never put credentials in context: targets writes the generated command into its metadata.

Files

A plain path is just a string. The file behind it can change without changing the key:

path <- file.path(root, "measurements.txt")
writeLines(c("3", "5", "8", "13"), path)
read_mean <- function(p) { cat("computing\n"); mean(scan(p, quiet = TRUE)) }
shared_run(read_mean(path), cache)
#> tarshare compute a249e13d28f4
#> computing
#> [1] 7.25

writeLines(c("30", "50"), path)
shared_run(read_mean(path), cache)
#> tarshare hit a249e13d28f4
#> [1] 7.25

shared_file() wraps a path in a descriptor whose identity is the SHA-256 of the contents. It still works as a path:

input <- shared_file(path)
shared_run(read_mean(input), cache)
#> tarshare compute 766619bd9b9f
#> computing
#> [1] 40

Replace the contents and the key changes:

writeLines(c("1", "2"), path)
input <- shared_file(path)
shared_run(read_mean(input), cache)
#> tarshare compute 431e53723fd7
#> computing
#> [1] 1.5

Move the file and the key does not change, because the contents did not:

moved <- file.path(root, "moved.txt")
invisible(file.rename(path, moved))
input <- shared_file(moved)
shared_run(read_mean(input), cache)
#> tarshare hit 431e53723fd7
#> [1] 1.5

If a trusted hash or an immutable version id already exists, pass it as id and the file is not read. Directories always need an explicit id. See vignette("files") for descriptors with several paths.

Context in a pipeline

In tar_shared(), context is evaluated once while _targets.R runs to build the graph. tarshare embeds a fingerprint of it in the generated command, so a change in declared context invalidates the local target in the ordinary targets way, and the shared key changes with it. Declare ambient state there:

tar_shared(
  scaled, scale_values(values), cache,
  context = shared_context(options = "analysis.scale")
)

Values that come from other targets are different: they do not exist yet when the graph is built. Put them in branch_context, which stays an expression that targets can see. The pipeline below draws two numbers per replicate. The replicate id is in the branch context, so each branch has its own key and its own draws:

script <- file.path(root, "_targets.R")
tar_helper(script, {
  library(targets)
  library(tarshare)
  cache <- shared_cache(!!cache_path)
  list(
    tar_target(replicate, 1:3),
    tar_shared(
      draws, stats::rnorm(2), cache,
      pattern = map(replicate),
      branch_context = list(replicate = replicate)
    )
  )
})
tar_make(script = script, store = file.path(root, "store-1"))
#> + replicate dispatched
#> ✔ replicate completed [0ms, 99 B]
#> + draws declared [3 branches]
#> tarshare compute f2c3dfb4c1e7
#> tarshare compute ce36c015699a
#> tarshare compute 91b98182afbb
#> ✔ draws completed [154ms, 191 B]
#> ✔ ended pipeline [305ms, 4 completed, 0 skipped]
tar_read(draws, store = file.path(root, "store-1"))
#> draws_b9f9cd8c0e23532e_1 draws_b9f9cd8c0e23532e_2 draws_3070b9ca6a0befd9_1 
#>                2.1954871               -0.7177479                1.7308572 
#> draws_3070b9ca6a0befd9_2 draws_4a4ad12a5814925f_1 draws_4a4ad12a5814925f_2 
#>                0.2540403                1.5230591                1.2043800

A second store runs the same pipeline and hits all three branches:

tar_make(script = script, store = file.path(root, "store-2"))
#> + replicate dispatched
#> ✔ replicate completed [0ms, 99 B]
#> + draws declared [3 branches]
#> tarshare hit f2c3dfb4c1e7
#> tarshare hit ce36c015699a
#> tarshare hit 91b98182afbb
#> ✔ draws completed [151ms, 191 B]
#> ✔ ended pipeline [303ms, 4 completed, 0 skipped]

Without branch_context, the three branches would share one key and one set of draws, because nothing else distinguishes them. The same is true of replicates that differ only in a loop index: give them distinct branch context or distinct seeds.

tar_shared_raw() is the programmatic form. It takes a resolved value for context and a quoted expression for branch_context.