Many expensive steps produce files rather than R values: alignments, index files, model checkpoints, rendered reports. This vignette shows how a shared computation writes files, how they come back on a hit, and how to identify input files by their contents.
library(targets)
library(tarshare)
root <- tempfile("tarshare-files-")
dir.create(root)
cache_path <- file.path(root, "cache")
cache <- shared_cache(cache_path)Writing files
Use format = "file" and write every output below
shared_path(). Inside a file computation,
shared_path("name") returns a path in a private staging
directory and creates any parent directories. Return the paths you want
back, or shared_path() for the whole directory:
values <- c(3, 5, 8, 13)
paths <- shared_run({
cat("computing\n")
summary <- c(n = length(values), mean = mean(values))
saveRDS(summary, shared_path("summary.rds"))
writeLines(format(summary), shared_path("summary.txt"))
writeLines("summary.rds summary.txt", shared_path("MANIFEST"))
c(rds = shared_path("summary.rds"), text = shared_path("summary.txt"))
}, cache, format = "file")
#> tarshare compute 7916bd567f28
#> computing
basename(paths)
#> [1] "summary.rds" "summary.txt"
readRDS(paths[["rds"]])
#> n mean
#> 4.00 7.25The returned paths point into the cache, not into staging. The whole staging directory was stored, so the manifest travelled with the two files you asked for even though it was not returned:
list.files(dirname(paths[["rds"]]))
#> [1] "MANIFEST" "summary.rds" "summary.txt"A hit returns the same paths without running the command:
paths_again <- shared_run({
cat("computing\n")
summary <- c(n = length(values), mean = mean(values))
saveRDS(summary, shared_path("summary.rds"))
writeLines(format(summary), shared_path("summary.txt"))
writeLines("summary.rds summary.txt", shared_path("MANIFEST"))
c(rds = shared_path("summary.rds"), text = shared_path("summary.txt"))
}, cache, format = "file")
#> tarshare hit 7916bd567f28
identical(paths, paths_again)
#> [1] TRUEThe files belong to the cache
Cached files are immutable by contract: downstream steps read them
and write their own outputs elsewhere. The cache does not change file
permissions to enforce this, but with the default
verify = TRUE every lookup re-hashes the bundle. A bundle
that no longer matches its record is treated as missing, the command
runs again, and the damaged entry is replaced:
writeLines("edited in place", paths[["text"]])
paths <- shared_run({
cat("computing\n")
summary <- c(n = length(values), mean = mean(values))
saveRDS(summary, shared_path("summary.rds"))
writeLines(format(summary), shared_path("summary.txt"))
writeLines("summary.rds summary.txt", shared_path("MANIFEST"))
c(rds = shared_path("summary.rds"), text = shared_path("summary.txt"))
}, cache, format = "file")
#> tarshare compute 7916bd567f28
#> computing
readLines(paths[["text"]])
#> [1] "4.00" "7.25"Turn verification off only when something else guarantees immutability, such as a read-only export of the cache. Record integrity and bundle completeness are checked either way.
Outputs must be created inside staging. A path from anywhere else is rejected before anything is published:
tryCatch(
shared_run({
elsewhere <- tempfile()
writeLines("x", elsewhere)
elsewhere
}, cache, format = "file"),
error = conditionMessage
)
#> tarshare compute e570a75d7f22
#> [1] "File outputs must be created inside shared_path()."Symbolic links inside the bundle are rejected too, and the cache never rewrites file contents, so avoid writing the staging path into a file.
Values that are not self-contained
The default format = "rds" is for values that can be
serialised and read back on their own. Live resources are refused:
tryCatch(shared_run(new.env(), cache), error = conditionMessage)
#> tarshare compute 7ac2b8463002
#> [1] "Cached outputs must be self-contained values, not live resources or functions."An R object that merely contains a path does not preserve the file
behind the path. Write such files through format = "file"
instead.
File targets in a pipeline
In a pipeline, a format = "file" shared target behaves
like a targets file target: its value is the vector of cache paths, and
downstream targets use them like any other file. The pipeline below
writes a table and counts its rows:
script <- file.path(root, "_targets.R")
tar_helper(script, {
library(targets)
library(tarshare)
cache <- shared_cache(!!cache_path)
list(
tar_target(values, c(3, 5, 8, 13)),
tar_shared(values_table, {
path <- shared_path("values.csv")
utils::write.csv(data.frame(value = values), path, row.names = FALSE)
path
}, cache, format = "file"),
tar_target(n_rows, nrow(utils::read.csv(values_table)))
)
})
store_1 <- file.path(root, "store-1")
tar_make(script = script, store = store_1)
#> + values dispatched
#> ✔ values completed [1ms, 60 B]
#> + values_table dispatched
#> tarshare compute 31320ee04d4c
#> ✔ values_table completed [63ms, 17 B]
#> + n_rows dispatched
#> ✔ n_rows completed [0ms, 49 B]
#> ✔ ended pipeline [214ms, 3 completed, 0 skipped]
tar_read(n_rows, store = store_1)
#> [1] 4A second store gets the table from the cache. The cheap downstream target still runs, because it is an ordinary target in a fresh store:
store_2 <- file.path(root, "store-2")
tar_make(script = script, store = store_2)
#> + values dispatched
#> ✔ values completed [0ms, 60 B]
#> + values_table dispatched
#> tarshare hit 31320ee04d4c
#> ✔ values_table completed [62ms, 17 B]
#> + n_rows dispatched
#> ✔ n_rows completed [1ms, 49 B]
#> ✔ ended pipeline [213ms, 3 completed, 0 skipped]
identical(tar_read(values_table, store = store_1), tar_read(values_table, store = store_2))
#> [1] TRUEBecause the shared target has format = "file", targets
also watches the cache files, and the usual skipping applies while they
are unchanged.
Identifying input files
A plain character path is an ordinary value: the key records the
string, not the file. shared_file() replaces it with a
descriptor that carries a content hash per path and still works as a
path. See vignette("identity") for why this matters.
sample_a <- file.path(root, "sample-a.txt")
sample_b <- file.path(root, "sample-b.txt")
writeLines(c("3", "5"), sample_a)
writeLines(c("8", "13"), sample_b)
inputs <- shared_file(c(a = sample_a, b = sample_b))
inputs
#> a
#> "/tmp/RtmpihRxoZ/tarshare-files-1bfe4589fb0d/sample-a.txt"
#> b
#> "/tmp/RtmpihRxoZ/tarshare-files-1bfe4589fb0d/sample-b.txt"
#> attr(,"class")
#> [1] "tarshare_file" "character"
#> attr(,"tarshare_id")
#> [1] "56c47cb32092661c2f3438298862e0759fa78694dbe78060f6369502f4386a09"
#> [2] "cb3e366a100041eb445e5ce5455bc2b035b1f4e2e2d7f0aa6bd7ce4a1369a39d"Descriptors keep names, order and ids together under subsetting and concatenation, so a pipeline can pass them around as it would any paths:
inputs["b"]
#> b
#> "/tmp/RtmpihRxoZ/tarshare-files-1bfe4589fb0d/sample-b.txt"
#> attr(,"class")
#> [1] "tarshare_file" "character"
#> attr(,"tarshare_id")
#> [1] "cb3e366a100041eb445e5ce5455bc2b035b1f4e2e2d7f0aa6bd7ce4a1369a39d"
c(inputs["b"], inputs["a"])
#> b
#> "/tmp/RtmpihRxoZ/tarshare-files-1bfe4589fb0d/sample-b.txt"
#> a
#> "/tmp/RtmpihRxoZ/tarshare-files-1bfe4589fb0d/sample-a.txt"
#> attr(,"class")
#> [1] "tarshare_file" "character"
#> attr(,"tarshare_id")
#> [1] "cb3e366a100041eb445e5ce5455bc2b035b1f4e2e2d7f0aa6bd7ce4a1369a39d"
#> [2] "56c47cb32092661c2f3438298862e0759fa78694dbe78060f6369502f4386a09"Replacement is refused: rebuild the descriptor with
shared_file() when a path changes.
as.character() drops the ids and gives back ordinary
paths.
Hashing a large file on every graph construction is wasteful when an
immutable identity already exists, such as a hash from a sample sheet or
an object version. Pass it as id and the file is not
read:
registered <- shared_file(sample_a, id = "sha256:established-upstream")
attr(registered, "tarshare_id")
#> [1] "sha256:established-upstream"Supplying an id is an assertion that the id describes the bytes used. The cache does not check it. Directories cannot be hashed automatically and always need an explicit id.
Use content identity only when the result does not depend on the
file’s name or location. If a name carries meaning, such as a sample
label, declare it in context as well.