Byte store adapter

A small key-to-bytes adapter over an OpendalFs prefix. It is intended as a substrate for Zarr-like chunk layouts and other index-driven readers that want store-relative keys without leaving Ropendal's byte-first filesystem API.

Usage

byte_store(fs, prefix = "")

store_cache(
  store,
  cache_dir = tools::R_user_dir("Ropendal", "cache"),
  validate = c("last_modified_size", "none")
)

store_block_cache(
  store,
  cache_dir = tools::R_user_dir("Ropendal", "cache"),
  block_size = 8 * 1024^2,
  validate = c("last_modified_size", "none")
)

store_cache_clear(store)

store_read(
  store,
  key,
  mode = c("bytes", "raw"),
  offset = NULL,
  size = NULL,
  end = NULL,
  result = c("auto", "flat", "nested"),
  batch_concurrency = NULL,
  read_concurrency = NULL,
  chunk_size = NULL,
  coalesce_gap = NULL
)

store_read_aio(
  store,
  key,
  mode = c("bytes", "raw"),
  offset = NULL,
  size = NULL,
  end = NULL,
  result = c("auto", "flat", "nested"),
  batch_concurrency = NULL,
  read_concurrency = NULL,
  chunk_size = NULL,
  coalesce_gap = NULL
)

store_write(
  store,
  key,
  data,
  batch_concurrency = NULL,
  write_concurrency = NULL,
  chunk_size = NULL
)

store_write_aio(
  store,
  key,
  data,
  batch_concurrency = NULL,
  write_concurrency = NULL,
  chunk_size = NULL
)

store_replace(
  store,
  key,
  data,
  batch_concurrency = NULL,
  write_concurrency = NULL,
  chunk_size = NULL
)

store_replace_aio(
  store,
  key,
  data,
  batch_concurrency = NULL,
  write_concurrency = NULL,
  chunk_size = NULL
)

store_exists(store, key, batch_concurrency = NULL)

store_exists_aio(store, key, batch_concurrency = NULL)

store_list(
  store,
  path = "",
  recursive = FALSE,
  limit = NULL,
  start_after = NULL
)

store_list_aio(
  store,
  path = "",
  recursive = FALSE,
  limit = NULL,
  start_after = NULL
)

store_delete(store, key, recursive = FALSE, batch_concurrency = NULL)

store_delete_aio(store, key, recursive = FALSE, batch_concurrency = NULL)

Arguments

fs

Ropendal filesystem handle.

prefix

Root-relative filesystem prefix used as the store root.

store

Byte store object returned by byte_store(), store_cache(), or store_block_cache().

cache_dir

Local directory used by store_cache() or store_block_cache().

validate

Cache validation strategy. "last_modified_size" compares parent object size and modification time before using a cached value. "none" skips modification-time checks; block caches still stat the parent to bound ranges and refresh when cached block metadata has a different object size.

block_size

Fixed block size in bytes for store_block_cache().

key

Store-relative object key or keys.

mode

Read materialization: OpendalBytes handles or raw vectors.

offset, size, end

Optional byte-range controls forwarded to the read operation.

result

Requested read result shape.

batch_concurrency

Optional maximum number of keys to process concurrently.

read_concurrency, write_concurrency

Optional per-object OpenDAL transfer concurrency.

chunk_size

Optional transfer chunk size in bytes.

coalesce_gap

Optional byte gap for coalescing nearby read ranges.

data

Raw vector, OpendalBytes, or list of raw vectors/byte handles.

path

Store-relative directory path for store_list().

recursive

Whether to list or delete recursively.

limit

Optional maximum listing entries.

start_after

Optional store-relative listing continuation marker.

Value

byte_store(), store_cache(), and store_block_cache() return byte-store objects. Store operations return the corresponding filesystem values with paths rewritten relative to the store root for listings.

Details

store_read() returns OpendalBytes by default so callers can stay below ordinary R raw-vector materialization. mode = "raw" or as.raw() exposes the raw-vector API; for OpendalBytes, as.raw() returns a raw ALTREP facade that materializes only when R requires raw-vector pointer access. The store layer is intentionally byte-only: serializers, deserializers, codecs, and format parsers belong above it.

store_cache() wraps a byte store in an explicit local full-object cache. It is useful for chunked layouts where each key is already a small block. Partial byte-range reads and non-default read shaping bypass this first cache layer. store_block_cache() wraps a byte store in an explicit fixed-size block cache for scalar complete or range reads with result = "auto" while keeping format semantics above the store layer. Vectorized ranges, vectorized keys, and non-auto result shapes bypass this cache adapter and read from the parent store directly.

Examples

root <- tempfile("ropendal-store-")
dir.create(root)
fs <- opendal("fs", root = root)
store <- byte_store(fs, "array.zarr")
store_write(store, "zarr.json", charToRaw("{}"))
#> [1] TRUE
as.raw(store_read(store, "zarr.json"))
#> [1] 7b 7d
cached <- store_cache(store, tempfile("ropendal-cache-"), validate = "none")
as.raw(store_read(cached, "zarr.json"))
#> [1] 7b 7d