Storage, caching, and full-release performance • RClinVarbitration

What is cached?

RClinVarbitration uses several distinct forms of reuse. Calling all of them a “cache” hides important lifecycle differences.

Download cache

rclinvarbitration_download_clinvar() stores official NCBI files under tools::R_user_dir("RClinVarbitration", "cache") by default. It accepts either release = "latest" or a specific monthly archive such as "2026-03".

latest <- rclinvarbitration_download_clinvar("latest")
archived <- rclinvarbitration_download_clinvar("2026-03")

The downloader fetches NCBI’s MD5 sidecar for VCV XML and for current flat files. A matching local file is reused. A stale or corrupt file is replaced only after a complete temporary download passes its checksum. NCBI does not publish adjacent MD5 sidecars for archived flat files, so those validation-only files are reused by name unless overwrite = TRUE.

latest is a mutable remote alias. For an auditable run, retain the returned URL and MD5 and assign an immutable release_id; selecting an explicit monthly archive is preferable.

Exact DuckDB extension artifacts

At package installation, the native C source is built separately for every supported DuckDB version. rclinvarbitration_enable() selects the artifact whose embedded DuckDB version and exact PRAGMA platform value match the active connection. Windows packages include both windows_amd64 and R-devel’s windows_amd64_mingw metadata identities over the same Rtools/MinGW machine code because DuckDB validates that identity literally. This is package-owned compatibility material, not downloaded code and not a cache that silently falls back to another DuckDB ABI.

Persistent relational release store

A file-backed DuckDB database is the durable, queryable conversion of the XML. After import, ordinary queries read typed relations and never rescan or decompress the XML. Multiple release_id values may coexist in one database. The clinvar_releases row records completion, source path, optional source URL and MD5, byte size, and import time.

library(DBI)
library(duckdb)

xml <- rclinvarbitration_download_clinvar("2026-03")
db <- file.path(tools::R_user_dir("RClinVarbitration", "data"), "clinvar.duckdb")
dir.create(dirname(db), recursive = TRUE, showWarnings = FALSE)
con <- dbConnect(duckdb(
  dbdir = db,
  config = list(
    allow_unsigned_extensions = "true",
    memory_limit = "2GB",
    preserve_insertion_order = "false",
    threads = "2"
  )
))
rclinvarbitration_enable(con)
rclinvarbitration_import_xml(con, xml, release_id = "ncbi-vcv-2026-03")

Passing the downloader’s returned vector directly lets the importer retain its URL and digest metadata. Extracting only the bare character path drops that R attribute; source_url and source_md5 can then be supplied explicitly.

Import lifecycle

The conversion has four phases:

XML.GZ forward scan
  -> temporary compact entity relation
  -> typed public relations
  -> clinvar_releases completion marker
  -> temporary relation dropped

The native table function performs one forward libxml2 scan and streaming gzip decompression. It emits one compact JSON-backed row per selected entity. SQL then projects those rows into the typed relations without an EAV pivot, XML DOM, or R data-frame copy.

The staging relation is a durable DuckDB table during conversion so its size is bounded by disk rather than R memory. It is dropped after projection, but DuckDB keeps the released blocks inside the database file for reuse. Therefore the physical file records the import high-water mark and can be substantially larger than its currently used blocks. This is reusable database capacity, not a second live copy of the XML after import.

Configure the database and DuckDB temp_directory on storage with substantial headroom. If the smallest possible single-release artifact matters more than fast reuse, publish selected relations to a fresh DuckDB database, Parquet, or DuckLake after validation; that compaction/publication time is separate from the import benchmark below.

Public relations commit independently so one enormous transaction is not held for the complete release. The release-catalogue marker is inserted only after all projections succeed. An R error removes partial rows and drops staging. If the process is killed so abruptly that cleanup cannot run, no completion marker exists; a later import drops stale staging and clears partial rows for the same release_id before writing the release again.

replace = TRUE stages and validates the XML scan before deleting the existing release. A scan failure therefore leaves the previous complete release intact. A later projection failure is cleaned up and does not restore the replaced rows, so production publication should use a new immutable release ID and switch consumers only after success.

What remains virtual?

Views such as clinvar_disease_submissions, clinvar_policy_decisions, clinvar_gene_summaries, and clinvar_semantic_documents are SQL definitions, not automatically refreshed materialized caches. DuckDB plans them against the stored source relations on each query.

For repeated delivery workloads, explicitly materialize a release- and policy-versioned result to Parquet, a DuckDB table, or DuckLake. Do not replace the auditable source tables with only the final classification label.

The package deliberately avoids ART indexes on release-scale tables. Their build and maintenance memory cost is undesirable during ingestion; analytical filters, scans, joins, Parquet statistics, and downstream materializations are the intended execution path.

DuckDB’s buffer manager may retain recently read blocks in process memory and spill intermediates to temp_directory. That runtime buffer cache is managed by DuckDB and disappears when the process exits. It is different from the durable database and the NCBI download cache.

Measured complete-release conversion

A complete NCBI VCV release was measured, rather than extrapolated from the one-record fixture. The committed machine-readable receipt is inst/benchmarks/full-release-2026-07-02.dcf, and tools/benchmark_full_release.R reproduces the measurement.

Measurement	Result
NCBI release	2026-07-02
Source MD5	`2e7e76ebbf668910b8688cc5e4284c1b`
Compressed XML	5,824,540,370 bytes (5.42 GiB)
Import wall time	1,312.493 seconds (21 min 52.5 sec)
Durable DuckDB file	23,978,586,112 bytes (22.33 GiB)
Peak process RSS	3,657,296 KiB (3.49 GiB)
VCV / allele rows	4,531,457 / 4,535,897
RCV / SCV rows	5,966,166 / 6,905,758
Condition / text rows	13,815,000 / 30,649,367

The run used DuckDB 1.5.3 on Linux, an Intel Core i5-13500, two DuckDB threads, a 2 GB DuckDB memory limit, and preserve_insertion_order = false. A DuckDB memory limit is not a process RSS limit: native parser, compression, allocator, and other process memory remain outside that configured buffer budget.

The timed scope is rclinvarbitration_import_xml() only. Connection creation, extension loading, schema initialization, final CHECKPOINT, and process startup are excluded. The complete process including those operations took 21 minutes 53.3 seconds.

At the final checkpoint, 29,874 of 91,471 256-KiB blocks were used and 61,597 were free. Thus the 22.33-GiB physical file contained about 7.29 GiB of live blocks and 15.04 GiB reserved for reuse after staging was dropped. Report the physical file size for disk planning; report used/free blocks as well when explaining logical storage.

Runtime and compression vary with DuckDB version, CPU, storage, filesystem cache, and configuration; this result is an observed reference, not a universal guarantee.

Run the benchmark with:

CLINVAR_DUCKDB_MEMORY_LIMIT=2GB \
CLINVAR_DUCKDB_THREADS=2 \
Rscript tools/benchmark_full_release.R \
  ClinVarVCVRelease_2026-03.xml.gz \
  clinvar-2026-03.duckdb \
  ncbi-vcv-2026-03 \
  clinvar-2026-03-benchmark.dcf

Use /usr/bin/time -v or the platform equivalent around that command when an operating-system peak-RSS measurement is required.