Skip to contents

Semantic retrieval, DuckLake, and VariantStory

Source: vignettes/semantic-ducklake-variantstory.Rmd
semantic-ducklake-variantstory.Rmd

Motivation beyond variant annotation

RClinVarbitration is not only a faster way to attach an aggregate label to an allele. The VCV release contains attributable submission text, condition and phenotype links, gene relations, observations, and literature identifiers. A relational representation makes that evidence reusable for:

  • phenotype- and text-driven dynamic gene-panel proposals;
  • retrieval of submission rationales and literature around a candidate;
  • disease-aware review of conflicting or VUS assertions;
  • prioritization of novel variants through gene, disease, phenotype, and literature context; and
  • release-to-release reanalysis.

Embedding similarity or a database neighbor must not itself reclassify a VUS or novel variant. It can propose candidates and retrieve source evidence. Clinical evidence admission, deterministic criteria, and human review remain separate steps.

Discovery-oriented views

Four views provide compact starting surfaces:

View Purpose
clinvar_semantic_documents attributable text with VCV, SCV, submitter, and submitted classification
clinvar_hpo_terms normalized HP:nnnnnnn identifiers from ClinVar cross-references
clinvar_literature_links PubMed, DOI, PMC, Bookshelf, and source URLs with assertion context
clinvar_gene_summaries disease-aware classification counts by release, policy profile, and gene

On the complete 2026-07-02 release these views expose 30,649,367 semantic text rows, including 4,213,102 comments and 5,937,873 submitted descriptions; 24,178 HPO links covering 2,628 distinct terms; and 8,252,297 citation/link rows. These are observed source counts, not claims that every text or link is independent or clinically admissible.

A complete scan of the gene summary view produced 92,789 genes in 20.7 seconds on the benchmark host. Materializing it once took 23.2 seconds, after which an exact BRCA1 lookup took 3 milliseconds. Workloads that repeatedly query gene summaries should therefore version a materialization rather than recompute the view for every request.

CREATE TABLE clinvar_gene_summary_cache AS
SELECT * FROM clinvar_gene_summaries;

SELECT *
FROM clinvar_gene_summary_cache
WHERE gene_key = 'ncbigene:672';

Using ducksemantics

ducksemantics can store HPO ontology structure and dense or late-interaction embeddings in DuckDB. The two packages can use one connection: RClinVarbitration owns ClinVar source semantics, while ducksemantics owns ontology and retrieval mechanics.

Select an attributable and bounded document collection before embedding. For example, comments and submitted descriptions are generally more useful than embedding every HGVS or assertion-method row.

library(ducksemantics)

# `con` is the already initialized RClinVarbitration DuckDB connection.
ducksemantics_init(con)

documents <- DBI::dbGetQuery(con, "
  SELECT semantic_document_id, text
  FROM clinvar_semantic_documents
  WHERE release_id = 'ncbi-vcv-2026-07-02'
    AND section IN ('comment', 'attribute:Description')
    AND length(text) >= 40
")

vectors <- ducksemantics_embed_cached(
  documents$text,
  provider = dense_provider,
  cache_dir = "clinvar-embedding-cache"
)
ducksemantics_embedding_batch(
  vectors,
  subject_id = documents$semantic_document_id,
  subject_kind = "clinvar_document",
  provider = "embeddinggemma-clinvar-v1",
  text = documents$text
) |>
  ducksemantics_write_embeddings(con, replace = TRUE)

dense_provider is deliberately not created by RClinVarbitration: model, weights digest, task prompt, dimensions, and privacy policy belong to the semantic run receipt. The stored subject_id joins every hit back to the exact ClinVar text row.

Direct HPO identifiers can join a pinned HPO graph without embedding:

SELECT h.vcv_accession, h.scv_entity_id, h.hpo_id, n.label
FROM clinvar_hpo_terms h
JOIN semantic_nodes n
  ON n.family = 'HPO' AND n.node_id = h.hpo_id
WHERE h.release_id = 'ncbi-vcv-2026-07-02';

After a semantic search writes or registers a semantic_hits(subject_id, score) relation, a transparent dynamic gene-panel proposal can be formed:

SELECT cg.symbol AS gene_symbol,
       max(h.score) AS best_text_score,
       count(DISTINCT d.semantic_document_id) AS supporting_documents,
       max(g.pathogenic_allele_count) AS clinvar_pathogenic_alleles
FROM semantic_hits h
JOIN clinvar_semantic_documents d
  ON d.semantic_document_id = h.subject_id
JOIN clinvar_genes cg
  ON cg.release_id = d.release_id AND cg.vcv_accession = d.vcv_accession
LEFT JOIN clinvar_gene_summary_cache g
  ON g.release_id = d.release_id AND g.symbol = cg.symbol
GROUP BY cg.symbol
ORDER BY best_text_score DESC, supporting_documents DESC;

This is a candidate panel with explicit retrieval support, not a validated gene-disease panel. Preserve score, source document IDs, release, model receipt, and policy profile so reviewers can inspect why a gene was proposed.

Publishing releases to DuckLake

The source database keeps attributable evidence. The publication table is tidy and long: decisions, SCVs, RCVs, and gene links are separate rows joined by stable allele and disease-decision keys.

tidy_path <- tempfile(fileext = ".parquet")
tidy <- rclinvarbitration_export_clinvarbitration_parquet(
  con,
  tidy_path,
  release_id = "ncbi-vcv-2026-07-02",
  assembly = "GRCh38",
  profile_id = "default",
  schema = "tidy"
)
ducklake::set_ducklake_connection(con)
ducklake::attach_ducklake("clinvar_lake", lake_path = "clinvar-lake")
publication <- rclinvarbitration_publish_ducklake(con, tidy)

The RGenomicsETL ducklake-r fork registers the Parquet in place. The publisher validates unique keys, uses persistent empty staging, and commits changed, new, and withdrawn decisions in one snapshot. Query that snapshot with ducklake::get_table_changes(). Compare releases with the policy fixed; compare policies with the source release fixed.

VariantStory boundary

VariantStory separates source observations, case-independent classification, case prioritization, evidence admission, and review. RClinVarbitration fits as a release-pinned supplementary-source adapter:

  • raw SCVs, text, HPO links, and citations are source observations;
  • clinvar_policy_decisions is a provider- and policy-scoped derived observation, not a native VariantStory SVC verdict;
  • clinvar_policy_pathogenic_alleles supports the named ClinVar P/LP prioritization module after exact assembly/allele and disease-context joins;
  • semantic/HPO retrieval may propose gene-condition candidates or literature claims, but cannot admit evidence on its own; and
  • DuckLake release and transform receipts allow VariantStory to recompute only contexts affected by a new ClinVar release or arbitration profile.

VariantStory does not yet expose an executable RClinVarbitration adapter. Until that contract is implemented and tested, use these relations as an explicit integration design, not a claim that ClinVar evidence is automatically mapped to SVC criteria.

Talos 11.1 compatibility target

Talos commit dc0278df0c0af80614963444f3766e22e8124c27 (version 11.1.0) is a useful new pinned oracle. It removes Talos’s private BCFtools fork because the required greedy coding/non-coding behavior is in upstream BCFtools 1.24, and invokes bcftools csq --greedy 1 --local-csq. It also introduces STRipy-derived short-tandem-repeat records as a separate Talos variant type.

That sharpens two integration boundaries:

  • DuckHTS/VariantStory compatibility should target official BCFtools 1.24 with the exact --greedy 1 --local-csq profile, not an unspecified BCFtools result or the former private fork.
  • STR evidence needs its own callset identity, repeat-count and disease-range semantics, inheritance logic, and evaluation profile. ClinVar gene/HPO/text relations can support retrieval for an STR locus, but an allele-level ClinVarbitration label must not be silently applied to a repeat expansion.

RClinVarbitration still supplies release-pinned ClinVar source observations; it does not own BCFtools consequence projection or STR genotype interpretation.

For VUS and novel-variant workflows, the safe sequence is:

case allele + consequence
  -> exact ClinVar allele/disease observations
  -> HPO and semantic candidate retrieval
  -> attributable text and literature inspection
  -> explicit evidence claims and admission
  -> deterministic condition-specific evaluation
  -> human review

This preserves the useful discovery signal while keeping source authority, model proposals, policy decisions, and clinical review distinguishable.