DuckDB-Native Semantic and Retrieval Relations • ducksemantics

ducksemantics is a DuckDB-native semantic and retrieval substrate for R. It owns ontology graphs, validated HPO observations, cataloged Monarch relations, snapshot-bound literature retrieval, source-grounded judgments, and optional local model/provider protocols. It does not own ClinVar/PubMed source storage, case ranking, or evaluation.

Bulk values are ordinary data frames or caller-owned DuckDB relations. S7 is reserved for real prompt, embedding, parser, and annotator provider protocols. See ARCHITECTURE.md for the current boundary.

Install

install.packages(
  "ducksemantics",
  repos = c("https://rgenomicsetl.r-universe.dev", "https://cloud.r-project.org")
)

Ontology candidates and accepted HPO observations

Lexical candidates are intentionally distinct from accepted observations. An accepted observation must carry its HPO identifier, exact zero-based half-open source span, context, method/provider provenance, confidence, and explicit accepted status.

library(ducksemantics)

documents <- data.frame(
  document_id = "case-001",
  source_text = "The patient has seizures.",
  stringsAsFactors = FALSE
)

observations <- data.frame(
  document_id = "case-001",
  hpo_id = "HP:0001250",
  start_offset = 16L,
  end_offset = 24L,
  source_text = "seizures",
  context_status = "present",
  method = "lexical_alias",
  provider_id = "local-lexicon",
  provider_version = "1",
  confidence = 1,
  status = "accepted",
  stringsAsFactors = FALSE
)

ducksemantics_hpo_observations(documents, observations)

##   document_id     hpo_id start_offset end_offset source_text context_status
## 1    case-001 HP:0001250           16         24    seizures        present
##          method   provider_id provider_version confidence   status
## 1 lexical_alias local-lexicon                1          1 accepted

The validator rejects hallucinated text, empty spans, out-of-bounds spans, and any context outside present, absent/negated, family_history, uncertain, conflict, or unsupported.

Cataloged Monarch facts

Callers provide both facts and a typed release catalog. A catalog row identifies one provider/release and supplies a Date/POSIXct effective date or a numeric source ordinal. Exact and as-of queries bind the provider plus a cataloged release; historical gene-disease holdouts are explicit anti-join audits.

releases <- data.frame(
  release_id = c("2024-01", "2025-01"),
  provider_id = "Monarch",
  effective_date = as.Date(c("2024-01-01", "2025-01-01")),
  source_ordinal = c(1, 2),
  stringsAsFactors = FALSE
)
gene_disease <- data.frame(
  gene_id = c("HGNC:1100", "HGNC:1100"),
  disease_id = c("MONDO:0000001", "MONDO:0000002"),
  release_id = c("2024-01", "2025-01"),
  provider_id = "Monarch",
  stringsAsFactors = FALSE
)

gene_disease <- ducksemantics_monarch_import(
  gene_disease, releases, relation = "gene_disease"
)
ducksemantics_monarch_gene_disease_holdout_audit(
  gene_disease, releases, provider_id = "Monarch",
  train_release_id = "2024-01", holdout_release_id = "2025-01"
)

##     gene_id    disease_id               predicate release_id provider_id
## 1 HGNC:1100 MONDO:0000002 biolink:associated_with    2025-01     Monarch
##   source_version attrs train_release_id holdout_release_id
## 1           <NA>  <NA>          2024-01            2025-01

Attached dated Monarch annotation packs

For an official dated Monarch DuckDB pack already available to DuckDB, attach it outside the package, verify its immutable receipt, and bind one exact typed catalog row. The projection does not attach URLs, download or collect pack rows into R, make a copy, or accept latest; it creates seven connection-local TEMP VIEWs over the attached pack. The role views retain every raw edge and add normalized roles only when explicit Biolink subject/object categories support the orientation. The source contract matches Monarch’s serialized text negated field and also projects node_has_phenotype. Filter on a supported_* association_status deliberately: malformed categories or negation, missing predicates, negated edges, and unsupported orientations are returned as statuses, never silently treated as support or causal evidence. negation_status still distinguishes an omitted optional qualifier from explicit false, explicit true, and malformed text; omission does not erase Monarch’s positive source assertion.

conn <- ducksemantics_connect()
DBI::dbExecute(
  conn,
  "ATTACH '/data/monarch-kg/2026-07-14/monarch-kg.duckdb' AS monarch (READ_ONLY)"
)

releases <- data.frame(
  provider_id = "infores:monarchinitiative",
  release_id = "2026-07-14",
  effective_date = as.Date("2026-07-14"),
  stringsAsFactors = FALSE
)
views <- ducksemantics_monarch_project_pack(
  conn, source_catalog = "monarch", releases = releases,
  provider_id = "infores:monarchinitiative", release_id = "2026-07-14"
)

DBI::dbGetQuery(
  conn,
  "SELECT gene_id, disease_id, predicate, primary_knowledge_source
   FROM semantic_monarch_gene_disease
   WHERE association_status IN (
     'supported_subject_to_object', 'supported_object_to_subject'
   )"
)

The returned views relation names views rather than materializing pack contents. The native association view preserves edge identifiers, predicate and category, primary/aggregator provenance, evidence/publication/qualifier arrays, agent and knowledge fields, taxa, negation, and raw original fields. No provider edges are voted or merged.

Snapshot-bound literature retrieval

RClinVarbitration owns append-only PubMed source history and ducksemantics neither imports nor stores literature. Retrieval consumes the same provider-scoped source-order projection:

snapshots: provider_id, snapshot_id, finite integer high_water_ordinal, and optional POSIXct effective_at;
article versions: provider_id, article_id, pmid, version_id, finite integer source_ordinal, and logical is_deleted;
version-bound sections: provider_id, article_id, pmid, version_id, finite integer source_ordinal, section, and exact non-empty text.

For a cataloged provider/snapshot, it selects the maximum source_ordinal <= high_water_ordinal per article, excludes an article when that selected event is deleted, and joins sections only on all version keys. Thus order is numeric (for example 10 follows 9), not lexical. Exact nonempty zero-based half-open source spans are returned.

This is directly derivable from RClinVarbitration without a storage adapter: pubmed_sources supplies provider, snapshot, source ordinal, and application timestamp; pubmed_articles supplies the event and deletion flag; and pubmed_abstracts supplies same-source version-bound sections. A caller may also project article_title into a title section. The relations remain caller-owned; ducksemantics does not write or materialize a shadow copy.

snapshots <- data.frame(
  provider_id = c("pubmed", "pubmed"),
  snapshot_id = c("baseline", "update"),
  high_water_ordinal = c(9, 10),
  effective_at = as.POSIXct(c("2025-01-09", "2025-01-10"), tz = "UTC"),
  stringsAsFactors = FALSE
)
articles <- data.frame(
  provider_id = "pubmed",
  article_id = c("pmid:123", "pmid:123"),
  pmid = "123",
  version_id = c("source-9", "source-10"),
  source_ordinal = c(9, 10),
  is_deleted = c(FALSE, FALSE),
  stringsAsFactors = FALSE
)
sections <- data.frame(
  provider_id = "pubmed",
  article_id = c("pmid:123", "pmid:123"),
  pmid = "123",
  version_id = c("source-9", "source-10"),
  source_ordinal = c(9, 10),
  section = "BACKGROUND",
  text = c("Seizure phenotype in a family", "Updated seizure phenotype"),
  stringsAsFactors = FALSE
)

ducksemantics_literature_retrieve(
  articles, sections, snapshots,
  query = "seizure",
  provider_id = "pubmed",
  snapshot_id = "update"
)

##   provider_id snapshot_id high_water_ordinal effective_at article_id pmid
## 1      pubmed      update                 10   2025-01-10   pmid:123  123
##   version_id source_ordinal    section start_offset end_offset source_text
## 1  source-10             10 BACKGROUND            8         15     seizure
##                section_text
## 1 Updated seizure phenotype