full repo can fall behind meta repo, undetected by the monitor (meta-vs-full load gap)

[tkuhn]

Summary

The full repo can fall behind its own meta repo, ending up with fewer nanopubs than meta reports. Because the monitor (and the Nanopub-Query-Loaded-Nanopub-Count/-Checksum headers) source their count/checksum from meta, this gap is invisible: the monitor shows full consensus while generic SPARQL queries against /repo/full silently return incomplete results.

Observed

All three query instances reported meta consensus (82210, checksum BLJdIPNk3QNCYys8…) — the value the monitor at https://monitor.knowledgepixels.com/ displays. But the full repo's own admin-graph count/checksum diverged:

repo	query.knowledgepixels.com	query.petapico.org	query.nanodash.net
meta	82210 BLJdIPNk…	82210 BLJdIPNk…	82210 BLJdIPNk…
full	82210 BLJdIPNk…	82134 ysRLX3H2…	82107 KIiaObIP…

(knowledgepixels' full matches only because it had just been FORCE_RELOADed.)

A per-week histogram of distinct ORCID signers confirms the gap is concentrated in recent (current-week) nanopubs — all prior weeks are byte-identical across instances. The same histogram run against /repo/meta is identical on all three (18 users in the current week); run against /repo/full it returns 18 / 14 / 2.

Why this is anomalous

executeLoading (NanopubLoader.java:402-499) submits the full/text/last30d/pubkey/type tasks, waits for all of them to succeed, and only then submits the meta task and bumps the count/checksum. By that ordering a nanopub reaches meta only after it is already in full, so full should always be ≥ meta. A full repo sitting below its own meta means full is missing nanopubs that meta already recorded — the "meta vs full load gap" already named in the comment at NanopubLoader.java:156.

So either:

• the gap was accumulated under older loader code (before the meta-deferred-until-others-succeed ordering), and persists because nothing reconciles it, or
• full-repo write failures are being swallowed somewhere without blocking the meta bump on those hosts.

Impact

• /repo/full SPARQL results are silently incomplete on affected instances.
• The monitor cannot detect it, since it only checks meta. "Consensus" on the dashboard does not imply full is synced.

Suggested actions

1. Detection: have the monitor (or a health check) compare per-repo npa:hasNanopubCount/npa:hasNanopubChecksum across meta and full (and ideally text/last30d), and flag any repo whose count/checksum drifts from meta.
2. Root cause: check whether full-repo load failures can complete without blocking the meta task (i.e. is the lock-step guarantee actually holding on all hosts?), and inspect loader logs on petapico/nanodash for swallowed full write errors around recent loads.
3. Remediation: FORCE_RELOAD rebuilds full to match meta (already done on knowledgepixels); apply to petapico/nanodash. Consider an automatic per-repo reconciliation rather than a manual full reload.

Reproduction

# count + checksum, run against /repo/meta and /repo/full on each instance
PREFIX npa: <http://purl.org/nanopub/admin/>
SELECT ?count ?checksum WHERE {
  GRAPH npa:graph {
    ?repo npa:hasNanopubCount ?count .
    ?repo npa:hasNanopubChecksum ?checksum .
  }
}

[tkuhn]

Root cause found

It is not an operator toggle of NANOPUB_QUERY_ENABLE_FULL_REPO — full was never disabled. The flag is being mis-read intermittently, and the read mechanism is the bug.

Evidence (petapico, 2026-06-01)

Diffing recent nanopubs between meta and full, all 76 missing ones share the same signature:

• present in meta (stamp + content)
• present in their per-pubkey repo (pubkey_*)
• absent from full — zero npa:hasLoadNumber stamp and zero content (never written, not deleted)
• spread across 6 signers and many types → not data-specific
• all created in the last ~week; full's latest write is in lock-step with meta now → the holes are frozen mid-stream, not a lagging tail

Why that pins it to the flag gate

In executeLoading the pubkey/type/meta tasks are ungated, but the full task is gated by if (FeatureFlags.fullRepoEnabled()) (NanopubLoader.java:414), and the wait loop requires every submitted non-meta task to succeed before the meta task is submitted. So meta=true + pubkey=true + full=false has exactly one explanation: the full task was never submitted, i.e. fullRepoEnabled() returned false for those individual loads. The lock-step ordering (defer-meta, since Jan) only guarantees full ≥ meta while the full task is submitted — the flag check sits before submission, outside the guard.

The mis-read

fullRepoEnabled() → Utils.getEnvString(...), which read the environment via Apache Commons Exec's EnvironmentUtils.getProcEnvironment():

String s = EnvironmentUtils.getProcEnvironment().get(envVarName);

getProcEnvironment() forks a subprocess (env/sh) and parses its stdout on every call. This runs on the per-nanopub hot path, for every feature flag, from the 4-thread loading pool — neither thread-safe nor robust under load. When it returns anything that isn't a clean "true", "true".equalsIgnoreCase(...) is false and the full write is silently skipped for that nanopub. The mechanism has existed since 2024, but only became a per-nanopub hot-path call in April 2026 (the per-repo flags, commit 44b16aa); recent RDF4J load pressure is what tipped it into intermittent failures.

Note on direction: a fork failure and an unset var both fall back to the default "true" (full stays enabled, safe). So this can only skip full if NANOPUB_QUERY_ENABLE_FULL_REPO is actually present in the process env (likely set to "true") and the subprocess read mangles/partials it — worth confirming on petapico.

Fix

PR #118 — switch Utils.getEnvString to System.getenv() (in-memory, thread-safe, deterministic; never forks or throws), and retire the same getProcEnvironment() call in the TripleStore constructor.

This stops the gap from growing but does not backfill the nanopubs already missing from full: those still need FORCE_RESYNC/FORCE_RELOAD (done on nanodash + knowledgepixels; petapico still pending). The detection suggestion in the original report — comparing per-repo npa:hasNanopubCount/-Checksum across meta/full — remains valuable as a guard, since the headers/monitor source only from meta.