Evidence before answers. PAIL is a local-first control plane between retrieval and an LLM. It verifies business identifiers and relations, withholds contradictory evidence, quarantines retrieved instructions, and releases a bounded, attributable packet only when the evidence survives deterministic checks.
Website · Five-minute setup · Offline Node Kit · Release checklist · Demo script · PostgreSQL capacity runbook · Real-data evaluation
This repository now contains the complete private middleware, the laboratory
website under apps/lab-site/, and the reviewed public developer surface under
public/developer-preview/. Only this repository is edited. The public preview
repository and the legacy website repository are downstream mirrors until their
cutover is complete.
The private monorepo must never be made public. See WORKSPACE.md for the daily workflow and Repository Structure for the visibility, naming, archive, and migration rules.
PAIL does not ask a model to decide whether retrieved evidence is true. It produces one of three explicit outcomes:
| Outcome | Meaning | Model call |
|---|---|---|
VERIFIED_PACKET |
Required anchors and relations survived the guard | Permitted from the bounded packet |
CONFLICT_WITHHELD |
Sources disagree on a protected relation | Blocked |
NO_VERIFIED_EVIDENCE |
Required evidence is missing | Blocked |
files / SQL / logs / external retriever
|
v
candidate evidence locator
|
v
exact anchors -> relation guard -> contradiction guard
|
+-------+-------+
| | |
release withhold silence
|
v
bounded facts + sources + unknowns -> optional LLM
The locator may rank broadly, but it cannot establish truth. Learned feedback, semantic retrieval, and symbolic/acoustic resonance remain subordinate to the deterministic evidence guards.
PAIL is strongest when a question pins a business identifier — a case, order, transaction, trace, shipment, claim, or ticket ID — and a wrong answer has a measurable operational consequence. That is where its deterministic identity and relation guards have something to grip.
Best for
- Questions over records, transactions, logs, and tickets that carry an identifier (payments and banking, IT operations, logistics, insurance, citizen-grievance and public services).
- Regulated or air-gapped settings that cannot send data to a hosted model.
- Any place a confidently wrong answer is worse than an honest "not enough evidence to answer."
Not designed for
- Summarizing or answering over free-text documents that carry no identifiers (contracts, wikis, prose knowledge bases). There the identity guard has little to verify; ordinary RAG is the better tool for that job.
Works alongside your existing stack
- Bring your own retriever. PAIL's external-document path accepts results from your current vector or keyword search, then enforces "do not answer beyond what the retrieved evidence supports" and attaches provenance — adding a grounding check without replacing your pipeline.
| Release gate | Result |
|---|---|
| Exact and separator-drift anchor queries | 500 / 500 |
| Missing required anchors failed closed | 25 / 25 |
| Generated text-PDF retrieval campaign | 100 / 100 |
| Synthetic POS/payment safety campaign | 140 / 140 guarded outcomes |
| Adversarial campaign query latency | 2.92 ms p50 / 4.00 ms p95 |
| Broad-context token case | 6,930 -> 502 tokens (92.76%) |
| Small-row packet token case | Negative reduction; control overhead disclosed |
These are controlled local measurements, not customer production results or a claim of universal hallucination elimination. See the reproducible benchmark section below for workload details and known limits.
- Relations before relevance: an embedding score cannot silently choose between conflicting customer, payment, order, or account ownership.
- Fail closed by construction: missing evidence is an output state, not an invitation to answer from model memory.
- Guarded learning: reviewed feedback may adjust future routing, but cannot override identity or contradiction guards.
- Evidence-aware compression: token reduction happens only after required facts and provenance survive; overhead on tiny records is reported honestly.
- Grammar-inspired executable flow: SutraFlow compiles typed cells with a strict invariant: only deterministic guard cells may establish evidence validity.
- Local and model-optional: ingestion, verification, audit, RCA views, and packet creation run without an LLM.
PAIL is not a foundation model, replacement database, complete Paninian grammar implementation, or causal oracle. The acoustic layer is optional routing and short-payload transport research, not a claim that sound creates semantic understanding.
- CSV, JSON, JSONL, text, logs, HTML, XML, PDF, DOCX, and WAV-metadata ingestion
- bounded ASCII ISO 8583 bitmap manifests for reviewed POS/payment dialects
- SQLite FTS5 default storage; optional DuckDB and PostgreSQL/pgvector adapters
- field-aware canonical anchor index with restart-safe migration for legacy stores
- exact-anchor supplementation, relation conflict hard-stop, and fail-closed Sutra guard
- external-retriever path protected by the same guards as local retrieval
- bounded evidence packets with exact
tiktokenmeasurement when available - disclosed approximate token counting when an air-gapped tokenizer cache is unavailable
- HMAC-authenticated compact packets for exact singleton evidence; integrity metadata stays outside the LLM prompt
- occurrence-preserving investigation ledger; analysis reruns do not create incidents
- reviewed feedback with bounded Bayesian influence; it cannot override guards
- prompt-injection, PII, output grounding, usage, and HMAC audit controls
- retrieved instruction-bearing fields quarantined before compression or optional LLM dispatch
- encrypted/authenticated/replay-protected binary memory capsules
- experimental small-payload text-to-WAV capsule transport
- optional NumPy JEPA world model for numeric sensor transition anomaly signals
- authenticated local HTTP API, dashboard, CLI, and LangChain-style adapters
- hidden control-directory exclusion and configurable wide-record bounds
- Unicode-safe exact-token truncation and malformed-query fail-closed responses
- atomic browser uploads that retain collision-safe source filenames
- PII masking before optional LLM calls, API responses, feedback storage, and proof-card export
- HMAC-signed proof cards with plain-language refusal and conflict explanations
- bounded pre-guard point-in-time queries (
--as-of) that exclude future and undated candidates - real-parser schema preview with dynamic comma/semicolon/tab/pipe CSV dialect detection
- optional viewer/analyst/admin API roles and conservative read-only database connectors
- explicit reviewed alias promotion; ordinary corrections never silently rewrite schema mappings
- concise operator CLI by default, with complete diagnostics available through
--json - standalone, branch, and gateway node profiles with bounded local resource admission
- encrypted interrupted-ingest queue, retry/backoff, dead-letter visibility, and signed delta journal
- authenticated backup/restore with bounded key retention and an optional cloud-free health coordinator
- one-command labelled folder evaluation for releases, conflicts, refusals, latency, and token accounting
The Offline Node Kit adds reliability around the evidence firewall without changing its decision boundary. It is domain-neutral: a node can serve a laptop, field unit, factory cell, branch office, clinic, vehicle computer, or private-network gateway.
pail --config pail.json node init --mode branch --profile laptop
pail --config pail.json evaluate-folder ./sanitized-fixture --output evaluation.json
pail --config pail.json node status
pail --config pail.json backup create /secure-media/branch-01.pailbackupResource limits are process-level admission controls. CPU throttling is cooperative; use an operating-system or container quota for a hard CPU cap. The optional coordinator exchanges signed health summaries, not evidence, and does not turn this release into a managed fleet service. See the Offline Node Kit guide.
python3 -m venv .venv
source .venv/bin/activate
pip install -e '.[documents,learning]'
pail startpail start initializes a private local workspace, performs any one-time index
upgrade and opens the authenticated dashboard. It does not print the token or
send source data elsewhere. Technical operators can continue to use preview,
ingest, query, rca, migrate, and serve directly.
PAIL has three explicit answer modes:
pail query "case_id=CASE-42 status cause" --answer-mode packet
pail query "case_id=CASE-42 status cause" --answer-mode deterministic
pail query "case_id=CASE-42 status cause" --answer-mode modelpacket is the default. deterministic creates readable wording only from
released fields and needs no model. model requires ollama_model in
pail.json; strict JSON-schema and grounding checks run before prose is
released. Models are not called for missing or contradictory evidence.
The repeatable local-model campaign generates synthetic banking, startup, insurance and robotics evidence:
PYTHONPATH=src nice -n 15 python benchmarks/local_model_eval.py \
--model qwen2.5:1.5b --output-format schema --questions 40 --runs 3See local model guidance for the claim boundary, recorded settings and why a five-query probe is not a universal model ranking.
./deploy/docker/generate-secrets.sh
docker compose up --build -d pail
docker compose --profile observability up -d
docker compose psThe default ports bind only to host loopback: PAIL on 8791, Prometheus on
9090, and Grafana on 3000. The PAIL image runs as UID 10001 with a
read-only root filesystem and all Linux capabilities dropped. Production mode
requires viewer/analyst/admin role enforcement; TLS and external identity stay
the responsibility of an organization-managed ingress or gateway. See
deploy/README.md for Kubernetes and Splunk boundaries.
files / records / external retriever
-> occurrence-preserving ingestion
-> coarse candidate location
-> optional bounded feedback and sound-route nudge
-> exact relation guard
-> fail-closed grammar/Sutra guard
-> evidence dedupe and token budget
-> cited packet
-> optional PII masking at the process boundary
-> optional external/local LLM
-> output citation and grounding verification
The locator may rank candidates. It may not decide truth. Learned and semantic scores cannot override deterministic guards. Symbolic-sound and Samaveda resonance are secondary ordering keys only; they cannot overturn a higher deterministic score.
Unknown dynamic *_id fields are joinable but conservatively single-valued.
Domains with legitimate collections should register schema cardinality or add
metadata.relation_cardinality, for example
{"line_item_id": "many"}. Conflicting single-valued IDs hard-stop; declared
one-to-many values remain separate evidence rows.
The query hot path uses fast symbolic/electric signatures and does not generate WAV files. The optional acoustic transport exports a small encrypted capsule as an FSK-like waveform and decodes it with checksum validation. This proves a controlled short-payload store/carry/forward path; it is not yet a modem for large files, radio, underwater links, or noisy industrial channels.
This build is suitable for local demos, research, and controlled small/medium pilots. Local role tokens are not enterprise SSO. Enterprise launch still requires organization-specific RBAC/SSO, KMS key rotation, retained backups, observability integration, real customer data A/B evaluation, multi-node storage, and independent security review. PDF image OCR requires an external OCR adapter; a scanned PDF is reported rather than silently treated as empty.
The included benchmarks/release_campaign.py and
benchmarks/release_adversarial_campaign.py measured this build on the local
development machine:
- 10,007 adversarial records ingested at 1,250 records/s; all 500 exact/dirty identifier queries passed with 2.92 ms p50 and 4.00 ms p95 latency
- all missing-anchor, lifecycle, planted-conflict and retrieved-instruction
gates passed independently; exact
tiktokenwas required - the same 500 small-row packets had -127.95% mean token reduction: the full control/provenance envelope outweighed the tiny candidate records
- 10,000/10,000 records ingested; 200/200 anchored queries retrieved correctly
- 50/50 safe case/separator-drift queries retrieved correctly
- 100/100 generated text PDFs ingested and retrieved, with zero failed files
- conflicting external evidence produced a hard stop
- exact
tiktokenbroad-context measurement: 6,930 input tokens to 502 packet tokens, saving 6,428 tokens (92.76%) - compact one-record lookups averaged 51.145 source tokens and 55.145 packet tokens; the four-token marker overhead is reported rather than called compression
- a three-arm sound ablation measured 32.5% top-1 for deterministic tie order, 98.33% with symbolic tie-breaking, and no additional top-1 gain from Samaveda on that synthetic vocabulary-drift fixture; this is not a general semantic claim
- an optimized persisted 50,000-record run completed in 151.93 seconds and produced a 410.11 MiB local store; exact, planted-conflict, and retrieved- instruction checks completed in 0.99 to 1.08 seconds
- a copy of the earlier 50,000-record store was migrated to the typed anchor index in 52.97 seconds; afterward exact and conflict queries completed in 229 ms and 191 ms, while an absent identifier failed closed in 1.4 ms
- a separate 10,000,000-record streaming preflight normalized and hashed every generated record in 39.15 seconds with 21.55 MiB peak RSS and zero failures; it did not persist or query those records and is not a 10M capacity claim
- the 1.4.1 POS campaign generated 4,027 terminal, switch, settlement and log records for 1,000 synthetic transactions; 100 exact/state queries, 20 conflicts and 20 absent IDs all produced the expected guarded outcome, and seven retrieved instructions were quarantined; query latency was 991 ms p50 and 1.30 s p95 on this single-process local run
See the POS/payment test method for the public schema references, reproducible command and limits of this generated campaign.
The current 50k persisted result projects to roughly 8.4 hours and 80 GiB at 10M under naive linear growth. Full large-scale persistence therefore remains an open engineering problem, not a passed release gate.
The inherited V1.2.1 machine-readable artifacts remain under
benchmarks/results/ and retain the version that produced them. The current
build does not relabel those historical measurements as new benchmark evidence.
PAIL is developed and maintained by an independent enterprise registered as a Micro, Small & Medium Enterprise (MSME) in India.
| Udyam Registration | UDYAM-BR-37-0084592 |
| raushanraj1112@gmail.com | |
| Phone | +91 92413 86853 |
For pilots, integration support, or commercial enquiries, use the email above. For security reports, follow the process in SECURITY.md.