diff --git a/reports/T-0121-INTEGRITY-SAFETY-AUDIT.md b/reports/T-0121-INTEGRITY-SAFETY-AUDIT.md new file mode 100644 index 00000000..04f29160 --- /dev/null +++ b/reports/T-0121-INTEGRITY-SAFETY-AUDIT.md @@ -0,0 +1,101 @@ + +Audits the authoritative SQLite claims corpus for credential, private-network, personal-path, and raw-code residue. +Checks generic, compiled-profile, entity-graph, and graph-observation supports for duplicates, orphans, and lifecycle violations. +Read this before any manual curation; it reports IDs and counts only and performs no database mutation. + + +# T-0121 integrity safety audit + +Observed 2026-08-13. Verdict: **FAIL sanitization; PASS structural lineage**. The failure is split: credentials and fenced code are historical, while private-IP and personal-path intake is ongoing. + +## Scope and method + +The authoritative SQLite database was opened with URI read-only mode. All 128,528 claims were scanned across `text`, `subject`, `predicate`, `object_value`, and `scope`. No matched value was printed or persisted; examples below are claim IDs only. The exact machine-readable evidence is in [t0121-integrity-safety-evidence.json](t0121-integrity-safety-evidence.json:1). + +Credential detection used the production decoded-variant scanner, which normalizes, decodes supported encodings, and returns finding labels without returning the secret. See `memorymaster/core/security.py:450-491`. Private IP, personal-path, fenced-code, and probable multiline-code checks were separate because bare private IPs are intentionally excluded from the ingest credential filter; the policy is documented at `memorymaster/core/security.py:55-60`. + +This is the safety subset only. Retrieval quality, memory usefulness, duplicates by meaning, contradiction quality, and curation decisions remain outside T-0121. + +## Finding S-01 — stored sanitization residue + +Severity: High. Exploitability: `EXPLOITABLE-LOW-EFFORT` for a process or operator already able to read the local database; no remote exploit path was evaluated. + +The required zero-residue criterion is not met: + +| Category | All stored | Confirmed | Other lifecycle states | Example human IDs | +|---|---:|---:|---|---| +| Credential or token detector | 65 | 0 | 48 archived, 6 stale, 11 superseded | `mm-4937`, `mm-9998`, `mm-c956`, `mm-88a3`, `mm-ee8d` | +| Bare private IP | 306 | 46 | 85 archived, 54 conflicted, 44 stale, 77 superseded | `mm-630f`, `mm-bccb`, `mm-56a6~2` | +| Personal path | 537 | 82 | 145 archived, 10 conflicted, 266 stale, 34 superseded | `mm-ef2a`, `mm-a2ba~2`, `mm-1a73` | +| Fenced code | 1,112 | 7 | 960 archived, 1 conflicted, 82 stale, 62 superseded | `mm-bc6f`, `mm-e1af`, `mm-1fb8~2` | +| Probable multiline code | 46 | 0 | 43 archived, 2 stale, 1 superseded | none confirmed | + +Evidence: `reports/t0121-integrity-safety-evidence.json:9-49`. + +The 65 credential/token detector hits are not confirmed claims, which limits ordinary trusted recall exposure, but they remain at rest and are labeled `public`, not `sensitive`. They are **not 65 confirmed secrets**. + +Redacted manual context review classified the 65 as: + +| Triage class | Count | Meaning | +|---|---:|---| +| Credential-bearing context | 20 | Historical context contains credential material; current validity was not tested | +| Non-secret context | 36 | False positive or a fixture, variable name, identifier, code example, or secret-file reference | +| Unresolved | 9 | Redaction removed too much context for a truthful classification | + +The live-secret count remains unknown. Evidence and exact human-ID sets: `reports/t0121-integrity-safety-evidence.json:86-122`. + +Value-safe identity grouping found **18 distinct credential identities across the 20 credential-bearing rows**. Exact comparison with the three previously accepted-risk credentials cannot be established safely because that decision supplies provider categories, not value fingerprints. The defensible result is therefore 0-3 possible matches and **15-18 different identities**. Context labels alone associate six identities with Supabase, none with the DashScope family, and twelve with neither; those labels are not identity proof. Values and hashes were neither emitted nor persisted. + +The current ingestion path does reject or classify sensitive claim content before graph extraction: `memorymaster/knowledge/entity_graph.py:182-199`. This finding is corpus residue, not evidence that the current graph-observation synthesizer accepted sensitive support. + +## Finding S-02 — recency separates historical residue from ongoing intake + +Severity: Medium. Exploitability: `BAD-PRACTICE`; this is a governance/intake-policy gap, not a demonstrated remote exploit. + +The 30-day window uses immutable `claims.created_at`, from 2026-07-14 20:05 UTC through 2026-08-13 20:05 UTC: + +| Category | Last 30 days | Confirmed in window | Latest hit | +|---|---:|---:|---| +| Credential or token detector | 0 | 0 | 2026-04-22 | +| Bare private IP | 61 | 22 | 2026-08-13 | +| Personal path | 50 | 32 | 2026-08-12 | +| Fenced code | 0 | 0 | 2026-05-12 | + +This makes credential and fenced-code residue historical. Private-IP and personal-path intake is ongoing and includes claims served by trusted recall. + +Recent private-IP hits by source agent: `claude-session=45`, `dream-worker=9`, `llm-stop-hook=4`, `atlas-llm-extractor=2`, `codex-session=1`. Recent personal-path hits: `claude-session=46`, `codex-session=2`, `dream-worker=1`, `atlas-llm-extractor=1`. + +For 2026-08-13 in Argentina, `claude-session` created 67 claims. It produced zero credential/token, personal-path, or fenced-code hits. It produced one private-IP hit, `mm-b33d`, currently conflicted. Value-redacted review confirms this is an actual NFS topology address, not a repo-relative-path false positive. + +Evidence: `reports/t0121-integrity-safety-evidence.json:52-83`. The production scanner intentionally excludes bare private IPv4 at ingest (`memorymaster/core/security.py:55-60`), which explains how the current operator rule can be violated without triggering the shared credential detector. + +## Finding I-01 — support lineage is structurally clean + +Severity: Informational. Exploitability: `BAD-PRACTICE` does not apply; no defect was found. + +All checked counts were zero: + +- SQLite foreign-key violations. +- Exact duplicate groups in `claim_evidence_links`, `claim_links`, `compiled_profile_supports`, `entity_edge_supports`, and `graph_observation_supports`. +- Orphans from those supports to claims, evidence, source items, entities, compiled facts, verbatim memories, entity edges, or graph observations. +- Entity-graph supports backed by non-confirmed, sensitive, observation-generated, or cross-claim-scope claims. +- Compiled-profile support session mismatches. + +Evidence: `reports/t0121-integrity-safety-evidence.json:124-170`. + +Graph-observation-specific support tables currently contain zero observations and zero support rows, so their orphan result is a valid empty-state check—not proof of live observation lineage. Entity-edge support has 268 rows, all attached to confirmed, non-sensitive, non-observation claims with exact claim/support scope. + +Entity records can retain the scope in which a canonical alias was first created; this is not counted as a lineage mismatch because the registry intentionally resolves an existing alias before creating a scoped entity (`memorymaster/knowledge/entity_registry.py:196-257`). Observation discovery partitions on the support row's exact scope and tenant (`memorymaster/knowledge/graph_observations.py:237-263`). + +## Disposition + +No claim, citation, support, event, or database row was deleted, rewritten, archived, or reclassified. This report and its count-only evidence artifact are the only changes. + +Remediation remains operator-gated. The proposed sequence is: + +1. Fix intake before cleanup through T-0127: reject bare private IPs and absolute personal paths on every claim-ingest surface while allowing repo-relative paths. +2. Add positive and negative enforcement tests for bare private IP, absolute personal path, repo-relative path, and fenced code. +3. Have a value-authorized human resolve the nine indeterminate credential hits; never convert the 20 credential-bearing contexts into “live secrets” without validity evidence. +4. Only after explicit operator approval, produce a no-write dry-run manifest separated into active served rows and historical at-rest rows. No bulk action should be inferred from detector matches. + +[SECTION COMPLETE: T-0121 safety subset] diff --git a/reports/t0121-integrity-safety-evidence.json b/reports/t0121-integrity-safety-evidence.json new file mode 100644 index 00000000..92f695c0 --- /dev/null +++ b/reports/t0121-integrity-safety-evidence.json @@ -0,0 +1,173 @@ +{ + "schema": "memorymaster.integrity-safety-audit.v1", + "task": "T-0121", + "observed_at": "2026-08-13T20:05:00Z", + "database": "authoritative SQLite; path intentionally omitted", + "database_open_mode": "read-only URI", + "mutations": 0, + "claims_scan": { + "rows_scanned": 128528, + "fields": ["text", "subject", "predicate", "object_value", "scope"], + "matched_content_emitted": false, + "credential_or_token": { + "total": 65, + "by_status": {"archived": 48, "stale": 6, "superseded": 11}, + "confirmed": 0, + "examples": ["mm-4937", "mm-9998", "mm-c956", "mm-88a3", "mm-ee8d"] + }, + "private_ip": { + "total": 306, + "by_status": {"archived": 85, "confirmed": 46, "conflicted": 54, "stale": 44, "superseded": 77}, + "examples_confirmed": ["mm-630f", "mm-bccb", "mm-56a6~2", "mm-a2ba~2", "mm-3442"] + }, + "personal_path": { + "total": 537, + "by_status": {"archived": 145, "confirmed": 82, "conflicted": 10, "stale": 266, "superseded": 34}, + "examples_confirmed": ["mm-ef2a", "mm-56a6~2", "mm-a2ba~2", "mm-3442", "mm-1a73"] + }, + "raw_code_fence": { + "total": 1112, + "by_status": {"archived": 960, "confirmed": 7, "conflicted": 1, "stale": 82, "superseded": 62}, + "examples_confirmed": ["mm-bc6f", "mm-e1af", "mm-1fb8~2", "mm-820a", "mm-f2ee"] + }, + "probable_multiline_code": { + "total": 46, + "by_status": {"archived": 43, "stale": 2, "superseded": 1}, + "confirmed": 0 + }, + "canonical_detector_counts": { + "compound_credential": 15, + "db_url_password": 2, + "hex_token": 1, + "home_path_unix": 71, + "home_path_windows": 24, + "password_assignment": 3, + "private_ip_port": 61, + "prose_password": 40, + "shell_abbr_password": 1, + "sshpass_flag": 2, + "telegram_bot_token": 3 + } + }, + "temporal_analysis": { + "basis": "immutable claims.created_at", + "window": "2026-07-14T20:05:00Z through 2026-08-13T20:05:00Z", + "credential_or_token": { + "last_30_days": 0, + "latest_created_at": "2026-04-22T22:16:01Z", + "latest_human_id": "mm-766b" + }, + "private_ip": { + "last_30_days": 61, + "last_30_days_confirmed": 22, + "by_source_agent": {"claude-session": 45, "dream-worker": 9, "llm-stop-hook": 4, "atlas-llm-extractor": 2, "codex-session": 1} + }, + "personal_path": { + "last_30_days": 50, + "last_30_days_confirmed": 32, + "by_source_agent": {"claude-session": 46, "codex-session": 2, "dream-worker": 1, "atlas-llm-extractor": 1} + }, + "raw_code_fence": { + "last_30_days": 0, + "latest_created_at": "2026-05-12T12:59:20Z", + "latest_human_id": "mm-7a4c" + }, + "today_claude_session": { + "day": "2026-08-13 America/Argentina/Buenos_Aires", + "claims": 67, + "credential_or_token": 0, + "private_ip": 1, + "personal_path": 0, + "raw_code_fence": 0, + "hit_human_ids": ["mm-b33d"], + "hit_statuses": ["conflicted"] + } + }, + "credential_triage": { + "detector_hits": 65, + "method": "manual context review after detector values, IPs, paths, emails and long identifiers were redacted", + "credential_bearing_context": { + "count": 20, + "meaning": "context contains credential material; current validity was not tested and these are not confirmed live secrets", + "human_ids": ["mm-4937", "mm-573b", "mm-c956", "mm-88a3", "mm-5c09", "mm-7693", "mm-ae83", "mm-cbbe", "mm-c52c", "mm-1823", "mm-84f7", "mm-b51c~2", "mm-0908", "mm-e6fe", "mm-36bd", "mm-7846", "mm-9879", "mm-f4f2", "mm-d0fd~5", "mm-21b4~2"] + }, + "non_secret_context": { + "count": 36, + "meaning": "detector false positive or non-secret fixture, variable name, code example, identifier, or secret-file reference", + "human_ids": ["mm-9998", "mm-f7c6", "mm-8f06", "mm-a900~2", "mm-7ae6", "mm-c0cc", "mm-b4ac", "mm-fcd8~3", "mm-b6e4", "mm-2db0", "mm-aa79", "mm-7440", "mm-014e", "mm-3095", "mm-2858", "mm-4f6a", "mm-1c63", "mm-808b", "mm-364a", "mm-543f", "mm-6819", "mm-0b57", "mm-5dc1", "mm-d7fb", "mm-ee8d", "mm-68bc", "mm-9932", "mm-0f1b", "mm-7307", "mm-6ad1", "mm-f2f1", "mm-b154~2", "mm-d754", "mm-ace4", "mm-f7d5", "mm-766b"] + }, + "unresolved": { + "count": 9, + "meaning": "redaction removed enough context that truth cannot be established without a separate value-aware human review", + "human_ids": ["mm-cab6", "mm-a188", "mm-52ae", "mm-2fa2", "mm-e836", "mm-7c1f~2", "mm-ccf2~2", "mm-d323", "mm-7466"] + }, + "live_secret_count": null, + "live_secret_count_reason": "No credential was tested; detector hits and credential-bearing historical context are not proof of current validity.", + "identity_analysis": { + "credential_bearing_context_rows": 20, + "distinct_credential_identities": 18, + "method": "in-memory SHA-256 grouping of detector-extracted values; values and hashes were not emitted or persisted", + "declined_reference_identities": 3, + "declined_reference_fingerprints_available": false, + "exact_declined_matches": null, + "exact_declined_matches_reason": "The accepted-risk decision identifies provider categories but supplies no safe identity fingerprints for comparison.", + "possible_declined_match_range": [0, 3], + "different_identity_range": [15, 18], + "context_labels_only": { + "supabase": 6, + "dashscope_family": 0, + "neither_named": 12, + "identity_proof": false + } + } + }, + "structural_integrity": { + "sqlite_foreign_key_violations": 0, + "duplicate_support_groups": { + "claim_evidence_links": 0, + "claim_links": 0, + "compiled_profile_supports": 0, + "entity_edge_supports": 0, + "graph_observation_supports": 0 + }, + "orphan_rows": { + "claim_evidence_links_to_claims": 0, + "claim_evidence_links_to_evidence_items": 0, + "claim_links_to_source_claims": 0, + "claim_links_to_target_claims": 0, + "compiled_profile_supports_to_facts": 0, + "compiled_profile_supports_to_verbatim": 0, + "entity_edge_supports_to_source_entities": 0, + "entity_edge_supports_to_target_entities": 0, + "entity_edge_supports_to_claims": 0, + "entity_edge_supports_without_entity_edge": 0, + "entity_edges_to_source_entities": 0, + "entity_edges_to_target_entities": 0, + "entity_edges_to_claims": 0, + "evidence_items_to_source_items": 0, + "graph_observations_to_claims": 0, + "graph_observation_supports_to_observations": 0, + "graph_observation_supports_to_claims": 0, + "graph_observation_supports_to_evidence": 0, + "graph_observation_supports_to_source_items": 0, + "graph_observation_supports_to_source_entities": 0, + "graph_observation_supports_to_target_entities": 0, + "graph_observation_supports_without_entity_edge_support": 0 + }, + "graph_support_lifecycle": { + "nonconfirmed_claims": 0, + "sensitive_claims": 0, + "observation_generated_claims": 0, + "claim_scope_mismatches": 0, + "compiled_profile_session_mismatches": 0 + }, + "row_counts": { + "claim_evidence_links": 229, + "claim_links": 365, + "compiled_profile_supports": 95, + "entity_edge_supports": 268, + "graph_observations": 0, + "graph_observation_supports": 0 + } + } +}