Skip to content

Complete projection integrity follow-up hardening - #123

Merged
joefeser merged 2 commits into
devfrom
codex/projection-followup-hardening
Sep 10, 2026
Merged

Complete projection integrity follow-up hardening#123
joefeser merged 2 commits into
devfrom
codex/projection-followup-hardening

Conversation

@joefeser

@joefeser joefeser commented Sep 10, 2026

Copy link
Copy Markdown
Owner

Summary

  • make SQLite declaration normalization quote-aware so future semantic literals cannot compare equal after case or whitespace substitution
  • pin clean case-distinct lifecycle filtering and unexpected trigger/view rejection with synthetic tests
  • replace version-as-capability quickstart guidance with an actual temporary projection read

Contract and compatibility impact

No schema identifiers, default query outputs, or published contracts change. Current v1 projection declarations remain compatible; formatting case and whitespace outside quoted tokens remain normalized, while quoted values and identifiers remain exact.

Security and authority

This strengthens generated projection identity and its regression evidence. It does not establish canonical-record authenticity or truth and creates no execution, disclosure, deployment, approval, or merge authority.

Validation

  • python3 -m unittest discover -s tests — 628 passed
  • ./scripts/run_conformance.sh — passed
  • ./scripts/validate_contracts.sh — 628 passed plus schema/fixture validation
  • python3 scripts/public_safety_check.py — passed across 350 commits, 4,559 historical objects, and 638 current paths
  • python3 scripts/run_cross_sqlite_matrix.py — passed across six discovered runtimes
  • documented temporary project/search capability proof — passed
  • git diff --check — passed

Closes #121
Closes #122

Summary by Sourcery

Harden SQLite projection integrity checks and capability validation without changing existing projection contracts.

Bug Fixes:

  • Preserve the exact semantics of quoted SQL strings and identifiers when comparing SQLite declarations.
  • Report SQLite projection creation failures as the typed projection-unavailable outcome instead of leaking runtime errors.

Enhancements:

  • Harden projection validation against unexpected triggers and table/view substitutions while preserving case-distinct lifecycle identities.
  • Replace SQLite version-based capability guidance with a behavioral temporary projection read.

Documentation:

  • Document quote-aware declaration comparison and behavioral SQLite/FTS5 capability verification in the quickstart and release notes.

Tests:

  • Add regression coverage for quoted declaration semantics, projection creation failures, case-distinct lifecycle filtering, and unexpected SQLite objects.

@sourcery-ai

sourcery-ai Bot commented Sep 10, 2026

Copy link
Copy Markdown

Reviewer's Guide

This PR hardens projection integrity by making declaration comparison quote-aware, adding regression tests for case-sensitive lifecycle identity and unexpected schema objects, and updating quickstart guidance to verify SQLite/FTS5 capability through an actual temporary projection read rather than a version floor.

Sequence diagram for behavioral SQLite capability verification

sequenceDiagram
    participant User
    participant CLI
    participant SQLite
    User->>CLI: project(record, --out temporary_directory)
    CLI->>SQLite: Build synthetic projection
    User->>CLI: search(records.sqlite, synthetic)
    CLI->>SQLite: Read projection and exercise FTS5 behavior
    alt capability demonstrated
        SQLite-->>CLI: Successful projection read
        CLI-->>User: Search result
    else capability unavailable
        SQLite-->>CLI: Capability failure
        CLI-->>User: projection-unavailable
    end
Loading

Flow diagram for quote-aware projection declaration validation

flowchart TD
    A[Read packaged declaration and sqlite_master declaration] --> B{Inside quoted token?}
    B -->|Yes| C[Preserve case whitespace and escapes]
    B -->|No| D[Ignore whitespace and normalize case]
    C --> E[Compare normalized declarations]
    D --> E
    E --> F{Declarations match?}
    F -->|Yes| G[Continue projection integrity checks]
    F -->|No| H[Reject projection]
Loading

File-Level Changes

Change Details Files
Made SQLite declaration identity normalization preserve quoted-token semantics while still ignoring formatting differences outside quoted tokens.
  • Implemented a quote-aware scanner for string, identifier, backtick, and bracket quoting, including doubled-quote escapes.
  • Documented that case and whitespace normalization applies only outside quoted tokens.
  • Added regression coverage for formatting equivalence and semantic differences in quoted values.
artifact_memory/projection.py
docs/contracts/v0-filesystem-and-projections.md
docs/release/v0.1.3-release-notes.md
tests/test_projection.py
Expanded projection integrity regression coverage for identity-preserving filtering and unexpected SQLite object rejection.
  • Added clean case-distinct lifecycle filtering coverage across literal and ranking search modes.
  • Added rejection tests for unexpected triggers and shadow table-to-view substitutions.
  • Pinned successful filtered receipts to verified integrity status and the expected surviving record.
tests/test_projection.py
Replaced SQLite version-based capability guidance with a behavioral temporary projection read.
  • Added quickstart commands to build and search a synthetic projection outside the repository.
  • Clarified that runtime version output is diagnostic only and capability is established by successful behavior.
  • Documented the typed projection-unavailable result for incapable runtimes.
docs/quickstart.md

Possibly linked issues


Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Harden SQLite projection declaration integrity

🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Preserve quoted SQL semantics while normalizing declaration formatting for projection identity
 checks.
• Add regressions for case-distinct lifecycle filtering and unexpected SQLite object substitutions.
• Document behavioral SQLite/FTS5 capability verification instead of version-based assumptions.
Diagram

graph TD
  A["Projection read"] --> B["Schema objects"] --> C["Quote-aware normalization"] --> D{"Contract matches?"}
  D -->|Yes| E["Integrity gate"] --> F["Verified query"]
  D -->|No| G["Reject projection"]
Loading
High-Level Assessment

The focused lexical scanner is appropriate because the contract only needs case and whitespace normalization outside SQLite quoted tokens. A general SQL parser or declaration round-trip was considered, but would add dependency or SQLite-reserialization complexity without improving this narrowly defined comparison.

Files changed (5) +147 / -4

Bug fix (1) +35 / -1
projection.pyMake SQL declaration normalization quote-aware +35/-1

Make SQL declaration normalization quote-aware

• Replaces global whitespace removal and lowercasing with a scanner that preserves string and identifier contents across single quotes, double quotes, backticks, and bracket quoting. Doubled quote escapes remain intact while insignificant whitespace and case outside quoted tokens are normalized.

artifact_memory/projection.py

Tests (1) +80 / -0
test_projection.pyExpand projection integrity regression coverage +80/-0

Expand projection integrity regression coverage

• Tests formatting-only declaration equivalence and preservation of quoted case and whitespace semantics. Adds clean case-distinct lifecycle filtering coverage plus fail-closed tests for unexpected triggers and FTS shadow-table replacement with a view.

tests/test_projection.py

Documentation (3) +32 / -3
v0-filesystem-and-projections.mdDefine quote-aware declaration comparison semantics +5/-1

Define quote-aware declaration comparison semantics

• Clarifies that projection validation normalizes case and whitespace only outside quoted tokens. Quoted identifiers, literals, whitespace, and doubled-quote escapes remain exact.

docs/contracts/v0-filesystem-and-projections.md

quickstart.mdDemonstrate SQLite capability through a projection read +25/-2

Demonstrate SQLite capability through a projection read

• Reframes the reported SQLite version as diagnostic information rather than proof of FTS5 integrity support. Adds commands that build and search a temporary synthetic projection to exercise the authoritative behavioral gate.

docs/quickstart.md

v0.1.3-release-notes.mdRecord strengthened declaration identity guarantees +2/-0

Record strengthened declaration identity guarantees

• Adds the quote-aware normalization behavior to the v0.1.3 projection integrity notes, emphasizing that quoted values and identifiers remain exact.

docs/release/v0.1.3-release-notes.md

sourcery-ai[bot]
sourcery-ai Bot previously approved these changes Sep 10, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="docs/quickstart.md" line_range="31" />
<code_context>
+projection outside the repository:
+
+```shell
+am_probe_dir="$(mktemp -d)"
+python3 -m artifact_memory project \
+  fixtures/synthetic/contracts/v0-valid-record.json \
+  --out "$am_probe_dir" \
+  --json
+python3 -m artifact_memory search \
+  "$am_probe_dir/records.sqlite" \
+  synthetic \
+  --json
+```
+
+A successful search demonstrates the required behavior for that loaded
</code_context>
<issue_to_address>
**nitpick (bug_risk):** The documented probe creates a directory with `mktemp -d` and never removes it, so every successful or failed quickstart verification leaves the generated SQLite projection and receipt files in the system temporary directory.

**Triggers:** When a user follows the documented capability probe.

**Suggested fix:** Wrap the commands in a cleanup trap, such as `trap 'rm -rf "$am_probe_dir"' EXIT`, after creating the temporary directory.

```suggestion
am_probe_dir="$(mktemp -d)"
trap 'rm -rf "$am_probe_dir"' EXIT
```
</issue_to_address>

Sourcery assessment

Approved.


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread docs/quickstart.md Outdated
@joefeser

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-10T14:39:58.267882Z a4b07b1 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@qodo-code-review

qodo-code-review Bot commented Sep 10, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. The capability probe crashes early ✓ Resolved 🐞 Bug ≡ Correctness
Description
The documented probe runs project before search, but projection creation executes the FTS5
schema without translating sqlite3.Error into ValidationFailure. When Python's loaded SQLite
build has no FTS5 module, the first command exits with an untyped traceback and the promised
projection-unavailable result is never reached.
Code

docs/quickstart.md[R32-35]

+python3 -m artifact_memory project \
+  fixtures/synthetic/contracts/v0-valid-record.json \
+  --out "$am_probe_dir" \
+  --json
Relevance

●●● Strong

Recent accepted findings require CLI validation boundaries to prevent uncaught exceptions and
preserve typed failure contracts.

PR-#79
PR-#86

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The added instructions invoke projection creation before search and promise that every incapable
build returns projection-unavailable. The project CLI catches only ValidationFailure, while
_create_sqlite directly executes the schema containing the FTS5 virtual table, so a missing FTS5
module raises an uncaught sqlite3.Error before the guarded search path can run.

docs/quickstart.md[27-44]
artifact_memory/cli.py[275-290]
artifact_memory/projection.py[356-360]
artifact_memory/schemas/core/index-sqlite.v1.sql[40-44]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The documented capability probe can fail with an uncaught SQLite exception during projection creation when the loaded runtime lacks FTS5, preventing the promised typed `projection-unavailable` outcome.

## Fix Focus Areas
- docs/quickstart.md[27-44]
- artifact_memory/cli.py[275-280]
- artifact_memory/projection.py[356-360]

## Recommended Fix
Translate SQLite failures encountered while creating the FTS5 projection into a `ValidationFailure` with the typed `projection-unavailable` code, ensuring the project command emits structured JSON instead of a traceback. Update the quickstart wording if necessary to clarify whether incapability is reported during projection creation or the subsequent search.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 18 rules
✅ Cross-repo context — repo relationships
Review mode: ⚖️ Balanced: This changes security-sensitive projection integrity validation and SQL declaration normalization across runtime logic and tests, but the logic is sufficiently localized for a careful single-pass review.

Grey Divider

Tip of the day
💡 Did you know, you can turn these tips off under Display preferences

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread docs/quickstart.md Outdated
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: a4b07b1b3d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant