Skip to content

Faster defaults: standard OCR by default, cheaper E-Books listing; fix main CI - #6

Merged
carthworks merged 1 commit into
mainfrom
claude/pensive-shannon-afywuj
Oct 6, 2026
Merged

carthworks merged 1 commit into
mainfrom
claude/pensive-shannon-afywuj

Conversation

@carthworks

Copy link
Copy Markdown
Owner

Speed

Standard OCR is now the default. Advanced layout analysis measured about 3× slower per page (5.7 s vs 1.9 s on a 300 DPI page), but it only helps documents with tables, forms or stamps.

  • The Upload page now defaults to standard OCR.
  • The toggle explains the trade-off ("About 3× slower per page — turn on for forms and tables").
  • Each user's choice is remembered.

The E-Books listing is 4.3× faster. GET /api/projects/ebooks used to read every document's full OCR text just to compute a word count and a 300-character snippet.

  • Word counts are now stored in ocr_text.word_count.
  • The count is refreshed wherever text is written (OCR, cleanup, batch), and backfilled at startup for existing databases.
  • The snippet is cut inside SQLite.
  • Benchmark with 2,000 documents of 6,000 words each: 579 ms → 135 ms per request, with an identical response.

Fixes red CI on main

CI has failed on the last 4 pushes to main, all lint errors:

  • ESLint: unused imports in Cleanup.jsx and Metadata.jsx, and an unused extractionError state. The error was already shown as a toast, so removing the state loses nothing.
  • ruff: import ordering in routes/projects.py and services/pdfa.py.

Testing

  • Backend: 46 tests pass with veraPDF, ruff clean. New tests cover:
    • E-Books entries use stored word counts and 300-character snippets (no full text in the response).
    • Editing the text updates the count.
    • Counts are backfilled for databases created before this change.
  • ESLint: 0 errors. Build passes.
  • In the browser:
    • Advanced OCR is off by default and stays on after a reload once enabled.
    • The E-Books page renders word counts.
    • No page errors.

Note for a follow-up (not changed here)

GET /api/projects/<id>/file?type=pdf (added on main in 777c673) now runs a full OCR of a scanned PDF inside the request when no searchable PDF exists. On a long document that blocks the viewer for minutes. It would be better as a background job, with the viewer showing the original until the job finishes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Jze3nDgZGqUXGceiu5RCAf


Generated by Claude Code

…x main CI

Speed:
- Upload page defaults to standard OCR. Advanced layout analysis measured
  ~3x slower per page (5.7s vs 1.9s on a 300 DPI page) and only helps
  tables/forms/stamps; the toggle now says so and remembers each user's
  choice (localStorage).
- GET /api/projects/ebooks read every document's full OCR text to compute a
  word count and a 300-char snippet. Word counts are now stored
  (ocr_text.word_count, refreshed whenever OCR/cleanup/batch writes text,
  backfilled on startup for existing databases) and the snippet is cut in
  SQLite. 2,000 documents x 6,000 words: 579 ms -> 135 ms per request, same
  response.

CI on main has been red since 3ae2fc5 (lint only): unused imports/state in
Cleanup.jsx and Metadata.jsx (the extraction error was already shown as a
toast), and import ordering in routes/projects.py and services/pdfa.py.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Jze3nDgZGqUXGceiu5RCAf
@ecc-tools

ecc-tools Bot commented Oct 6, 2026

Copy link
Copy Markdown

ECC Tools / Security Evidence

Commit: 0621362afcec3e1417f80442cf15bd4fc42a4385

Security evidence gate passed (success)

No security-sensitive scanner-evidence gap detected.

Mode: enforce

Scanned 10 changed file(s). No missing scanner-evidence signal was detected.

Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission.

@ecc-tools

ecc-tools Bot commented Oct 6, 2026

Copy link
Copy Markdown

ECC Tools / PR Risk Taxonomy

Commit: 0621362afcec3e1417f80442cf15bd4fc42a4385

PR taxonomy review recommended (neutral)

Detected 1 PR taxonomy bucket(s): CI/CD Recommendation.

Scanned 10 changed file(s).

Roadmap taxonomy buckets:

CI/CD Recommendation

CI, dependency, coverage, and contract signals should be routed into follow-up checks or verification work.

Signals:

  • Regression coverage may lag behind the diff
  • API contract changes may ship without integration coverage
  • API implementation changes may ship without contract artifact updates
  • User-facing UI changes may ship without browser coverage
  • 0 CI or workflow path(s) changed

Paths:

  • backend/app/db.py
  • backend/app/routes/batch.py
  • backend/app/routes/documents.py
  • backend/app/routes/projects.py
  • backend/app/services/ocr_tasks.py
  • backend/app/services/pdfa.py
  • backend/tests/test_workflow.py
  • src/pages/Cleanup.jsx

Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission.

@ecc-tools

ecc-tools Bot commented Oct 6, 2026

Copy link
Copy Markdown

ECC Tools / Reference Set Readiness

Commit: 0621362afcec3e1417f80442cf15bd4fc42a4385

Reference set readiness gaps detected (neutral)

Reference evidence present for 0/7 areas (0%) across 10 changed file(s).

This check is based on files changed in this PR. Repository-level readiness is still reported by /ecc-tools analyze comments and generated manifests.

Area Status Evidence / Next Step
Deep analyzer corpus Missing Add analyzer fixture, golden, benchmark, or reference-set files that can catch analyzer regressions.
RAG/evaluator comparison Missing Add retrieval or evaluator reference-set comparison fixtures with expected ranking behavior.
PR salvage/review corpus Missing Add stale-PR, review-thread, reopen-flow, or salvage reference cases for queue cleanup automation.
Discussion triage corpus Missing Add public discussion triage fixtures, golden cases, or reference sets for informational, answered, and no-response classifications.
Harness compatibility Missing Add cross-harness, adapter-compliance, or harness-audit evidence for Claude, Codex, OpenCode, Zed, dmux, and agent surfaces.
Security evidence Missing Attach security evidence such as SBOMs, SARIF, audit reports, or AgentShield evidence packs.
CI failure-mode evidence Missing Add captured CI failure logs, dry-run fixtures, or troubleshooting docs for common workflow failure modes.

Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission.

@ecc-tools

ecc-tools Bot commented Oct 6, 2026

Copy link
Copy Markdown

ECC Tools / Hosted Promotion Readiness

Commit: 0621362afcec3e1417f80442cf15bd4fc42a4385

Hosted promotion readiness passed (success)

No hosted promotion evidence gaps detected across 10 changed file(s); 0 corpus scenarios had matching evidence.

This check compares PR file changes against the evaluator/RAG promotion corpus in src/analyzers/fixtures/evaluator-rag-corpus.ts.
Hosted output scoring inspected 0 completed cached hosted job results.

No evaluator corpus scenarios matched this PR.

Check publication was denied or unavailable. An app owner must enable Checks: read and write, and the installation owner must approve the updated permission.

@carthworks
carthworks merged commit 123a8ff into main Oct 6, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants