diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md index c2e04ab10..ceef227af 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/STATUS.md @@ -79,10 +79,11 @@ child. 03B3B3A merged through PR #233. It installs only the approved `defusedxml` wheel and adds the shared bounded OPC/OOXML container security capability. -03B3B3B is the active successor. Its bounded DOCX extraction and durable -omission-fact implementation passed focused tests and required internal review; -hosted CI, CodeRabbit, and human merge remain pending. PPTX, XLSX, image, AUTH, -and sufficiency work remain inactive. +03B3B3B merged through PR #234. It adds bounded DOCX extraction and durable +omission facts on the shared OOXML boundary. 03B3B3C implements bounded PPTX +slide/notes extraction and has passed focused deterministic gates and required +internal review; hosted CI, CodeRabbit, and human merge remain pending. XLSX, +image, AUTH, and sufficiency work remain inactive. AUTH `WS-XINT-002-04B` follows the complete hidden split-03B series and activates only fixed-service binding and guide read. ART-03C then removes the diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B3C-pptx-extractor.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B3C-pptx-extractor.md index a1d10032b..ad47f7c2a 100644 --- a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B3C-pptx-extractor.md +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/chunks/WS-ART-001-03B3B3C-pptx-extractor.md @@ -31,6 +31,10 @@ P2 backend/app/modules/artifacts/guide_pptx.py backend/app/modules/artifacts/guide_extraction.py backend/app/modules/artifacts/guide_extraction_worker.py +backend/scripts/run_test_lanes.py +backend/tests/test_artifact_architecture.py +backend/tests/test_guide_bindings.py +backend/tests/test_guide_extraction.py backend/tests/test_guide_pptx.py backend/tests/fixtures/guide_pptx/** docs/spec_artifact_storage_service.md @@ -44,22 +48,71 @@ DOCX/XLSX behavior, unapproved packages, framework/AUTH/Celery/submission change ## Acceptance criteria - Require exact PPTX classification and the shared OOXML security boundary. -- Accept exactly 300 slides and reject 301 before extraction. Traverse slides - by presentation order, shapes by XML document order, and text paragraphs/runs - in document order; append speaker notes after their owning slide in notes XML - order. Omit masters, comments, hidden metadata, and embedded objects while - recording omission facts. Exceeding D42's output limit produces unusable - `limit_exceeded`, never partial agent input. +- Use exact policy identity `guide-extraction-v4`; obsolete `unsupported` or + older PPTX evidence cannot replay as current-policy success. The parent + result protocol accepts successful PPTX evidence only with the exact boolean + omission keys `truncated`, `omitted`, `masters`, `comments`, + `hidden_metadata`, `non_text_objects`, and `embedded_objects`. + `truncated=false`; `omitted` is true exactly when any category is true. + Persistence and replay comparison bind the complete omission mapping to the + same canonical output. +- Emit compact sorted JSON with one `slides` array. Each entry has the exact + shape `{"notes":[...],"number":N,"text":[...]}`. `number` is the + one-based presentation position. `text` contains one string per DrawingML + `a:p` in slide shape-tree XML order, with `a:t` run text concatenated in XML + order, `a:tab` represented as tab, and `a:br` represented as newline. + Hyperlink display text and placeholder text remain visible. Grouped shapes + recurse in XML order; table paragraphs follow table row/cell XML order. + Empty text-bearing paragraphs, empty slides, and empty notes remain explicit. + Pictures, charts, diagrams, media, OLE/package objects, alt text, and other + non-text drawing metadata never enter canonical output. +- `notes` uses the same paragraph/run rules and follows notes shape-tree XML + order. Notes placeholders of type `hdr`, `ftr`, `dt`, `sldNum`, and `sldImg` + are metadata and are omitted; body/object notes and ordinary text shapes are + retained. Notes are embedded only in their owning slide entry, never emitted + as an independent slide. +- `ppt/presentation.xml` `p:sldIdLst` is the sole slide-order authority. Resolve + each `r:id` through `ppt/_rels/presentation.xml.rels` using the exact passive + Transitional + `http://schemas.openxmlformats.org/officeDocument/2006/relationships/slide` + or Strict `http://purl.oclc.org/ooxml/officeDocument/relationships/slide` + relationship type. Resolve an optional notes slide through that slide's + `.rels` using the corresponding exact Transitional or Strict `notesSlide` + URI; a package may not mix those relationship namespaces. Resolve + case-variant stored part names through the validator-approved case-folded + map. Missing/dangling/duplicate IDs or targets, cross-root/type mismatches, + one notes part owned by multiple slides, and orphan slide or notes parts fail + `malformed/pptx_relationship_conflict`; they are never silently omitted. +- Accept exactly 300 ordered slides and reject 301 before slide/notes extraction + with `limit_exceeded/pptx_slide_limit`. Missing/unreadable presentation parts + use `malformed/pptx_presentation_unavailable`. Non-well-formed or unsafe XML + is rejected first by the shared boundary as `malformed/ooxml_unsafe_xml`; + well-formed invalid presentation roots use + `malformed/pptx_invalid_presentation_xml`, invalid slide roots use + `malformed/pptx_invalid_slide_xml`, and invalid notes roots use + `malformed/pptx_invalid_notes_xml`. Invalid relationship XML or ownership + remains `malformed/pptx_relationship_conflict`. Reject traversal beyond 64 + nested shape/text/container levels as + `malformed/pptx_nesting_limit`. Exceeding D42's exact 4 MiB output limit produces unusable + `limit_exceeded/output_limit`, never partial agent input. +- Successful omission facts record master/handout/notes-master parts, + comment/comment-author parts, hidden document/custom/alt/visibility metadata, + and passive non-text drawing objects. Active embedded content remains a + shared OOXML malformed rejection before PPTX extraction. - Prove deterministic slide/notes output, unsafe/malformed input, child-only - imports, exact 300/301 and output boundaries, crash, timeout, cancellation, - cleanup, approval-gate, and coverage behavior. + imports, exact relationship/orphan handling, 300/301, depth and output + boundaries, crash, timeout, cancellation, cleanup, approval-gate, complete + cross-process omission facts, v4 persistence/replay identity, and coverage + behavior. Assign the focused PPTX module to the existing canonical hosted + semantic lane without changing lane or coverage policy. ## Verification commands ```bash (cd backend && uv run ruff check app tests) (cd backend && python scripts/check_guide_extractor_dependencies.py) -(cd backend && uv run pytest -q tests/test_guide_ooxml.py tests/test_guide_pptx.py tests/test_guide_extraction.py --cov=app.modules.artifacts --cov-report=term-missing --cov-fail-under=90) +(cd backend && uv run pytest -q tests/test_guide_ooxml.py tests/test_guide_pptx.py tests/test_guide_extraction.py tests/test_guide_bindings.py tests/test_artifact_architecture.py tests/test_ci_test_lanes.py) +(cd backend && uv run pytest -q tests/test_guide_pptx.py --cov=app.modules.artifacts.guide_pptx --cov-report=term-missing --cov-fail-under=90) (metadata_dir="$(mktemp -d)" && trap 'rm -rf "$metadata_dir"' EXIT && (cd backend && WORKSTREAM_TEST_ADMIN_DATABASE_URL=postgresql+asyncpg://workstream:workstream@localhost:5433/postgres .venv/bin/python scripts/run_isolated_tests.py --metadata-json "$metadata_dir/result.json" --timeout-seconds 12600 -- .venv/bin/python -m pytest -q --ignore=tests/test_isolated_database_runner.py --cov=app --cov-report=term-missing --cov-fail-under=78)) python3 scripts/check_stale_artifact_contracts.py python3 scripts/check_markdown_links.py diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B3B3C-external-review-response.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B3B3C-external-review-response.md new file mode 100644 index 000000000..e59e624bb --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B3B3C-external-review-response.md @@ -0,0 +1,50 @@ +# External Review Response: WS-ART-001-03B3B3C + +Reviewed PR: `#235` + +Reviewed head: `c3d8fd367d66` + +Reviewed at: `2026-07-31` + +## Comments addressed + +- Threaded the existing shape-tree traversal depth into paragraph text + traversal so the exact 64-level limit cannot reset at an `a:p` boundary. +- Read slide visibility from the parsed `p:sld` root rather than the + presentation's `p:sldId`, and retained `hidden_metadata` provenance without + discarding visible slide text. +- Reused the existing `_replace_member` test helper for mixed namespace, wrong + relationship type, and malformed relationship-root fixtures. +- Added a cross-boundary nesting regression whose outer containers and inner + paragraph nesting are each individually below the limit but exceed 64 in + combination. + +## Comments deferred + +None. + +The CodeRabbit docstring-coverage warning is not a code finding: the hosted +Backend `Docstring coverage` step passed on the reviewed PR head. No coverage +rule or production docstring was changed in response to that stale warning. + +## Human decisions needed + +None. + +## Commands rerun + +```text +cd backend +.venv/bin/python -m ruff format app/modules/artifacts/guide_pptx.py tests/test_guide_pptx.py +.venv/bin/python -m ruff check app tests scripts +.venv/bin/python -m pytest -q tests/test_guide_pptx.py \ + --cov=app.modules.artifacts.guide_pptx --cov-branch \ + --cov-report=term-missing --cov-fail-under=90 +``` + +Result: 21 passed; 94.94 percent branch coverage. + +## Remaining risks + +Hosted Backend and Agent Gates must rerun on the repair commit. CodeRabbit must +confirm the updated head or leave only resolved/outdated threads before merge. diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B3B3C-internal-review-evidence.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B3B3C-internal-review-evidence.md new file mode 100644 index 000000000..00a487450 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B3B3C-internal-review-evidence.md @@ -0,0 +1,61 @@ +# Internal Review Evidence: WS-ART-001-03B3B3C + +Reviewed against trusted main: `a1b3fd0e410d` + +Reviewed at: `2026-07-31` + +## Candidate + +Hidden deterministic PPTX slide and notes extraction on the merged shared +OOXML boundary. The candidate emits bounded canonical content and exact durable +omission facts without activating AUTH, guide sufficiency, XLSX/image work, or +contributor submissions. + +## Deterministic Evidence + +- Ruff, approved extractor-dependency gate, stale artifact contracts, Markdown + links, lane integrity, and `git diff --check`: PASS; +- focused PPTX adapter suite: 21 passed at 94.43 percent branch coverage; +- focused OOXML, PPTX, extraction, architecture, and lane suite: PASS; +- DB-backed binding/replay tests and repository-wide coverage remain delegated + to the hosted sharded Backend gate; no local full-suite run was used. + +## Reviewer Results + +| Reviewer | Result | Blocking findings | +|---|---|---| +| architecture | PASS | none | +| security/auth | PASS | none | +| QA/test | PASS WITH LOW RISKS | none | +| senior engineering | PASS | none | +| product/ops | PASS WITH LOW RISKS | none | +| reuse/dedup | PASS WITH LOW RISKS | none | +| CI integrity | PASS | none | +| test delta | PASS | none | +| docs | PASS | none | + +## Material Repairs + +- restricted canonical text to the exact slide/notes shape tree and explicit + text-body/table capabilities; +- skipped chart, picture, media, and embedded-object subtrees while preserving + bounded omission evidence; +- made every semantic and omission traversal depth bounded, including table + discovery and skipped notes placeholders; +- required exact relationship roots, direct relationship rows, matching + Transitional/Strict families, unique slide and relationship identities, and + exact slide/notes ownership; +- recorded hidden slide visibility and hidden metadata inside omitted subtrees; +- documented shared OOXML malformed-XML precedence and the exact PPTX v4 + omission schema. + +## Accepted Low Risks + +- DB-backed immutable persistence/replay and repository-wide coverage evidence + must come from the hosted Backend gate on the committed PR head. +- DOCX and PPTX currently repeat small adapter-loading and bounded JSON-output + patterns; deduplication should be considered only if XLSX would repeat them. + +Valid findings addressed: yes + +Open sub-agent sessions: none diff --git a/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B3B3C-pr-trust-bundle.md b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B3B3C-pr-trust-bundle.md new file mode 100644 index 000000000..5b832ba50 --- /dev/null +++ b/.agent-loop/initiatives/WS-ART-001-immutable-artifact-storage/reviews/WS-ART-001-03B3B3C-pr-trust-bundle.md @@ -0,0 +1,105 @@ +# WS-ART-001-03B3B3C PR Trust Bundle + +## Chunk + +`WS-ART-001-03B3B3C` — PPTX Extractor (L1). + +## Goal and human-approved intent + +Add deterministic, bounded PPTX slide and notes extraction after exact PPTX +classification and the merged OOXML security boundary. Preserve original +verified guide bytes as authoritative. This hidden chunk must not activate +AUTH, invoke guide sufficiency, add XLSX/image behavior, or touch contributor +submission ZIPs. + +## What changed and why + +- Added an isolated PPTX adapter with exact presentation-order slide and notes + extraction, deterministic compact JSON, and fixed omission evidence. +- Advanced PPTX evidence to `guide-extraction-v4` and bound its complete result + schema to immutable persistence and replay checks. +- Reused the worker-owned shared OOXML validator and kept the PPTX adapter + import confined to the isolated child. +- Added the canonical semantic lane, architecture and worker-order proofs, + DB-backed replay cases, and the exact storage-service contract. + +## Design chosen and alternatives rejected + +The adapter accepts only bytes that pass the shared OOXML boundary. It resolves +the exact presentation relationship graph, traverses only approved shape-tree +text bodies and tables, and returns canonical content plus a fixed omission +map. Rejected alternatives were request-path parsing, direct provider access, +generic ZIP handling, raw binary agent input, broad XML scanning, partial +output, or a second authorization/OOXML protocol. + +## Scope control and product behavior + +Only hidden guide-source extraction changes. There is no route, authorization +activation, Celery continuation, sufficiency invocation, provider read, guide +binding, submission, checker, review, contribution, payment, or reputation +change. Artifact/parser failures remain bounded internal outcomes rather than +guide-insufficiency decisions. + +## Acceptance criteria proof + +- Presentation order, grouped shapes, tables, notes ownership, exact paragraph + semantics, Strict/Transitional relationships, and empty structures have + deterministic canonical-output tests. +- Masters, comments, hidden metadata, passive non-text objects, and embedded + objects use an exact boolean omission schema bound to v4 evidence. +- Malformed relationship roots, duplicates, dangling/cross-root targets, + orphans, shared ownership, namespace mixing, and invalid parts fail closed. +- Exactly 300 slides pass; 301 slides, depth beyond 64, and output beyond 4 MiB + fail without partial agent input. +- The isolated runner proves v4 protocol transport and scratch cleanup; + architecture tests prove worker-only adapter imports. +- Persistence tests bind canonical output and complete omission facts and reject + obsolete PPTX policy evidence as a replay target. + +## Tests and checks + +- Ruff and approved extractor-dependency gate — pass. +- Focused OOXML/PPTX/extraction/architecture/lane suite — pass. +- PPTX module branch coverage — 94.94 percent on the external-review repair + head (21 tests). +- Stale artifact contracts, Markdown links, lane integrity, and + `git diff --check` — pass. +- Hosted Backend/Agent Gates retain DB-backed replay, repository-wide coverage, + and semantic-lane proof; no local full-suite run was used. + +## Test delta and CI integrity + +No test, assertion, lane, workflow, dependency rule, or coverage threshold was +removed, skipped, or weakened. The new PPTX module joins the existing +`shared_foundations` lane. No dependency or workflow file changed. + +## Reviewer results + +Architecture, security, senior engineering, CI integrity, test delta, and docs +pass. QA, product/ops, and reuse/dedup pass with only documented low risks. All +blocking traversal, non-text leakage, relationship, visibility, depth, and +contract/test findings were repaired and re-reviewed. + +## External review + +CodeRabbit identified two correctness gaps and one test-maintainability issue: +depth reset at the paragraph boundary, slide visibility read from `p:sldId` +instead of `p:sld`, and repeated ZIP-fixture mutation code. All three are +repaired with focused regression proof. Its docstring warning was stale; the +hosted Backend docstring gate passed. Hosted checks must rerun on the repair +head before human merge approval. + +## Remaining risks and follow-up + +DB-backed replay and repository-wide coverage evidence is hosted-only. If the +next XLSX adapter repeats the small OOXML loader or serializer patterns, assess +a bounded shared helper then rather than widening this chunk. XLSX, image, +durable sufficiency continuation, AUTH activation, and legacy cutover remain +separate later chunks. + +## Human review focus and merge ownership + +Review exact relationship ownership/order, structure-aware text eligibility, +omission semantics, v4 replay identity, depth/output limits, and absence of AUTH +or sufficiency activation. A human owns merge approval; the agent will not +merge this PR. diff --git a/backend/app/modules/artifacts/guide_extraction.py b/backend/app/modules/artifacts/guide_extraction.py index 6088239e8..fd7e5e670 100644 --- a/backend/app/modules/artifacts/guide_extraction.py +++ b/backend/app/modules/artifacts/guide_extraction.py @@ -16,12 +16,13 @@ EXTRACTION_POLICY_VERSION = "guide-extraction-v1" PDF_EXTRACTION_POLICY_VERSION = "guide-extraction-v2" DOCX_EXTRACTION_POLICY_VERSION = "guide-extraction-v3" +PPTX_EXTRACTION_POLICY_VERSION = "guide-extraction-v4" EXTRACTOR_VERSION = "1" MAXIMUM_INPUT_BYTES = 32 * 1024 * 1024 MAXIMUM_OUTPUT_BYTES = 4 * 1024 * 1024 MAXIMUM_PROTOCOL_BYTES = (MAXIMUM_OUTPUT_BYTES * 6) + 1024 WALL_TIMEOUT_SECONDS = 60 -_SUPPORTED = frozenset({"plain_text", "markdown", "json", "csv", "pdf", "docx"}) +_SUPPORTED = frozenset({"plain_text", "markdown", "json", "csv", "pdf", "docx", "pptx"}) _DEFAULT_OMISSION_FACTS = {"truncated": False, "omitted": False} _DOCX_OMISSION_KEYS = frozenset( { @@ -36,6 +37,17 @@ "field_instructions", } ) +_PPTX_OMISSION_KEYS = frozenset( + { + "truncated", + "omitted", + "masters", + "comments", + "hidden_metadata", + "non_text_objects", + "embedded_objects", + } +) @dataclass(frozen=True, slots=True) @@ -166,11 +178,12 @@ def _valid_omission_facts(detected_format: str, status: object, value: object) - isinstance(key, str) and isinstance(item, bool) for key, item in value.items() ): return False - if detected_format != "docx" or status != "extracted": + if detected_format not in {"docx", "pptx"} or status != "extracted": return value == _DEFAULT_OMISSION_FACTS - if frozenset(value) != _DOCX_OMISSION_KEYS or value["truncated"]: + expected_keys = _DOCX_OMISSION_KEYS if detected_format == "docx" else _PPTX_OMISSION_KEYS + if frozenset(value) != expected_keys or value["truncated"]: return False - categories = _DOCX_OMISSION_KEYS - {"truncated", "omitted"} + categories = expected_keys - {"truncated", "omitted"} return value["omitted"] == any(value[key] for key in categories) @staticmethod @@ -212,4 +225,6 @@ def extraction_policy_version(detected_format: str) -> str: return PDF_EXTRACTION_POLICY_VERSION if detected_format == "docx": return DOCX_EXTRACTION_POLICY_VERSION + if detected_format == "pptx": + return PPTX_EXTRACTION_POLICY_VERSION return EXTRACTION_POLICY_VERSION diff --git a/backend/app/modules/artifacts/guide_extraction_worker.py b/backend/app/modules/artifacts/guide_extraction_worker.py index 48e65e1e4..b2ba89fcf 100644 --- a/backend/app/modules/artifacts/guide_extraction_worker.py +++ b/backend/app/modules/artifacts/guide_extraction_worker.py @@ -138,6 +138,7 @@ def _extract( detected_format: str, pdf_extractor: Callable[[bytes], str] | None = None, docx_extractor: Callable[[bytes], tuple[str, dict[str, bool]]] | None = None, + pptx_extractor: Callable[[bytes], tuple[str, dict[str, bool]]] | None = None, ) -> str | tuple[str, dict[str, bool]]: if detected_format == "pdf": if pdf_extractor is None: @@ -147,6 +148,10 @@ def _extract( if docx_extractor is None: raise ExtractionFailure("parser_failure", "parser_unavailable") return docx_extractor(payload) + if detected_format == "pptx": + if pptx_extractor is None: + raise ExtractionFailure("parser_failure", "parser_unavailable") + return pptx_extractor(payload) text = _decode_text(payload) if detected_format in {"plain_text", "markdown"}: return text @@ -225,6 +230,25 @@ def bounded_extract(payload: bytes) -> tuple[str, dict[str, bool]]: return bounded_extract +def _load_pptx_extractor( + validate_ooxml: Callable[[bytes, str], object], +) -> Callable[[bytes], tuple[str, dict[str, bool]]]: + """Load the PPTX adapter after limits but before descriptor-only seccomp.""" + from app.modules.artifacts.guide_pptx import PptxExtractionFailure, extract_pptx + + def bounded_extract(payload: bytes) -> tuple[str, dict[str, bool]]: + try: + extracted = extract_pptx( + payload, + validate_ooxml=lambda exact_payload: validate_ooxml(exact_payload, "pptx"), + ) + except PptxExtractionFailure as exc: + raise ExtractionFailure(exc.status, exc.code) from exc + return extracted.canonical_output, extracted.omission_facts + + return bounded_extract + + def main() -> int: """Apply resource/isolation controls and emit one bounded JSON result.""" detected_format = sys.argv[1] if len(sys.argv) == 2 else "" @@ -232,15 +256,24 @@ def main() -> int: _install_limits() pdf_extractor = None docx_extractor = None + pptx_extractor = None if detected_format == "pdf": pdf_extractor = _load_pdf_extractor() elif detected_format == "docx": docx_extractor = _load_docx_extractor(_load_ooxml_security()) + elif detected_format == "pptx": + pptx_extractor = _load_pptx_extractor(_load_ooxml_security()) _install_seccomp() payload = sys.stdin.buffer.read(_MAXIMUM_INPUT_BYTES + 1) if len(payload) > _MAXIMUM_INPUT_BYTES: raise ExtractionFailure("limit_exceeded", "input_limit") - extracted = _extract(payload, detected_format, pdf_extractor, docx_extractor) + extracted = _extract( + payload, + detected_format, + pdf_extractor, + docx_extractor, + pptx_extractor, + ) if isinstance(extracted, tuple): output, omission_facts = extracted else: diff --git a/backend/app/modules/artifacts/guide_pptx.py b/backend/app/modules/artifacts/guide_pptx.py new file mode 100644 index 000000000..8992e35a9 --- /dev/null +++ b/backend/app/modules/artifacts/guide_pptx.py @@ -0,0 +1,459 @@ +"""Deterministic bounded PPTX extraction after shared OOXML validation.""" + +from __future__ import annotations + +from collections.abc import Callable +import codecs +from dataclasses import dataclass +from io import BytesIO +import json +import posixpath +from pathlib import PurePosixPath +from xml.etree import ElementTree +import zipfile + + +# Preload the ZIP filename codec before the isolated child installs seccomp. +codecs.lookup("cp437") +MAXIMUM_OUTPUT_BYTES = 4 * 1024 * 1024 +MAXIMUM_SLIDES = 300 +MAXIMUM_NESTING_DEPTH = 64 +_P_NS = "http://schemas.openxmlformats.org/presentationml/2006/main" +_A_NS = "http://schemas.openxmlformats.org/drawingml/2006/main" +_R_NS = "http://schemas.openxmlformats.org/officeDocument/2006/relationships" +_STRICT_R_NS = "http://purl.oclc.org/ooxml/officeDocument/relationships" +_REL_NS = "http://schemas.openxmlformats.org/package/2006/relationships" +_PRESENTATION_TAG = f"{{{_P_NS}}}presentation" +_SLIDE_TAG = f"{{{_P_NS}}}sld" +_NOTES_TAG = f"{{{_P_NS}}}notes" +_SLIDE_ID_LIST_TAG = f"{{{_P_NS}}}sldIdLst" +_SLIDE_ID_TAG = f"{{{_P_NS}}}sldId" +_RELATIONSHIP_ID_ATTRIBUTE = f"{{{_R_NS}}}id" +_STRICT_RELATIONSHIP_ID_ATTRIBUTE = f"{{{_STRICT_R_NS}}}id" +_RELATIONSHIP_TAG = f"{{{_REL_NS}}}Relationship" +_RELATIONSHIPS_TAG = f"{{{_REL_NS}}}Relationships" +_PARAGRAPH_TAG = f"{{{_A_NS}}}p" +_TEXT_TAG = f"{{{_A_NS}}}t" +_TAB_TAG = f"{{{_A_NS}}}tab" +_BREAK_TAG = f"{{{_A_NS}}}br" +_SHAPE_TAG = f"{{{_P_NS}}}sp" +_NON_VISUAL_SHAPE_PROPERTIES_TAG = f"{{{_P_NS}}}nvSpPr" +_NON_VISUAL_PROPERTIES_TAG = f"{{{_P_NS}}}nvPr" +_COMMON_SLIDE_DATA_TAG = f"{{{_P_NS}}}cSld" +_SHAPE_TREE_TAG = f"{{{_P_NS}}}spTree" +_TEXT_BODY_TAG = f"{{{_P_NS}}}txBody" +_PLACEHOLDER_TAG = f"{{{_P_NS}}}ph" +_NON_TEXT_TAGS = frozenset( + { + f"{{{_P_NS}}}pic", + f"{{{_P_NS}}}contentPart", + f"{{{_P_NS}}}media", + f"{{{_P_NS}}}video", + f"{{{_P_NS}}}audio", + } +) +_EMBEDDED_TAGS = frozenset({f"{{{_P_NS}}}oleObj", f"{{{_P_NS}}}externalData"}) +_GRAPHIC_FRAME_TAG = f"{{{_P_NS}}}graphicFrame" +_TABLE_TAG = f"{{{_A_NS}}}tbl" +_NON_CONTENT_NOTE_PLACEHOLDERS = frozenset({"hdr", "ftr", "dt", "sldnum", "sldimg"}) +_TRANSITIONAL_BASE = "http://schemas.openxmlformats.org/officeDocument/2006/relationships" +_STRICT_BASE = "http://purl.oclc.org/ooxml/officeDocument/relationships" +_RELATIONSHIP_FAMILIES = { + _TRANSITIONAL_BASE: { + "slide": f"{_TRANSITIONAL_BASE}/slide", + "notesSlide": f"{_TRANSITIONAL_BASE}/notesSlide", + }, + _STRICT_BASE: { + "slide": f"{_STRICT_BASE}/slide", + "notesSlide": f"{_STRICT_BASE}/notesSlide", + }, +} +_PPTX_OMISSION_KEYS = ( + "masters", + "comments", + "hidden_metadata", + "non_text_objects", + "embedded_objects", +) + + +@dataclass(frozen=True, slots=True) +class PptxExtraction: + """One canonical PPTX result and its fixed bounded omission facts.""" + + canonical_output: str + omission_facts: dict[str, bool] + + +class PptxExtractionFailure(Exception): + """Carry one bounded PPTX extraction outcome.""" + + def __init__(self, status: str, code: str) -> None: + super().__init__(code) + self.status = status + self.code = code + + +@dataclass(frozen=True, slots=True) +class _Relationship: + relationship_type: str + target: str + + +def _new_omissions(names: frozenset[str]) -> dict[str, bool]: + return { + "truncated": False, + "omitted": False, + "masters": any( + name.startswith(("ppt/slidemasters/", "ppt/notesmasters/", "ppt/handoutmasters/")) + for name in names + ), + "comments": any(name.startswith(("ppt/comments/", "ppt/commentauthors")) for name in names), + "hidden_metadata": any(name.startswith(("docprops/", "customxml/")) for name in names), + "non_text_objects": False, + "embedded_objects": False, + } + + +def _xml(payload: bytes, *, code: str, root_tag: str) -> ElementTree.Element: + try: + root = ElementTree.fromstring(payload) + except ElementTree.ParseError as exc: + raise PptxExtractionFailure("malformed", code) from exc + if root.tag != root_tag: + raise PptxExtractionFailure("malformed", code) + return root + + +def _relationship_family(relationship_type: str, kind: str) -> str | None: + for family, types in _RELATIONSHIP_FAMILIES.items(): + if relationship_type == types[kind]: + return family + return None + + +def _source_for_relationship_part(name: str) -> str: + path = PurePosixPath(name) + if len(path.parts) < 3 or path.parts[-2] != "_rels" or not path.name.endswith(".rels"): + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + return (path.parent.parent / path.name.removesuffix(".rels")).as_posix() + + +def _relationship_target(part_name: str, target: str) -> str: + if not target or target.startswith(("/", "\\")) or "\\" in target: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + source = _source_for_relationship_part(part_name) + normalized = posixpath.normpath(posixpath.join(posixpath.dirname(source), target)) + if normalized.startswith("../") or not normalized.startswith("ppt/"): + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + return normalized.casefold() + + +def _relationships(payload: bytes, *, part_name: str) -> dict[str, _Relationship]: + try: + root = ElementTree.fromstring(payload) + except ElementTree.ParseError as exc: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") from exc + if root.tag != _RELATIONSHIPS_TAG: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + relationships: dict[str, _Relationship] = {} + for element in root: + if element.tag != _RELATIONSHIP_TAG: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + relationship_id = element.get("Id") + relationship_type = element.get("Type") + target = element.get("Target") + if ( + not relationship_id + or not relationship_type + or not target + or relationship_id in relationships + ): + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + relationships[relationship_id] = _Relationship( + relationship_type=relationship_type, + target=_relationship_target(part_name, target), + ) + return relationships + + +def _paragraph_text(paragraph: ElementTree.Element, depth: int) -> str: + values: list[str] = [] + + def walk(element: ElementTree.Element, depth: int) -> None: + if depth > MAXIMUM_NESTING_DEPTH: + raise PptxExtractionFailure("malformed", "pptx_nesting_limit") + if element.tag == _TEXT_TAG: + values.append(element.text or "") + return + if element.tag == _TAB_TAG: + values.append("\t") + return + if element.tag == _BREAK_TAG: + values.append("\n") + return + for child in element: + walk(child, depth + 1) + + walk(paragraph, depth) + return "".join(values) + + +def _paragraphs( + root: ElementTree.Element, + omissions: dict[str, bool], + *, + notes: bool, +) -> list[str]: + paragraphs: list[str] = [] + + shape_tree = root.find(f"./{_COMMON_SLIDE_DATA_TAG}/{_SHAPE_TREE_TAG}") + if shape_tree is None: + return paragraphs + + def record_metadata(element: ElementTree.Element) -> None: + if any(key in element.attrib for key in ("descr", "title")) or str( + element.get("show", "1") + ).casefold() in {"0", "false", "off", "no"}: + omissions["hidden_metadata"] = True + + def scan_omissions(element: ElementTree.Element, depth: int) -> None: + if depth > MAXIMUM_NESTING_DEPTH: + raise PptxExtractionFailure("malformed", "pptx_nesting_limit") + record_metadata(element) + if element.tag in _NON_TEXT_TAGS: + omissions["non_text_objects"] = True + elif element.tag in _EMBEDDED_TAGS: + omissions["embedded_objects"] = True + for child in element: + scan_omissions(child, depth + 1) + + def walk_graphic_frame(element: ElementTree.Element, depth: int) -> bool: + if depth > MAXIMUM_NESTING_DEPTH: + raise PptxExtractionFailure("malformed", "pptx_nesting_limit") + record_metadata(element) + if element.tag == _TABLE_TAG: + walk(element, depth, text_allowed=True) + return True + found_table = False + for child in element: + found_table = walk_graphic_frame(child, depth + 1) or found_table + return found_table + + def walk(element: ElementTree.Element, depth: int, *, text_allowed: bool = False) -> None: + if depth > MAXIMUM_NESTING_DEPTH: + raise PptxExtractionFailure("malformed", "pptx_nesting_limit") + record_metadata(element) + if element.tag in _NON_TEXT_TAGS: + omissions["non_text_objects"] = True + scan_omissions(element, depth) + return + if element.tag in _EMBEDDED_TAGS: + omissions["embedded_objects"] = True + scan_omissions(element, depth) + return + if element.tag == _SHAPE_TAG and notes: + placeholder = element.find( + f"./{_NON_VISUAL_SHAPE_PROPERTIES_TAG}/" + f"{_NON_VISUAL_PROPERTIES_TAG}/{_PLACEHOLDER_TAG}" + ) + placeholder_type = ( + str(placeholder.get("type", "body")).casefold() + if placeholder is not None + else "body" + ) + if placeholder_type in _NON_CONTENT_NOTE_PLACEHOLDERS: + omissions["hidden_metadata"] = True + scan_omissions(element, depth) + return + if element.tag == _GRAPHIC_FRAME_TAG: + if not walk_graphic_frame(element, depth): + omissions["non_text_objects"] = True + return + if element.tag == _TEXT_BODY_TAG: + text_allowed = True + if element.tag == _PARAGRAPH_TAG and text_allowed: + paragraphs.append(_paragraph_text(element, depth)) + return + for child in element: + walk(child, depth + 1, text_allowed=text_allowed) + + walk(shape_tree, 0) + return paragraphs + + +def _part_names(names: frozenset[str], prefix: str) -> frozenset[str]: + return frozenset( + name + for name in names + if name.startswith(prefix) and name.endswith(".xml") and "/_rels/" not in name + ) + + +def _ordered_parts( + archive: zipfile.ZipFile, + stored: dict[str, str], + presentation: ElementTree.Element, + *, + names: frozenset[str], + omissions: dict[str, bool], +) -> tuple[list[tuple[str, str | None]], str]: + relationship_name = "ppt/_rels/presentation.xml.rels" + try: + presentation_relationships = _relationships( + archive.read(stored[relationship_name]), + part_name=relationship_name, + ) + except KeyError as exc: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") from exc + slide_list = presentation.find(_SLIDE_ID_LIST_TAG) + slide_ids = [] if slide_list is None else list(slide_list.findall(_SLIDE_ID_TAG)) + if len(slide_ids) > MAXIMUM_SLIDES: + raise PptxExtractionFailure("limit_exceeded", "pptx_slide_limit") + ordered_slides: list[str] = [] + family: str | None = None + seen_ids: set[str] = set() + seen_slide_ids: set[str] = set() + for slide_id in slide_ids: + canonical_slide_id = slide_id.get("id") + if not canonical_slide_id or canonical_slide_id in seen_slide_ids: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + seen_slide_ids.add(canonical_slide_id) + relationship_ids = tuple( + value + for attribute in (_RELATIONSHIP_ID_ATTRIBUTE, _STRICT_RELATIONSHIP_ID_ATTRIBUTE) + if (value := slide_id.get(attribute)) is not None + ) + if len(relationship_ids) != 1: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + relationship_id = relationship_ids[0] + if not relationship_id or relationship_id in seen_ids: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + seen_ids.add(relationship_id) + relationship = presentation_relationships.get(relationship_id) + if relationship is None: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + current_family = _relationship_family(relationship.relationship_type, "slide") + if current_family is None or (family is not None and family != current_family): + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + family = current_family + if relationship.target in ordered_slides or relationship.target not in stored: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + ordered_slides.append(relationship.target) + if frozenset(ordered_slides) != _part_names(names, "ppt/slides/"): + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + resolved: list[tuple[str, str | None]] = [] + owned_notes: set[str] = set() + for slide_name in ordered_slides: + path = PurePosixPath(slide_name) + rel_name = (path.parent / "_rels" / f"{path.name}.rels").as_posix() + notes_name = None + if rel_name in stored: + slide_relationships = _relationships( + archive.read(stored[rel_name]), + part_name=rel_name, + ) + notes = [ + relationship + for relationship in slide_relationships.values() + if _relationship_family(relationship.relationship_type, "notesSlide") is not None + ] + if len(notes) > 1: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + if notes: + notes_relationship = notes[0] + notes_family = _relationship_family( + notes_relationship.relationship_type, "notesSlide" + ) + if notes_family != family: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + notes_name = notes_relationship.target + if notes_name not in stored or notes_name in owned_notes: + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + owned_notes.add(notes_name) + resolved.append((slide_name, notes_name)) + if frozenset(owned_notes) != _part_names(names, "ppt/notesslides/"): + raise PptxExtractionFailure("malformed", "pptx_relationship_conflict") + return resolved, family or _TRANSITIONAL_BASE + + +def _canonical_slides( + archive: zipfile.ZipFile, + stored: dict[str, str], + parts: list[tuple[str, str | None]], + omissions: dict[str, bool], + *, + maximum_output_bytes: int, +) -> str: + serialized: list[str] = [] + byte_count = len(b'{"slides":[]}') + for number, (slide_name, notes_name) in enumerate(parts, 1): + slide = _xml( + archive.read(stored[slide_name]), + code="pptx_invalid_slide_xml", + root_tag=_SLIDE_TAG, + ) + if str(slide.get("show", "1")).casefold() in {"0", "false", "off", "no"}: + omissions["hidden_metadata"] = True + notes: list[str] = [] + if notes_name is not None: + notes_root = _xml( + archive.read(stored[notes_name]), + code="pptx_invalid_notes_xml", + root_tag=_NOTES_TAG, + ) + notes = _paragraphs(notes_root, omissions, notes=True) + entry = { + "notes": notes, + "number": number, + "text": _paragraphs(slide, omissions, notes=False), + } + encoded = json.dumps(entry, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + projected = byte_count + len(encoded.encode("utf-8")) + (1 if serialized else 0) + if projected > maximum_output_bytes: + raise PptxExtractionFailure("limit_exceeded", "output_limit") + serialized.append(encoded) + byte_count = projected + return '{"slides":[' + ",".join(serialized) + "]}" + + +def extract_pptx( + payload: bytes, + *, + validate_ooxml: Callable[[bytes], object], + maximum_output_bytes: int = MAXIMUM_OUTPUT_BYTES, +) -> PptxExtraction: + """Validate and extract one exact PPTX package without partial output.""" + try: + validate_ooxml(payload) + with zipfile.ZipFile(BytesIO(payload)) as archive: + stored = {info.filename.casefold(): info.filename for info in archive.infolist()} + names = frozenset(stored) + presentation = _xml( + archive.read(stored["ppt/presentation.xml"]), + code="pptx_invalid_presentation_xml", + root_tag=_PRESENTATION_TAG, + ) + omissions = _new_omissions(names) + parts, _family = _ordered_parts( + archive, + stored, + presentation, + names=names, + omissions=omissions, + ) + output = _canonical_slides( + archive, + stored, + parts, + omissions, + maximum_output_bytes=maximum_output_bytes, + ) + except PptxExtractionFailure: + raise + except KeyError as exc: + raise PptxExtractionFailure("malformed", "pptx_presentation_unavailable") from exc + except (OSError, RuntimeError, ValueError, zipfile.BadZipFile) as exc: + raise PptxExtractionFailure("malformed", "pptx_presentation_unavailable") from exc + omissions["omitted"] = any(omissions[key] for key in _PPTX_OMISSION_KEYS) + return PptxExtraction(canonical_output=output, omission_facts=omissions) diff --git a/backend/scripts/run_test_lanes.py b/backend/scripts/run_test_lanes.py index 9649cd391..9430b5967 100644 --- a/backend/scripts/run_test_lanes.py +++ b/backend/scripts/run_test_lanes.py @@ -110,6 +110,7 @@ class TestLane: "tests/test_guide_extractor_dependencies.py", "tests/test_guide_ooxml.py", "tests/test_guide_pdf.py", + "tests/test_guide_pptx.py", "tests/test_local_artifact_store.py", "tests/test_s3_artifact_store.py", "tests/test_test_lane_evidence.py", diff --git a/backend/tests/test_artifact_architecture.py b/backend/tests/test_artifact_architecture.py index 8e9ba6776..c64a7adef 100644 --- a/backend/tests/test_artifact_architecture.py +++ b/backend/tests/test_artifact_architecture.py @@ -602,6 +602,23 @@ def test_docx_adapter_is_confined_to_the_isolated_worker() -> None: assert adapter_importers == {"modules/artifacts/guide_extraction_worker.py"} +def test_pptx_adapter_is_confined_to_the_isolated_worker() -> None: + adapter_importers: set[str] = set() + for path in APP_ROOT.rglob("*.py"): + relative = path.relative_to(APP_ROOT).as_posix() + for node in ast.walk(_tree(path)): + if isinstance(node, ast.Import) and any( + alias.name == "app.modules.artifacts.guide_pptx" for alias in node.names + ): + adapter_importers.add(relative) + elif ( + isinstance(node, ast.ImportFrom) + and node.module == "app.modules.artifacts.guide_pptx" + ): + adapter_importers.add(relative) + assert adapter_importers == {"modules/artifacts/guide_extraction_worker.py"} + + def test_ooxml_parser_dependency_and_adapter_are_confined_to_the_isolated_worker() -> None: dependency_importers: set[str] = set() adapter_importers: set[str] = set() diff --git a/backend/tests/test_guide_bindings.py b/backend/tests/test_guide_bindings.py index 6477e1dfa..8a1a92d04 100644 --- a/backend/tests/test_guide_bindings.py +++ b/backend/tests/test_guide_bindings.py @@ -183,6 +183,35 @@ def _docx_payload() -> bytes: return output.getvalue() +def _pptx_payload() -> bytes: + output = BytesIO() + presentation = ( + b'' + b'' + ) + relationships = ( + b'' + ) + slide = ( + b'' + b"visible" + b"" + ) + with zipfile.ZipFile(output, "w", zipfile.ZIP_DEFLATED) as archive: + archive.writestr("[Content_Types].xml", b"") + archive.writestr("_rels/.rels", b"") + archive.writestr("ppt/presentation.xml", presentation) + archive.writestr("ppt/_rels/presentation.xml.rels", relationships) + archive.writestr("ppt/slides/slide1.xml", slide) + return output.getvalue() + + def _preparation( tmp_path: Path, **limit_changes: Any ) -> tuple[ArtifactPreparationService, ArtifactScratchManager]: @@ -513,8 +542,23 @@ async def _create_binding(factory, ids: dict[str, UUID]) -> UUID: "field_instructions": False, }, ), + ( + _pptx_payload(), + "application/vnd.openxmlformats-officedocument.presentationml.presentation", + "pptx", + '{"slides":[{"notes":[],"number":1,"text":["visible"]}]}', + { + "truncated": False, + "omitted": True, + "masters": False, + "comments": False, + "hidden_metadata": False, + "non_text_objects": True, + "embedded_objects": False, + }, + ), ], - ids=("json", "docx"), + ids=("json", "docx", "pptx"), ) async def test_extraction_publishes_deterministic_content_and_exact_usage( isolated_database_env: str, @@ -566,7 +610,9 @@ async def test_extraction_publishes_deterministic_content_and_exact_usage( classification_facts={}, ) ) - prepared = await preparation.prepare(_byte_stream(payload), media_type=media_type) + prepared = await preparation.prepare( + _byte_stream(payload), media_type=media_type + ) request = GuideExtractionRequest( project_id=ids["project"], guide_id=ids["guide"], @@ -854,7 +900,7 @@ async def test_successful_replay_requires_the_current_extraction_policy( @pytest.mark.asyncio -@pytest.mark.parametrize("detected_format", ["pdf", "docx"]) +@pytest.mark.parametrize("detected_format", ["pdf", "docx", "pptx"]) async def test_new_format_support_replaces_obsolete_policy_budget_without_replay( isolated_database_env: str, tmp_path: Path, @@ -867,9 +913,12 @@ async def test_new_format_support_replaces_obsolete_policy_budget_without_replay writer.write(stream) payload = stream.getvalue() media_type = "application/pdf" - else: + elif detected_format == "docx": payload = _docx_payload() media_type = "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + else: + payload = _pptx_payload() + media_type = "application/vnd.openxmlformats-officedocument.presentationml.presentation" digest = "sha256:" + hashlib.sha256(payload).hexdigest() engine = create_async_engine(isolated_database_env) factory = async_sessionmaker(engine, expire_on_commit=False) @@ -950,9 +999,7 @@ async def test_new_format_support_replaces_obsolete_policy_budget_without_replay assert budget.policy_version == extraction_policy_version(detected_format) assert budget.claimed_slots == 1 preparation, scratch = _preparation(tmp_path) - prepared = await preparation.prepare( - _byte_stream(payload), media_type=media_type - ) + prepared = await preparation.prepare(_byte_stream(payload), media_type=media_type) service = GuideExtractionService(factory, GuideExtractionRegistry()) extracted = await service.extract_prepared(request, prepared) replay = await service.claim_materialization_slot(request) diff --git a/backend/tests/test_guide_extraction.py b/backend/tests/test_guide_extraction.py index 7e57f8364..f720bb059 100644 --- a/backend/tests/test_guide_extraction.py +++ b/backend/tests/test_guide_extraction.py @@ -245,6 +245,57 @@ def parse(_payload: bytes) -> tuple[str, dict[str, bool]]: } +def test_pptx_worker_orders_limits_validator_adapter_seccomp_then_parsing( + monkeypatch: pytest.MonkeyPatch, +) -> None: + events: list[str] = [] + writes: list[bytes] = [] + omissions = { + "truncated": False, + "omitted": False, + "masters": False, + "comments": False, + "hidden_metadata": False, + "non_text_objects": False, + "embedded_objects": False, + } + monkeypatch.setattr(worker_module, "_install_limits", lambda: events.append("limits")) + + def load_ooxml(): + events.append("validator_import") + return lambda _payload, _detected_format: object() + + def load_pptx(validate_ooxml): + events.append("adapter_import") + assert validate_ooxml(b"PK", "pptx") is not None + + def parse(_payload: bytes) -> tuple[str, dict[str, bool]]: + events.append("parse") + return '{"slides":[]}', omissions + + return parse + + monkeypatch.setattr(worker_module, "_load_ooxml_security", load_ooxml) + monkeypatch.setattr(worker_module, "_load_pptx_extractor", load_pptx) + monkeypatch.setattr(worker_module, "_install_seccomp", lambda: events.append("seccomp")) + monkeypatch.setattr(worker_module.sys, "argv", ["worker", "pptx"]) + monkeypatch.setattr(worker_module.sys, "stdin", SimpleNamespace(buffer=BytesIO(b"PK"))) + monkeypatch.setattr( + worker_module.os, + "write", + lambda _fd, value: (writes.append(bytes(value)), len(value))[1], + ) + + assert worker_module.main() == 0 + assert events == ["limits", "validator_import", "adapter_import", "seccomp", "parse"] + assert json.loads(b"".join(writes)) == { + "status": "extracted", + "error_code": None, + "output": '{"slides":[]}', + "omission_facts": omissions, + } + + @pytest.mark.parametrize( ("detected_format", "payload", "expected"), [ diff --git a/backend/tests/test_guide_pptx.py b/backend/tests/test_guide_pptx.py new file mode 100644 index 000000000..13a299163 --- /dev/null +++ b/backend/tests/test_guide_pptx.py @@ -0,0 +1,581 @@ +"""Focused proofs for deterministic bounded PPTX extraction.""" + +from __future__ import annotations + +from io import BytesIO +import json +import subprocess +import zipfile + +import pytest + +from app.modules.artifacts.guide_extraction import GuideExtractionRunner +from app.modules.artifacts.guide_ooxml import OoxmlSecurityFailure, validate_ooxml +from app.modules.artifacts.guide_pptx import ( + PptxExtraction, + PptxExtractionFailure, + extract_pptx as _extract_pptx, +) + + +_P = "http://schemas.openxmlformats.org/presentationml/2006/main" +_A = "http://schemas.openxmlformats.org/drawingml/2006/main" +_R = "http://schemas.openxmlformats.org/officeDocument/2006/relationships" +_REL = "http://schemas.openxmlformats.org/package/2006/relationships" +_TRANSITIONAL = "http://schemas.openxmlformats.org/officeDocument/2006/relationships" +_STRICT = "http://purl.oclc.org/ooxml/officeDocument/relationships" + + +def _presentation(ids: list[str], *, family: str = _TRANSITIONAL) -> bytes: + slides = "".join(f'' for index, rid in enumerate(ids)) + relationship_namespace = _R if family == _TRANSITIONAL else _STRICT + return ( + f'' + f"{slides}" + ).encode() + + +def _relationships(rows: list[tuple[str, str, str]]) -> bytes: + values = "".join( + f'' + for rid, relationship_type, target in rows + ) + return f'{values}'.encode() + + +def _slide(body: str = "") -> bytes: + return f'{body}'.encode() + + +def _notes(body: str = "") -> bytes: + return f'{body}'.encode() + + +def _text_shape(paragraphs: list[str], *, placeholder: str | None = None, alt: str = "") -> str: + placeholder_xml = f'' if placeholder else "" + content = "".join(f"{paragraph}" for paragraph in paragraphs) + return ( + f'' + f"{placeholder_xml}" + f"{content}" + ) + + +def _package( + slides: list[bytes], + *, + notes: dict[int, bytes] | None = None, + family: str = _TRANSITIONAL, + additions: dict[str, bytes] | None = None, + presentation_name: str = "ppt/presentation.xml", +) -> bytes: + notes = notes or {} + slide_rows = [ + (f"rId{index}", f"{family}/slide", f"slides/slide{index}.xml") + for index in range(1, len(slides) + 1) + ] + members: dict[str, bytes] = { + "[Content_Types].xml": b"", + "_rels/.rels": b"", + presentation_name: _presentation([row[0] for row in slide_rows], family=family), + "ppt/_rels/presentation.xml.rels": _relationships(slide_rows), + } + for index, slide in enumerate(slides, 1): + members[f"ppt/slides/slide{index}.xml"] = slide + if index in notes: + members[f"ppt/notesSlides/notesSlide{index}.xml"] = notes[index] + members[f"ppt/slides/_rels/slide{index}.xml.rels"] = _relationships( + [ + ( + "rIdNotes", + f"{family}/notesSlide", + f"../notesSlides/notesSlide{index}.xml", + ) + ] + ) + members.update(additions or {}) + output = BytesIO() + with zipfile.ZipFile(output, "w", zipfile.ZIP_DEFLATED) as archive: + for name, body in members.items(): + archive.writestr(name, body) + return output.getvalue() + + +def _replace_member(payload: bytes, name: str, body: bytes) -> bytes: + with zipfile.ZipFile(BytesIO(payload)) as source: + members = {info.filename: source.read(info) for info in source.infolist()} + members[name] = body + output = BytesIO() + with zipfile.ZipFile(output, "w", zipfile.ZIP_DEFLATED) as archive: + for member_name, member_body in members.items(): + archive.writestr(member_name, member_body) + return output.getvalue() + + +def extract_pptx( + payload: bytes, + *, + maximum_output_bytes: int = 4 * 1024 * 1024, +) -> PptxExtraction: + def bounded_validate(exact_payload: bytes) -> object: + try: + return validate_ooxml(exact_payload, detected_format="pptx") + except OoxmlSecurityFailure as exc: + raise PptxExtractionFailure(exc.status, exc.code) from exc + + return _extract_pptx( + payload, + validate_ooxml=bounded_validate, + maximum_output_bytes=maximum_output_bytes, + ) + + +def test_extracts_slides_tables_groups_and_notes_in_exact_order() -> None: + slide = _slide( + _text_shape( + [ + "onetwo", + "threefour", + ] + ) + + "grouped" + + "" + + "cell-a" + + "cell-b" + + "" + ) + notes = _notes( + _text_shape(["ignored footer"], placeholder="ftr") + + _text_shape(["speaker note"], placeholder="body") + ) + + result = extract_pptx(_package([slide, _slide()], notes={1: notes})) + + assert result.canonical_output == ( + '{"slides":[{"notes":["speaker note"],"number":1,' + '"text":["one\\ttwo","three\\nfour","grouped","cell-a","","cell-b"]},' + '{"notes":[],"number":2,"text":[]}]}' + ) + assert result.omission_facts == { + "truncated": False, + "omitted": True, + "masters": False, + "comments": False, + "hidden_metadata": True, + "non_text_objects": False, + "embedded_objects": False, + } + assert extract_pptx(_package([slide, _slide()], notes={1: notes})) == result + + +def test_records_every_pptx_omission_without_exposing_metadata() -> None: + slide = _slide( + _text_shape(["visible"], alt=' descr="secret alt"') + + "picture secret" + + "object secret" + + "chart secret" + + "" + ) + payload = _package( + [slide], + additions={ + "ppt/slideMasters/slideMaster1.xml": b"", + "ppt/comments/comment1.xml": b"", + "docProps/core.xml": b"", + }, + ) + + result = extract_pptx(payload) + + assert json.loads(result.canonical_output) == { + "slides": [{"notes": [], "number": 1, "text": ["visible"]}] + } + assert result.omission_facts == { + "truncated": False, + "omitted": True, + "masters": True, + "comments": True, + "hidden_metadata": True, + "non_text_objects": True, + "embedded_objects": True, + } + assert "secret" not in result.canonical_output + + +def test_relationship_conflicts_and_orphans_fail_closed() -> None: + orphan = _package([_slide()], additions={"ppt/slides/slide2.xml": _slide()}) + with pytest.raises(PptxExtractionFailure) as orphaned: + extract_pptx(orphan) + assert (orphaned.value.status, orphaned.value.code) == ( + "malformed", + "pptx_relationship_conflict", + ) + + mixed = _replace_member( + _package([_slide()], notes={1: _notes()}, family=_TRANSITIONAL), + "ppt/slides/_rels/slide1.xml.rels", + _relationships([("rIdNotes", f"{_STRICT}/notesSlide", "../notesSlides/notesSlide1.xml")]), + ) + with pytest.raises(PptxExtractionFailure) as namespace: + extract_pptx(mixed) + assert (namespace.value.status, namespace.value.code) == ( + "malformed", + "pptx_relationship_conflict", + ) + + wrong_type = _replace_member( + _package([_slide()]), + "ppt/_rels/presentation.xml.rels", + _relationships([("rId1", f"{_TRANSITIONAL}/slideLayout", "slides/slide1.xml")]), + ) + with pytest.raises(PptxExtractionFailure) as relationship_type: + extract_pptx(wrong_type) + assert (relationship_type.value.status, relationship_type.value.code) == ( + "malformed", + "pptx_relationship_conflict", + ) + + relationship_root = _replace_member( + _package([_slide()]), + "ppt/_rels/presentation.xml.rels", + ( + b'' + ), + ) + with pytest.raises(PptxExtractionFailure) as invalid_relationship_root: + extract_pptx(relationship_root) + assert (invalid_relationship_root.value.status, invalid_relationship_root.value.code) == ( + "malformed", + "pptx_relationship_conflict", + ) + + duplicate_slide_ids = _replace_member( + _package([_slide(), _slide()]), + "ppt/presentation.xml", + _presentation(["rId1", "rId2"]).replace(b'id="257"', b'id="256"'), + ) + with pytest.raises(PptxExtractionFailure) as duplicate_id: + extract_pptx(duplicate_slide_ids) + assert (duplicate_id.value.status, duplicate_id.value.code) == ( + "malformed", + "pptx_relationship_conflict", + ) + + +def test_complete_relationship_identity_and_ownership_matrix_fails_closed() -> None: + base = _package([_slide()]) + dangling = _replace_member( + base, + "ppt/_rels/presentation.xml.rels", + _relationships([("rId1", f"{_TRANSITIONAL}/slide", "slides/slide2.xml")]), + ) + duplicate_relationship = _replace_member( + base, + "ppt/_rels/presentation.xml.rels", + _relationships( + [ + ("rId1", f"{_TRANSITIONAL}/slide", "slides/slide1.xml"), + ("rId1", f"{_TRANSITIONAL}/slide", "slides/slide1.xml"), + ] + ), + ) + duplicate_reference = _replace_member( + _package([_slide(), _slide()]), + "ppt/presentation.xml", + _presentation(["rId1", "rId1"]), + ) + cross_root = _replace_member( + base, + "ppt/_rels/presentation.xml.rels", + _relationships([("rId1", f"{_TRANSITIONAL}/slide", "../word/document.xml")]), + ) + orphan_notes = _package([_slide()], additions={"ppt/notesSlides/notesSlide1.xml": _notes()}) + shared_notes = _package([_slide(), _slide()], notes={1: _notes(), 2: _notes()}) + shared_notes = _replace_member( + shared_notes, + "ppt/slides/_rels/slide2.xml.rels", + _relationships( + [("rIdNotes", f"{_TRANSITIONAL}/notesSlide", "../notesSlides/notesSlide1.xml")] + ), + ) + + for payload in ( + dangling, + duplicate_relationship, + duplicate_reference, + cross_root, + orphan_notes, + shared_notes, + ): + with pytest.raises(PptxExtractionFailure) as conflict: + extract_pptx(payload) + assert (conflict.value.status, conflict.value.code) == ( + "malformed", + "pptx_relationship_conflict", + ) + + +def test_hidden_slide_visibility_is_recorded_without_discarding_text() -> None: + payload = _package( + [ + _slide(_text_shape(["visible text"])).replace( + b" None: + accepted = extract_pptx(_package([_slide()] * 300)) + assert len(json.loads(accepted.canonical_output)["slides"]) == 300 + + with pytest.raises(PptxExtractionFailure) as excessive: + extract_pptx(_package([_slide()] * 301)) + assert (excessive.value.status, excessive.value.code) == ( + "limit_exceeded", + "pptx_slide_limit", + ) + + +def test_output_and_nesting_limits_are_exact_and_never_partial() -> None: + payload = _package([_slide(_text_shape(["bounded"]))]) + expected = extract_pptx(payload).canonical_output + assert ( + extract_pptx(payload, maximum_output_bytes=len(expected.encode())).canonical_output + == expected + ) + with pytest.raises(PptxExtractionFailure) as output: + extract_pptx(payload, maximum_output_bytes=len(expected.encode()) - 1) + assert (output.value.status, output.value.code) == ("limit_exceeded", "output_limit") + + nested = "deep" + for _ in range(65): + nested = f"{nested}" + with pytest.raises(PptxExtractionFailure) as nesting: + extract_pptx(_package([_slide(_text_shape([nested]))])) + assert (nesting.value.status, nesting.value.code) == ( + "malformed", + "pptx_nesting_limit", + ) + + grouped = "" * 65 + _text_shape(["deep"]) + "" * 65 + with pytest.raises(PptxExtractionFailure) as grouped_nesting: + extract_pptx(_package([_slide(grouped)])) + assert (grouped_nesting.value.status, grouped_nesting.value.code) == ( + "malformed", + "pptx_nesting_limit", + ) + + outer = "" * 55 + inner = "" * 10 + "deep" + "" * 10 + combined = outer + _text_shape([inner]) + "" * 55 + with pytest.raises(PptxExtractionFailure) as combined_nesting: + extract_pptx(_package([_slide(combined)])) + assert (combined_nesting.value.status, combined_nesting.value.code) == ( + "malformed", + "pptx_nesting_limit", + ) + + +def test_only_shape_tree_text_enters_canonical_output() -> None: + slide = ( + f'' + f"{_text_shape(['visible'])}" + "extension secret" + "" + ).encode() + result = extract_pptx(_package([slide])) + assert result.canonical_output == ('{"slides":[{"notes":[],"number":1,"text":["visible"]}]}') + + +def test_skipped_non_text_subtrees_still_record_hidden_metadata() -> None: + slide = _slide( + '' + "not content" + ) + result = extract_pptx(_package([slide])) + assert result.canonical_output == '{"slides":[{"notes":[],"number":1,"text":[]}]}' + assert result.omission_facts["hidden_metadata"] is True + assert result.omission_facts["non_text_objects"] is True + + +def test_graphic_frame_table_discovery_preserves_the_outer_depth_limit() -> None: + nested_table = ( + "" + + "" * 65 + + "deep table" + + "" + + "" * 65 + + "" + ) + with pytest.raises(PptxExtractionFailure) as nesting: + extract_pptx(_package([_slide(nested_table)])) + assert (nesting.value.status, nesting.value.code) == ( + "malformed", + "pptx_nesting_limit", + ) + + +def test_deep_notes_placeholder_cannot_bypass_the_nesting_limit() -> None: + nested_placeholder = ( + '' + + "" * 65 + + '' + + "" * 65 + + "" + ) + with pytest.raises(PptxExtractionFailure) as nesting: + extract_pptx(_package([_slide()], notes={1: _notes(nested_placeholder)})) + assert (nesting.value.status, nesting.value.code) == ( + "malformed", + "pptx_nesting_limit", + ) + + deep_metadata_body = ( + '' + '' + + "" * 65 + + "footer" + + "" * 65 + + "" + ) + with pytest.raises(PptxExtractionFailure) as metadata_nesting: + extract_pptx(_package([_slide()], notes={1: _notes(deep_metadata_body)})) + assert (metadata_nesting.value.status, metadata_nesting.value.code) == ( + "malformed", + "pptx_nesting_limit", + ) + + +@pytest.mark.parametrize( + ("replacement", "code"), + [ + ({"ppt/presentation.xml": b""}, "pptx_invalid_presentation_xml"), + ({"ppt/slides/slide1.xml": b""}, "pptx_invalid_slide_xml"), + ( + {"ppt/notesSlides/notesSlide1.xml": b""}, + "pptx_invalid_notes_xml", + ), + ], +) +def test_invalid_xml_and_roots_have_stable_bounded_codes( + replacement: dict[str, bytes], code: str +) -> None: + payload = _package([_slide()], notes={1: _notes()}, additions=replacement) + with pytest.raises(PptxExtractionFailure) as invalid: + extract_pptx(payload) + assert (invalid.value.status, invalid.value.code) == ("malformed", code) + + +def test_validated_part_names_resolve_case_insensitively() -> None: + payload = _package( + [_slide(_text_shape(["visible"]))], + presentation_name="ppt/Presentation.xml", + ) + result = extract_pptx(payload) + assert result.canonical_output == ('{"slides":[{"notes":[],"number":1,"text":["visible"]}]}') + + +def test_strict_relationship_namespace_is_supported_without_mixing() -> None: + result = extract_pptx( + _package( + [_slide(_text_shape(["strict"]))], + family=_STRICT, + ) + ) + assert result.canonical_output == ('{"slides":[{"notes":[],"number":1,"text":["strict"]}]}') + + +def test_missing_presentation_after_validation_has_stable_bounded_code() -> None: + output = BytesIO() + with zipfile.ZipFile(output, "w", zipfile.ZIP_DEFLATED) as archive: + archive.writestr("[Content_Types].xml", "") + + with pytest.raises(PptxExtractionFailure) as unavailable: + _extract_pptx(output.getvalue(), validate_ooxml=lambda _: object()) + assert (unavailable.value.status, unavailable.value.code) == ( + "malformed", + "pptx_presentation_unavailable", + ) + + with pytest.raises(PptxExtractionFailure) as invalid_container: + _extract_pptx(b"not-a-zip", validate_ooxml=lambda _: object()) + assert (invalid_container.value.status, invalid_container.value.code) == ( + "malformed", + "pptx_presentation_unavailable", + ) + + +def test_real_isolated_runner_uses_pptx_v4_and_complete_omissions(tmp_path) -> None: + payload = _package([_slide("")]) + result = GuideExtractionRunner().extract( + BytesIO(payload), detected_format="pptx", workspace=tmp_path + ) + assert (result.status, result.policy_version) == ("extracted", "guide-extraction-v4") + assert result.canonical_output == '{"slides":[{"notes":[],"number":1,"text":[]}]}' + assert result.omission_facts == { + "truncated": False, + "omitted": True, + "masters": False, + "comments": False, + "hidden_metadata": False, + "non_text_objects": True, + "embedded_objects": False, + } + assert list(tmp_path.iterdir()) == [] + + +@pytest.mark.parametrize( + "omission_facts", + [ + {"truncated": False, "omitted": False}, + { + "truncated": False, + "omitted": False, + "masters": True, + "comments": False, + "hidden_metadata": False, + "non_text_objects": False, + "embedded_objects": False, + }, + ], +) +def test_parent_rejects_incomplete_or_inconsistent_pptx_omission_facts( + tmp_path, monkeypatch: pytest.MonkeyPatch, omission_facts: dict[str, bool] +) -> None: + class CompletedProcess: + returncode = 0 + pid = 123 + + def __init__(self, *_args, **_kwargs) -> None: + pass + + def communicate(self, _payload, timeout): + del timeout + return json.dumps( + { + "status": "extracted", + "error_code": None, + "output": '{"slides":[]}', + "omission_facts": omission_facts, + } + ).encode(), b"" + + monkeypatch.setattr(subprocess, "Popen", CompletedProcess) + result = GuideExtractionRunner().extract( + BytesIO(b"guide"), detected_format="pptx", workspace=tmp_path + ) + assert (result.status, result.error_code) == ( + "parser_failure", + "invalid_executor_output", + ) diff --git a/docs/spec_artifact_storage_service.md b/docs/spec_artifact_storage_service.md index bf5ce46fd..440c8b7a4 100644 --- a/docs/spec_artifact_storage_service.md +++ b/docs/spec_artifact_storage_service.md @@ -1304,14 +1304,33 @@ presence is recorded through the fixed boolean omission keys `truncated`, `omitted`, `headers`, `footers`, `comments`, `tracked_deletions`, `embedded_objects`, `hidden_text`, and `field_instructions`. Successful DOCX extraction always records `truncated=false`; `omitted` is true exactly when any -category boolean is true. Other successful format extractors retain the exact -default `{"truncated":false,"omitted":false}`. Active embedded +category boolean is true. Successful PDF, plain-text, Markdown, JSON, and CSV +extractors retain the exact default `{"truncated":false,"omitted":false}`. +Active embedded content remains a malformed OOXML rejection. The isolated-child result protocol, immutable extracted-content fact, and replay comparison bind those omissions to the same canonical output. Output over 4 MiB fails before any partial result is usable. Traversal beyond 64 nested document/container levels fails deterministically as malformed rather than consuming a transient parser retry. +PPTX extraction runs only after exact `pptx` classification and the same +shared OOXML boundary. Policy `guide-extraction-v4` follows +`ppt/presentation.xml` slide order and resolves only exact matching +Transitional or Strict slide and notes-slide relationships; mixed namespaces, +orphaned or duplicated parts, and cross-root targets fail closed. It emits +compact sorted JSON with one `slides` array whose entries contain the exact +one-based slide `number`, ordered paragraph `text`, and ordered `notes`. +DrawingML run text, tabs, breaks, grouped shapes, and table paragraphs retain +XML order. Non-content notes placeholders and non-text objects never enter the +canonical output. + +Successful PPTX evidence binds the fixed boolean omission keys `truncated`, +`omitted`, `masters`, `comments`, `hidden_metadata`, `non_text_objects`, and +`embedded_objects` to the canonical output. Exactly 300 slides are accepted; +301 fails before semantic extraction. Traversal beyond 64 levels and canonical +output beyond 4 MiB fail without partial agent input. Active embedded content +remains a malformed rejection at the shared OOXML boundary. + For 03B3A the isolation contract is descriptor-only parsing after trusted imports, enforced by a default-deny Linux libseccomp profile with an explicit syscall allowlist plus fixed resource limits: 32 MiB input, 4 MiB