Merge pull request #202 from Ontos-AI/feat/wangbinqi/page-citation-source-api#205
Merged
Conversation
…ntic navigation (#200) * feat: implement node assembler for page memory hierarchy to support shared page content and VLM-based node summarization * feat: implement page asset integration into node assembly and chunk connections with support for custom probes * feat: enhance document profiling and page memory integration with skip shard plan option - Added `skip_shard_plan` parameter to `run_lightweight_anatomy` and `profile_document` to allow bypassing LLM shard decisions during profiling. - Updated `memory_service` to utilize `skip_shard_plan` for page memory processing. - Introduced asset annotation functionality for visualizing extracted assets on pages. - Refactored page section mapping and node assembly to support internal section body pages. - Improved environment configuration for Java dependencies in Docker setup. * feat: enhance retrieval and document ingestion with new chunk types and effective parse track handling - Updated `data_type` in `RetrievalQueryRequest` to support new chunk types: page (7) and text+image+table (8). - Refactored document ingestion service to apply effective parse track based on file extension and user-defined settings. - Introduced new validation for parse track handling in oversized PDF processing. - Enhanced tests to cover new chunk types and parse track logic, ensuring robust validation and functionality. * refactor: decouple page memory from image URIs and simplify node metadata structure * feat: integrate summary engine for enhanced document parsing and asset handling - Updated document parser to utilize the new summary engine for images, tables, and text, improving the extraction of titles, summaries, and entities. - Refactored image and table handling to streamline the summarization process, replacing legacy methods with unified calls to the summary engine. - Enhanced markdown parsing to support new entity and asset title fields, ensuring comprehensive data capture during document processing. - Introduced serialization for typed entities to maintain structured data integrity across parsed documents. * feat: implement concurrent page tagging and enrich image and table asset metadata with LLM-extracted entities and titles * feat: standardize page_memory track for PDF/PPTX, implement page-count limits, and refactor section summary publication to initialization. * chore: align parser env defaults * refactor: replace legacy data_type integer with flexible chunk_types set across retrieval services * feat: implement intermediate node chain collapse and add utilities for cross-page table continuity analysis * Add thread-safe trace logging, remove unused heading dataframes, and implement a text-only tagging mode for page memory processing. * refactor: remove budget tracking dependency from page memory services * chore: fix lint and type errors caught by ruff and pyright - Remove unused pandas import in docx/parser.py (F401) - Remove unused collapse_page_ranges import in fine_hierarchy.py (F401) - Rename unused ctx variable to _ctx in debug_page_memory.py (F841) - Remove unused tempfile import in test_page_memory_cross_page_table_merge.py (F401) - Fix greenlet value type: filter None from scope_results list in memory_service.py (reportAssignmentType) - Fix BeautifulSoup Tag narrowing in page_assets.py: use isinstance(x, Tag) guards for find_all/append calls (reportAttributeAccessIssue, reportOptionalMemberAccess) * fix: resolve alembic revision ID conflict causing cycle detection failure The feature branch introduced a migration with revision ID 'a1b2c3d4e5f6' (replace_data_type_with_chunk_types), which collided with an existing main branch migration sharing the same ID (add_qstash_tracking_columns). Alembic detected this as a cycle and blocked all migration tests. Fix: Assigned a unique revision ID 'f0d85d209e68' to the chunk_types migration and rebased its down_revision onto 'f9d0e1f2a3b4' (add_document_metadata_to_documents), which is the current main head. The Alembic graph now has a single clean head with no cycles. * test: fix failing tests from page-memory native hierarchy branch - test_agentic_evidence_renderer_contract.py: Update expected string format to match 'Pages 225-226' instead of 'Pages: 225, 226'. - test_page_memory_cross_page_table_merge.py: Remove unused 'budget' kwarg from merge_cross_page_tables() test calls. - test_doc_profile_anatomy_contract.py: Mock 'PDF_PROFILE_TOC_ENABLED' = True since the feature branch relies on it to test TOC profile attempting logic. * refactor: remove page_memory_work_summary_20260624.md file and streamline page memory processing documentation * feat: enhance skeleton extractor with native hierarchy support and agentic navigation improvements - Add utility functions for page memory processing in _utils.py - Refactor skeleton_extractor with enhanced hierarchy building and cross-page table merge - Improve agentic navigation with structured state management and tool enhancements - Add contract tests for agentic discovery selection - Integrate hierarchy locator improvements for document structure analysis Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove dead commented code and propagate PageLocateConfig to pending TOC fallback - Remove old _estimate_page_offset code block (superseded by _calibrate_offset_via_vlm) - Pass page_memory_config through to _resolve_pending_tocs so its PageLocateResidualAgent fallback uses the same PageLocateConfig as the primary path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: remove PageLocateResidualAgent fallback from skeleton extractor The offset-guided bulk anchoring with binary search handles all breakpoint scenarios. The expensive grep+VLM residual agent fallback is no longer needed. When offset anchoring is unavailable or returns empty, we fall through with calibration_overrides only and let the hierarchy resolver handle the rest via physical hints. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: resolve PR #200 lint and typecheck failures - Remove unused imports flagged by ruff F401 across page_memory and document_agent modules - Fix 6 pyright errors in skeleton_extractor.py: - min() on list[int|None]: use explicit loop with type-narrowed append - arithmetic on TitleNode.printed_page after filter: extract local int variable with explicit None guard Co-authored-by: Cursor <cursoragent@cursor.com> * revert: restore unrelated WIP changes that broke contract tests The previous commit accidentally bundled in-progress refactors (coordinator._run_h1_boundary_pipeline removal and related document_agent changes) together with the CI fixes. These broke 3 contract tests in test_doc_profile_anatomy_contract.py that depend on the h1 boundary pipeline hook. Revert those files to e28f57c so only the lint/typecheck fixes remain. Co-authored-by: Cursor <cursoragent@cursor.com> * revert: remove unconfirmed remove_h1 WIP, keep PR #200 CI-fix only propose_shard_plan.py and skeleton_extractor.py had accidentally picked up in-progress remove_h1_match_plan.md changes (h1_result -> Optional, TOC leaf-based shard cutting). That refactor is not yet confirmed and touches more files/tests than were caught here, so revert both files back to their e28f57c baseline. Net diff vs e28f57c is now limited to the original CI fixes: - memory_service.py: drop 2 dead imports (replace, scope_id_for_pages) - skeleton_extractor.py: 6 pyright type-narrowing guards for printed_page (int | None) arithmetic and min() over Optional lists Verified: make lint, make typecheck, and full apps/worker/tests suite (124 passed) all green with coordinator.py's _run_h1_boundary_pipeline intact. Co-authored-by: Cursor <cursoragent@cursor.com> * feat: prune out-of-scope TOC nodes before offset-guided anchoring When a PDF is truncated (physical pages < TOC page references), leaves with printed_page + offset > page_count are pruned from the tree before anchoring. This eliminates unnecessary bisect/recalibrate cycles and prevents unlocated fallback noise. Also adjusts _verify_offset_tail to prefer non-boundary leaves (printed_page + offset < page_count) for more reliable VLM verification. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Cursor <cursoragent@cursor.com>
…urce-api feat: add page citation source API
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
Deployment Notes
Checklist