feat: position-aware spatial constraints for VLM title extraction in page memory#212
Open
EricNGOntos wants to merge 36 commits into
Open
feat: position-aware spatial constraints for VLM title extraction in page memory#212EricNGOntos wants to merge 36 commits into
EricNGOntos wants to merge 36 commits into
Conversation
…hared page content and VLM-based node summarization
…onnections with support for custom probes
…p shard plan option - Added `skip_shard_plan` parameter to `run_lightweight_anatomy` and `profile_document` to allow bypassing LLM shard decisions during profiling. - Updated `memory_service` to utilize `skip_shard_plan` for page memory processing. - Introduced asset annotation functionality for visualizing extracted assets on pages. - Refactored page section mapping and node assembly to support internal section body pages. - Improved environment configuration for Java dependencies in Docker setup.
…nd effective parse track handling - Updated `data_type` in `RetrievalQueryRequest` to support new chunk types: page (7) and text+image+table (8). - Refactored document ingestion service to apply effective parse track based on file extension and user-defined settings. - Introduced new validation for parse track handling in oversized PDF processing. - Enhanced tests to cover new chunk types and parse track logic, ensuring robust validation and functionality.
…t handling - Updated document parser to utilize the new summary engine for images, tables, and text, improving the extraction of titles, summaries, and entities. - Refactored image and table handling to streamline the summarization process, replacing legacy methods with unified calls to the summary engine. - Enhanced markdown parsing to support new entity and asset title fields, ensuring comprehensive data capture during document processing. - Introduced serialization for typed entities to maintain structured data integrity across parsed documents.
…set metadata with LLM-extracted entities and titles
…t limits, and refactor section summary publication to initialization.
…set across retrieval services
…r cross-page table continuity analysis
…implement a text-only tagging mode for page memory processing.
- Remove unused pandas import in docx/parser.py (F401) - Remove unused collapse_page_ranges import in fine_hierarchy.py (F401) - Rename unused ctx variable to _ctx in debug_page_memory.py (F841) - Remove unused tempfile import in test_page_memory_cross_page_table_merge.py (F401) - Fix greenlet value type: filter None from scope_results list in memory_service.py (reportAssignmentType) - Fix BeautifulSoup Tag narrowing in page_assets.py: use isinstance(x, Tag) guards for find_all/append calls (reportAttributeAccessIssue, reportOptionalMemberAccess)
…lure The feature branch introduced a migration with revision ID 'a1b2c3d4e5f6' (replace_data_type_with_chunk_types), which collided with an existing main branch migration sharing the same ID (add_qstash_tracking_columns). Alembic detected this as a cycle and blocked all migration tests. Fix: Assigned a unique revision ID 'f0d85d209e68' to the chunk_types migration and rebased its down_revision onto 'f9d0e1f2a3b4' (add_document_metadata_to_documents), which is the current main head. The Alembic graph now has a single clean head with no cycles.
- test_agentic_evidence_renderer_contract.py: Update expected string format to match 'Pages 225-226' instead of 'Pages: 225, 226'. - test_page_memory_cross_page_table_merge.py: Remove unused 'budget' kwarg from merge_cross_page_tables() test calls. - test_doc_profile_anatomy_contract.py: Mock 'PDF_PROFILE_TOC_ENABLED' = True since the feature branch relies on it to test TOC profile attempting logic.
…line page memory processing documentation
…e-hierarchy' into feat/wuchengke/page-memory-native-hierarchy # Conflicts: # packages/shared-python/shared/services/retrieval/hydration/result_assembly.py # packages/shared-python/shared/services/retrieval/search/lexical_text.py
…entic navigation improvements - Add utility functions for page memory processing in _utils.py - Refactor skeleton_extractor with enhanced hierarchy building and cross-page table merge - Improve agentic navigation with structured state management and tool enhancements - Add contract tests for agentic discovery selection - Integrate hierarchy locator improvements for document structure analysis Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…emory-native-hierarchy # Conflicts: # apps/worker/app/services/page_memory/skeleton_extractor.py
…ding TOC fallback - Remove old _estimate_page_offset code block (superseded by _calibrate_offset_via_vlm) - Pass page_memory_config through to _resolve_pending_tocs so its PageLocateResidualAgent fallback uses the same PageLocateConfig as the primary path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ctor The offset-guided bulk anchoring with binary search handles all breakpoint scenarios. The expensive grep+VLM residual agent fallback is no longer needed. When offset anchoring is unavailable or returns empty, we fall through with calibration_overrides only and let the hierarchy resolver handle the rest via physical hints. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unused imports flagged by ruff F401 across page_memory and
document_agent modules
- Fix 6 pyright errors in skeleton_extractor.py:
- min() on list[int|None]: use explicit loop with type-narrowed append
- arithmetic on TitleNode.printed_page after filter: extract local
int variable with explicit None guard
Co-authored-by: Cursor <cursoragent@cursor.com>
- Introduced toc_page_offset to both PageAnatomyMap and AgentBlackboard to facilitate better handling of table of contents page offsets. - Updated relevant functions to utilize the new toc_page_offset for improved shard planning and TOC processing. - Removed the match_h1_pages tool as it was deemed unnecessary for current functionality. Co-authored-by: Cursor <cursoragent@cursor.com>
When a PDF is truncated (physical pages < TOC page references), leaves with printed_page + offset > page_count are pruned from the tree before anchoring. This eliminates unnecessary bisect/recalibrate cycles and prevents unlocated fallback noise. Also adjusts _verify_offset_tail to prefer non-boundary leaves (printed_page + offset < page_count) for more reliable VLM verification. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…type Add derive_chapter_boundaries() for L1/L2-aware shard planning with physical page ranges. Extend Shard.anchor_type to accept "toc_chapter_boundary" alongside existing types. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…-h1-match # Conflicts: # apps/worker/app/services/page_memory/_utils.py # apps/worker/app/services/page_memory/skeleton_extractor.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Default retrieval now uses the classic 3-channel BM25 + RRF top-K path. Only explicit use_agentic=True triggers the full agentic workflow (LLM doc-select + navigation). Removes dead RETRIEVAL_AGENTIC_ENABLED env var reference from contract test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test previously asserted that use_agentic=False was ignored and still routed to the workflow. Now that the switch is functional, the test verifies that use_agentic=False correctly routes to classic_topk. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
RETRIEVAL_DECOMPOSITION_ENABLED and RETRIEVAL_AGENTIC_INLINE_TABLE_CHAR_LIMIT are defined in AIConfig but never accessed at runtime. Remove them to reduce confusion. Agentic vs classic routing is now controlled per-request via use_agentic. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…tion for VLM title extraction
…y definitions in prompt service
…-h1-match # Conflicts: # apps/worker/app/services/page_memory/memory_service.py # apps/worker/app/services/page_memory/page_tagger.py
Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds position-aware spatial constraints and boundary detection to the VLM-based title extraction pipeline in page memory, and refactors the associated prompt instructions for clarity.
Changes
packages/shared-python/shared/services/ai/prompt_service.pypage-memory-vlm-titleprompt builder to accept spatial context parameters:coarse_title,is_first_page,is_last_page,next_sibling_title, andscan_direction.scan_direction(left-to-right vs. right-to-left).spatial_preambleblock injected into the VLM prompt when context is available, giving the model boundary hints (e.g. "AFTER X", "BEFORE Y") to suppress out-of-scope headings.packages/shared-python/shared/models/schemas/page_memory_config.pyscan_directionfield (default:top_to_bottom_left_to_right) toPageMemoryConfigfor configurable reading order.apps/worker/app/services/page_memory/page_tagger.pytag_page_titles()with two new optional parameters:coarse_skeletonsandscan_direction._build_page_title_context_map()to compute per-pagePageTitleContext(first/last page flags, sibling titles, direction) from the coarse skeleton list._title_exclusive_end()helper to determine the last page owned by a skeleton before the next sibling begins._tag_vlm_titles()now acceptstitle_ctx: PageTitleContext | Noneand injects it into the promptparasdict.title_ctxthrough the_detect_oneclosure.apps/worker/app/services/page_memory/memory_service.pytag_page_titles()call site to pass bothcoarse_skeletons,scan_direction(from this PR) andmax_concurrent(from PR Add page-memory VLM capacity gate #208).Why
Fat-leaf pages spanning multiple headings frequently caused the VLM to surface headings from adjacent sections. By anchoring each page to its coarse-skeleton boundary (first/last page flags, next sibling title), the prompt can now tell the model "only report headings that appear BEFORE
next_sibling_title", dramatically reducing cross-section noise.Testing
make checkpasses (lint + typecheck: 0 errors).max_concurrentand spatial context params are preserved and work together.