[Draft] CU + Foundry IQ extensions (additive refactor of PR #1)#3
Draft
yungshinlintw wants to merge 59 commits into
Draft
[Draft] CU + Foundry IQ extensions (additive refactor of PR #1)#3yungshinlintw wants to merge 59 commits into
yungshinlintw wants to merge 59 commits into
Conversation
When TOOLBOX_MCP_URL is empty, the agent falls back to local-direct mode: - Connects to inventory MCP server at INVENTORY_MCP_URL (default localhost:8001) - Wraps work-orders REST API at WORK_ORDERS_API_URL (default localhost:8002) as FunctionTools (list, get, create, update) - KB search fallback remains unchanged This allows running the full agent locally without any cloud Toolbox dependency. Set TOOLBOX_MCP_URL to switch back to Toolbox mode. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tion - Add '+' button to ChatInput for PDF/image file uploads (OpenAI-style) - File attachment is conditional: only shown when AZURE_CONTENTUNDERSTANDING_ENDPOINT is set - Gateway exposes GET /api/features for UI feature flags - Agent integrates ContentUnderstandingContextProvider from agent-framework - Files sent as base64, converted to Content.from_data() for CU analysis - System prompt updated: agent auto-detects work orders from CU output, presents extracted fields, asks for confirmation, then creates via API - MessageBubble shows image thumbnails and PDF chips for user messages - README and .env.example updated with new AZURE_CONTENTUNDERSTANDING_ENDPOINT var - Added agent-framework-azure-contentunderstanding to pyproject.toml Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace inline 'import json as _json' with top-level json import to prevent 'cannot access local variable' error during CU file processing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Clarify that WO IDs in uploaded documents are paper form references, not existing system records - Instruct agent to immediately create WO on user confirmation without re-asking for details - Add 'save' and 'commit' as confirmation triggers - Tell agent to remember extracted fields across conversation turns Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Previously defaulted to 5s timeout which deferred analysis to background, requiring a follow-up message to get results. Now waits until CU finishes so the first response includes the full extraction. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Move demo assets from top-level demo_files/ into content_understanding/demo_files/ to co-locate them with the CU tooling. Remove old WO-618 files (replaced by anonymized versions). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add expected JSON output files for fiber-splice-restoration.pdf and scanned_work_order.png. Both use fictitious data (fake location, J. Martinez as technician). These serve as ground truth for validating the custom cu_demo_work_order analyzer. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add create_work_order_analyzer.py which creates/recreates the cu_demo_work_order analyzer in Azure Content Understanding. - Field schema aligned to WorkOrderCreate API (title, description, status, priority, assigned_technician, location, due_date, parts_needed) - assigned_technician explicitly targets the sign-off table to avoid confusing it with the 'Field Technical Contact' in the header - Loads .env from repo root — no duplicate config needed - Supports --analyze FILE and --analyze-only FILE flags Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Update ChatInput accept attribute to include all CU-supported media types: PDF, Office docs, plain text, images, audio, and video formats. Also update the file chip icon to show videocam/audiotrack/picture_as_pdf based on the attached file's MIME type. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- create_agent(cu_mode) selects CU analyzer based on mode:
'none' → no CU provider
'basic' → prebuilt-layout
'work_order' → cu_demo_work_order custom analyzer
- Filenames namespaced as '{filename}:{cu_mode}' so the same file can be
re-analyzed under a different mode without hitting the session duplicate check
- /api/chat reads cu_mode from request body and passes through
- /api/features returns cu_modes list alongside content_understanding flag
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add a CU Context Provider section above Activity in the sidebar, visible only when AZURE_CONTENTUNDERSTANDING_ENDPOINT is configured. States: - None — no CU, attachment button hidden - Basic CU — prebuilt-layout general document extraction - Classify & Analyze Work Order — cu_demo_work_order custom analyzer Selected mode is sent with every /api/chat request. Header style matches the Activity section (same font, icon, border treatment, no all-caps). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add splicing-safety-cert.pdf — a single-page training certificate from the Pacific Northwest Fiber Training Institute, intentionally non-work-order in structure. Used to demonstrate the 'Classify & Analyze Work Order' CU mode against an unrelated document type. The custom analyzer partially extracts (course name as title, org address as location) but returns null for assigned_technician and empty parts_needed, making it a good demo contrast against the actual work order PDF. Also add gen_training_cert_pdf.py (separate from the WO script) to regenerate it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add create_classify_and_analyze.py: creates cu_demo_classify_and_analyze classifier that routes work_order → cu_demo_work_order (field extraction) and other → prebuilt-layout (markdown fallback) - Prerequisite check: exits with guidance if cu_demo_work_order not found - Update agent.py: work_order mode now uses cu_demo_classify_and_analyze instead of cu_demo_work_order directly - Add content_understanding/README.md: documents two-step setup order, demo files, and UI mode descriptions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Set return_details=False on both cu_demo_work_order and
cu_demo_classify_and_analyze to reduce payload size
- Add content_understanding/tests/ with live pytest suite:
- test_work_order_analyzer.py: compares extracted fields against
expected JSONs in demo_files/ (status, priority, technician,
parts_needed, due_date exact match; title/schema presence checks)
- test_classifier.py: verifies classification routing for all 3
demo files (WO PDF → work_order, scanned PNG → work_order,
training cert → other)
- conftest.py: shared client fixture, helpers, file paths
- Add pytest dev dependency
- All 16 tests pass
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ntegration Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Rename demo_files to use doc-type prefix for demo clarity:
fiber-splice-restoration.{pdf,json} → work_order_fiber_splice.{pdf,json}
scanned_work_order.{png,json} → work_order_scanned.{png,json}
splicing-safety-cert.pdf → safety_cert_splicing.pdf
- Update all references in tools, tests, scripts, README
- Add test_basic_cu_mode.py covering two missing demo scenarios:
BasicCuMode: work order PDF/PNG → markdown, no structured fields
WrongAnalyzerOnCert: training cert through WO analyzer → partial/
mismatched extraction (intentional demo contrast)
- Add get_markdown_from_result helper to conftest
- All 24 tests pass
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- UI: show + attachment button regardless of CU mode selection (previously hidden when mode was 'none') - Agent: when cu_mode='none', pass file bytes directly to OpenAI without CU processing — OpenAI will reject unsupported types like .docx, demonstrating why CU matters - Add scripts/gen_work_order_docx.py and *.docx to .gitignore (local-only demo files) Demo flow with work_order_fiber_splice.docx: None (OpenAI only) → OpenAI rejects .docx Basic CU → CU extracts markdown, agent reads it Classify & Analyze → classified as work_order, fields extracted Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…e mode When cu_mode='none' and an unsupported file type (e.g. .docx) is attached, immediately yield an error message explaining: - OpenAI only supports images and PDF natively - Which file(s) are unsupported - How to fix it: switch to Basic CU or Classify & Analyze Previously the agent silently ignored the file and asked for a work order ID, which was confusing during the demo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…rted Instead of blocking early, we now: 1. Yield a 'warning' SSE event explaining OpenAI cannot read the file type 2. Still run the agent — OpenAI's confused response reinforces the limitation The warning renders as an amber callout banner above the agent response, making the demo contrast clear: banner + confused agent response = compelling reason to switch to CU mode. Stack: agent.py (yield warning) → api_server.py (SSE forward) → client.ts (onWarning callback) → useChat.ts (store on message) → MessageBubble.tsx (amber banner) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace raw Python exception traceback with a readable delta message. Catches the 400 BadRequestError from the agent stream, extracts the human-readable OpenAI 'message' field, and yields: '❌ OpenAI rejected the request: Missing required parameter...' + hint to switch to CU mode The amber warning banner + clean error message together make the demo contrast clear and presentable. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add WORK_ORDER_DOCX constant + .docx MIME type to conftest - TestWorkOrderDocx: tests core fields (status, priority, due_date, description, parts_needed all pass); xfail for title/assigned_technician/ location — known format limitation since analyzer was trained on PDF/image - TestClassification: xfail for docx routing — classifier trained on PDF/image so docx routes to 'other'; documents expected behavior for the demo All 30 tests pass (4 xfailed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add 'Work Order Display Format' section: - Full API field → friendly name mapping table (id, title, description, status, priority, assigned_technician, location, due_date, parts_needed, created_at, updated_at) - Defined 'standard work order card' template with emoji status/priority - created_at/updated_at hidden unless requested (in <details> block) - Applied same friendly names to document extraction table - Use '—' for fields not found in uploaded document Ensures consistent, user-friendly display for both API-fetched work orders and CU-extracted document fields. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- PDF/docx: blank sign-off (OPEN work order), add Dispatch Information block with J. Martinez as assigned tech (buried as routing metadata) - Field Technical Contact header shows Marcus Tran (site contact, not technician) — deliberately ambiguous for Basic CU + LLM demo - Update cu_demo_work_order analyzer: assigned_technician now points to Dispatch Information block, not sign-off table - Update content_understanding/README.md with full step-by-step demo walkthrough: None → Basic CU → Classify & Analyze, with explanation of why Basic CU returns the wrong technician - Add bonus demo sections: safety cert classification, handwritten PNG - Add Content Understanding Demo section to root README with quick demo sequence and setup instructions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove docx from gitignore so the demo file is committed with the repo. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… wrong result Replace 'Assigned Tech: J. Martinez' label with a routing audit entry: 'Route → J. Martinez | Status: Pending Accept' Without the explicit 'Assigned Tech:' label the LLM reading flat Basic CU markdown sees 'Field Technical Contact: Marcus Tran' as the most prominent named contact and returns Marcus Tran (wrong). The custom analyzer description now points to the name after 'Route →', so it still returns J. Martinez (correct). - gen_work_order_pdf.py: dispatch_table uses 'Dispatch Log' + 'Route → <name>' - gen_work_order_docx.py: same structure, same text - create_work_order_analyzer.py: assigned_technician desc updated to match - content_understanding/README.md: explanation updated - Regenerated PDF + docx demo files - All 30 tests pass, 4 xfailed (unchanged) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Demo ambiguity design: - Field Technical Contact header: John Smith (Network Operations Supervisor) → LLM returns John Smith as technician (WRONG) in Basic CU mode - Sign-Off table Print Name: J. Martinez (pre-filled, no 'Assigned' label) - Dispatch Log: Route → J. Martinez (what custom analyzer reads) → Custom analyzer returns J. Martinez (CORRECT) Changes: - site_contact renamed to John Smith / Network Operations Supervisor - SIGN_FIELDS: Print Name = J. Martinez, Signature stays blank - sign_table(): separate signature vs print_name columns - docx: same structure — John Smith header, J. Martinez in sign-off Print Name - README: updated explanation with John Smith as the decoy name - All 30 tests pass, 4 xfailed (unchanged) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add top-priority 'Document Upload' rule that short-circuits ALL skill classification when a file attachment is present. The agent must: - Never load a skill - Never call get_work_order / list_work_orders / field-briefing - Read the CU-extracted content already in context - Present results in the Work Order Extraction Table format Also add an explicit guard row at the top of the skill mapping table. Tighten the Document Upload section opening to reinforce no-API rule. Fixes: field-briefing skill being invoked when docx is uploaded with a message like 'Get work order detail'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Having J. Martinez in both the sign-off Print Name AND Dispatch Log made it too easy for the LLM to find. Now J. Martinez appears ONLY as 'Route → J. Martinez' in the Dispatch Log row, which reads as routing metadata. The LLM sees John Smith under 'Field Technical Contact' and returns that instead (wrong). Custom analyzer still finds J. Martinez via the explicit 'Route →' description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- WorkOrder/WorkOrderCreate/WorkOrderUpdate API models now include site_technician (str | None) - CU analyzer schema adds site_technician field with description explaining it is optional - assigned_technician description updated to fall back for handwritten docs without Dispatch Log - PDF and docx header updated: 2×4 table showing both Assigned Field Technician (John Smith, misdirection) and On-Site Technician (—, empty) side by side - Expected JSONs updated with site_technician: null - Tests: site_technician added to WO_FIELDS; test_site_technician_is_empty added to all 3 test classes Demo verification confirmed: - Basic CU LLM returns 'John Smith' for assigned_technician (WRONG — misdirection works) - Custom analyzer returns 'J. Martinez' for assigned_technician, site_technician: null (CORRECT) - 33 passed, 4 xfailed (docx format limitations, expected) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Explains the misdirection using the new explicit header label and the Dispatch Log routing entry format. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
tests/test_foundry_iq_demo.py — 18 integration tests (pytest):
TestAssets: verify env vars, indexer success, index doc count, gateway flag
TestPartsInventoryFIB009: minimal plain-text vs standard HTML <td> cells
TestOTDRFiber03: key demo test — minimal F-03 row has 3 numerics (ORL@1310
blank cell collapsed, 46.1 misattributed); standard has explicit <td></td>
empty cell confirming ORL@1310 was not recorded
Run: cd services/foundry-iq-docs && pytest tests/ -v
content-understanding/FOUNDRY_IQ_SETUP.md — step-by-step setup guide:
prerequisites, script usage, manual verification steps, .env values,
demo questions, troubleshooting (indexer status, API key auth, CU pricing)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…top=5 PDF changes (otdr-acceptance-results.pdf): - Table headers: remove colored background (BRAND_BLUE/HEADER_GREEN) → white - Table headers: text color changed to black (was white on dark bg, unreadable) - Removed GRID lines from OTDR measurements and Splice Loss tables (grid-free tables make minimal parser column alignment harder, strengthening the demo contrast between modes) Test fix (test_standard_index_f03_loss_values_correct): - Search with top=5 and combine all snippets; F-03 measurement row may land in a later chunk than the Job Summary table Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Primary demo question is now: 'What is the ORL reading at 1310nm for fiber F-03?' - Minimal: returns ~46.1 (wrong — the ORL@1550 value shifted left) - Standard: correctly reports ORL@1310 as blank/not recorded Secondary demo question (FIB-009 parts inventory) unchanged. Removes outdated F-04 insertion loss question. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ing/ - Move setup-foundry-iq-cu-demo.sh → services/foundry-iq-docs/content-understanding/scripts/ - Move test_foundry_iq_demo.py → services/foundry-iq-docs/content-understanding/tests/ - Move demo PDFs → services/foundry-iq-docs/content-understanding/docs/ - Add --teardown/--recreate flags and container creation sleep fix to setup script - Update test to use DefaultAzureCredential / az-login instead of requiring AZURE_SEARCH_ADMIN_KEY - Update agent.py: remove AZURE_SEARCH_ADMIN_KEY dep; Search auth falls back to DefaultAzureCredential - Fix _REPO_ROOT parents depth (3→4) in test file; fix minimal OTDR tests to search top=5 - Update .env.example with AZURE_RESOURCE_GROUP, FOUNDRY_RESOURCE_GROUP, FOUNDRY_ACCOUNT_NAME All 17 integration tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The previous approach used httpx.Auth (injected via auth_flow) which is only called during call_tool(), not during the initial MCP connect/initialize handshake. This caused a 403 on connect, surfaced as a ConnectError. Fix: bake the api-key (or bearer token) into AsyncClient's default_headers so every request — including the MCP initialization — is authenticated. Also fall back to AZURE_SEARCH_ADMIN_KEY when AZURE_SEARCH_API_KEY is unset, and update the FOUNDRY_IQ_SETUP.md demo prompt to include 'Check the KB'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Inventory data belongs in the live Inventory MCP API, not a stale static PDF in the knowledge base. Removes the file, the secondary demo question section from FOUNDRY_IQ_SETUP.md, and the TestPartsInventoryFIB009 test class. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Align UI text with official MS Learn contentExtractionMode definitions: - Rename section to 'KB Extraction Mode' (reflects the actual API param) - Minimal: 'Plain text only · no layout analysis' (was vague 'standard text extraction') - Standard label: 'Standard · Azure CU' (cleaner formatting) - Standard desc: 'Full pipeline · Azure Content Understanding' (SDK definition) - Standard note: 'HTML table output preserves column structure' (explains the real CU benefit) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ompt - Rename content_understanding/ -> content-understanding/ to match project hyphen convention (services/foundry-iq-docs/content-understanding/) - Update 14 file refs (README, docstrings, scripts, conftest, gitignore) - Remove TestScannedWorkOrderPng.test_assigned_technician_matches_expected (handwritten OCR is non-deterministic on this fixture) - Replace hardcoded 'J. Martinez'/'Dispatch Log'/'Route ->' example in cu_demo_work_order analyzer prompt with generic dispatch/assignment instructions - Add *.local.md and .env.bak to .gitignore (local-only notes) Note: cu_demo_work_order analyzer will need recreation via create_work_order_analyzer.py --recreate to pick up new prompt.
- Add AGENT_MODE=local-direct as explicit opt-in for Toolbox-bypass mode. When TOOLBOX_MCP_URL is unset the previous silent fallback still works (back-compat), but the new env var makes intent unambiguous. - Introduce env-gated CU flags: _CU_ENABLED (driven by AZURE_CONTENTUNDERSTANDING_ENDPOINT) and _CU_FOUNDRY_IQ_ENABLED (driven by FOUNDRY_IQ_MINIMAL_MCP_URL + FOUNDRY_IQ_STANDARD_MCP_URL). Requests for foundry_iq_mode with missing config now warn instead of silently trying to create an unusable MCP tool. - Split system prompt: move document-upload + work-order-extraction sections into system_prompt_cu.md, appended only when cu_mode is basic|work_order AND CU endpoint is configured. With CU disabled the agent receives the same base prompt as main. - Gate _LoggingCUWrapper verbose output behind CU_VERBOSE_LOGGING=1. Default behavior drops info-level CU traces to debug to keep prod logs clean; errors still surface as warnings. - Update module docstring to describe both modes and the additive CU extension surface.
Six tests covering: - module defaults match main (LOCAL_DIRECT=False, CU_ENABLED=False, CU_FOUNDRY_IQ_ENABLED=False) - base system prompt has no CU sections - CU prompt appended only when cu_active=True - AGENT_MODE=local-direct opt-in - _CU_ENABLED driven by AZURE_CONTENTUNDERSTANDING_ENDPOINT - _CU_FOUNDRY_IQ_ENABLED requires both IQ URLs
- README env-var table: add AGENT_MODE=local-direct, INVENTORY_MCP_URL, WORK_ORDERS_API_URL, FOUNDRY_IQ_MINIMAL_MCP_URL, FOUNDRY_IQ_STANDARD_MCP_URL, CU_VERBOSE_LOGGING. Mark each optional CU/IQ var as 'additive'. - New docs/integration.md describing the compatibility matrix, mode-vs-flag separation, and the open Toolbox session-id bug that blocks the long-term Option-2 cleanup.
yungshinlintw
commented
Jun 1, 2026
yungshinlintw
commented
Jun 1, 2026
yungshinlintw
commented
Jun 1, 2026
yungshinlintw
commented
Jun 2, 2026
yungshinlintw
commented
Jun 2, 2026
yungshinlintw
commented
Jun 2, 2026
yungshinlintw
commented
Jun 2, 2026
# Conflicts: # README.md # src/fibey/agent/agent.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft — keeps PR #1 (yslin/cu_update) open for comparison.
Refactor of #1 with the same feature set, restructured to be strictly additive on top of
main. With all CU/IQ env vars unset, the agent behaves identically tomain— same prompt, same tools, same logs.What changed vs PR #1
content_understanding/→content-understanding/(matches project hyphen convention); remove flaky handwritten-OCR test; generalize thecu_demo_work_orderanalyzer prompt (no more hardcodedJ. Martinez/Route →/Dispatch Logtext)--recreateto pick up the new promptAGENT_MODE=local-directopt-in; env-gated_CU_ENABLED/_CU_FOUNDRY_IQ_ENABLED; system prompt split (system_prompt_cu.mdonly appended when CU is active);CU_VERBOSE_LOGGINGto silence[CU]info-level traces by defaulttests/test_agent_additive.pyasserting defaults matchmain, prompt-append gating, and env-driven flag wiringdocs/integration.mddescribing compatibility matrix and the open Toolbox session-id bug that blocks Option-2 long-term cleanupOut of scope for this PR
Mcp-Session-Idin its initialize response, so the Python SDK drops the session andtools/listfails withSession terminated. Verified against protocolVersion2025-06-18,2025-03-26, and2024-11-05. Once fixed we'll register IQ KBs upstream and drop_create_foundry_iq_mcpin a follow-up PR.agent.pyintocontent_understanding.py+foundry_iq.py) — deferred to keep this PR focused.Test plan
Compatibility matrix
See docs/integration.md.