simplify workflow history and add page tracking

[hanna-paasivirta]

Short Description

Supports merging workflow and job code chats OpenFn/lightning#4109

Adds page navigation tracking to both job_chat and workflow_chat services to improve context awareness and auto-refresh documentation when users navigate between different jobs or workflows. When users switch pages (different job, workflow, or adaptor), the conversation history shows page context via prefixes like [pg:job_code/Transform data/language-common], and job_chat automatically refreshes RAG documentation to ensure relevant adaptor docs are retrieved.

Fixes #350

Implementation Details

Page Context Construction

• Services now construct page metadata from the context payload:
  • job_chat: Extracts current page_name and adaptor from context.page_name and context.adaptor, sets type to "job_code"
  • workflow_chat: Extracts current page_name from context.page_name, sets type to "workflow"
• Gracefully handles missing page data - page information is not necessary
• Services return last_page in response meta (populated with current page info). This is then received in the next conversation turn, along with an updated current context.page_name

History Prefixing

• add_page_prefix() helper function prefixes user messages in history with format: [pg:type/name/adaptor] or [pg:type/name]
  • Example: [pg:job_code/Get FHIR data/language-fhir] (job_chat with adaptor)
  • Example: [pg:workflow/fridge-statistics-processing] (workflow_chat)
• Helper handles partial data gracefully - builds prefix from available fields only
• The model receives prefixed content in message history for better context awareness

Simplified History Format

• Both services now store only text responses in history (no nested JSON structures in workflow_chat showing previously edited YAMLs)
• Example assistant message: "I'll add error handling..." instead of {"text": "I'll add error handling...", "yaml": "..."}

Navigation Detection & RAG Auto-refresh

• Added has_navigated() helper to job_chat to detect page changes
• Compares current_page (constructed from context) with last_page (received from meta)
• Navigation detected when type, name, OR adaptor changes
• When navigation is detected, job_chat automatically sets refresh_rag=true to fetch fresh documentation for the new page/adaptor
• Prevents stale documentation when switching between adaptors or jobs

Backward Compatibility

• job_chat always includes meta key with rag data (maintains existing behaviour)
• workflow_chat conditionally includes meta only when page data is present (no breaking change as meta was never returned before)
• Added context field to workflow_chat Payload (previously only in job_chat)
• All page-related fields are optional - services work normally without them

Prompt Changes

• Explain prefixes, attachment redaction and that the user may have navigated between jobs/workflows.

Tests Added

• Basic tests to check conversation flow and that prefixes are added. I struggled to draft a good test to check that the model won't confuse different jobs/workflows and is aware of the context changing.

Anthropic's structured outputs NOT used

• Tried Anthropic's structured outputs beta feature. It did not work as expected (answers keep getting stuck in loops) so removed these. See Use Anthropic structured outputs #310

Summary of changes relevant to Lightning

• Apollo services receive last_page from Lightning via meta key (or None on first turn). meta is already passed unchanged between Lightning and Apollo for job_chat, so no change needed.
• Apollo services receive current_page in the context key.

Requests for Lightning:

• workflow_chat service calls: Add a context key in workflow_chat calls, with a current_page key. This should just be the name of the workflow/step, like 'fetch-fhir-data'.
• job_chat service calls: Similarly, add a current_page key to the context key
• workflow_chat service calls: If Apollo returns a meta key, it should be passed back unchanged in the next conversation turn. (It would be ok to add an empty meta key at the beginning of a conversation too)

AI Usage

Please disclose how you've used AI in this work (it's cool, we just want to know!):

• Code generation (copilot but not intellisense)
• Learning or fact checking
• Strategy / design
• Optimisation / refactoring
• Translation / spellchecking / doc gen
• Other
• I have not used AI

You can read more details in our Responsible AI Policy

[josephjclark]

put page.get(name) into an if as well

We should know the page type because in job chat, the page type is always job chat