feat: Implement extensible chat pipeline architecture with hook system

[robch]

Overview

This PR introduces a comprehensive pipeline architecture to replace the monolithic chat loop in FunctionCallingChat, providing better separation of concerns, extensibility, and testability.

Core Pipeline Components

Pipeline Infrastructure

• ChatPipeline: Main pipeline orchestrator with hook support at 14 different execution points
• ChatContext: Shared state container that flows through the entire pipeline execution
• ChatPipelineFactory: Factory for constructing and configuring pipelines with defaults
• IPipelineStage: Interface enabling composable, testable pipeline stages

Pipeline Stages

1. AIStreamingStage: Handles AI model communication and streaming responses
2. FunctionDetectionStage: Detects and validates function calls in AI responses
3. FunctionExecutionStage: Executes detected functions with error handling
4. LoopDecisionStage: Determines if another pipeline iteration is needed
5. MessagePersistenceStage: Saves messages to chat history

Hook System

• HookPoint enum: Defines 14 hookable points throughout execution lifecycle
• IHookHandler: Interface for implementing custom hooks with context access
• Hook Implementations: Action, Lambda, Simple variants for different use cases
• ChatPipelineExtensions: Fluent API for easy hook configuration

Key Changes

Refactored Code

• FunctionCallingChat.cs: Simplified from complex nested loops to clean pipeline usage
• Removed manual state management in favor of ChatContext
• Better error handling and control flow

Documentation

• HOOK-API-QUICK-REFERENCE.md: Complete hook system API reference
• PIPELINE-HOOK-USAGE-EXAMPLES.md: Practical usage examples and patterns
• IMPLEMENTATION-COMPLETE.md: Implementation notes and decisions
• todo/pipeline-hook-architecture-spec.md: Detailed architecture specification
• todo/hook-architecture-gaps.md: Known limitations and future work

Benefits

? Better Separation of Concerns: Each stage has a single, clear responsibility
? Enhanced Testability: Stages can be tested independently with mock contexts
? Extensibility: Add custom behavior via hooks without modifying core code
? Clearer Control Flow: Pipeline execution is explicit and easy to follow
? Foundation for Features: Enables logging, metrics, debugging, and more
? Maintainability: Smaller, focused components are easier to understand and modify

Hook Points Available

The pipeline exposes 14 hook points for customization:

• BeforePipeline / AfterPipeline
• BeforeAIStreaming / AfterAIStreaming
• BeforeFunctionDetection / AfterFunctionDetection
• BeforeFunctionExecution / AfterFunctionExecution / OnFunctionExecutionError
• BeforeLoopDecision / AfterLoopDecision
• BeforeMessagePersistence / AfterMessagePersistence
• OnPipelineError

Testing Status

• ? Build succeeds with only benign warnings (async methods without await)
• ? Version command works correctly
• ? All components compile successfully
• ?? Comprehensive integration testing needed
• ?? Unit tests for individual stages needed

Next Steps

1. Add comprehensive unit tests for each stage
2. Add integration tests for full pipeline execution
3. Test with various hook configurations
4. Performance testing to ensure no regression
5. Consider adding metrics/logging hooks as examples

Files Changed

24 files changed, 6,194 insertions(+), 137 deletions(-)

New Files:

• Pipeline infrastructure (8 files)
• Pipeline stages (5 files)
• Hook handlers (4 files)
• Documentation (5 files)

Modified Files:

• FunctionCallingChat.cs (simplified significantly)

---

Note: This is marked as a draft PR as it needs comprehensive testing before merging.