feature: Introducing AO-Hybrid Orchestrator

[anoopchandra538]

Description

This PR introduces the Assistant Orchestrator Hybrid, a new orchestrator designed to address scalability limitations of the existing assistant-orchestrator when managing large numbers of agents. The hybrid approach combines semantic search (pgvector embeddings) with BM25 lexical matching to provide accurate and efficient agent selection at scale.

Problem Statement

The original assistant-orchestrator relies on LLM-based agent selection, which becomes increasingly expensive and slow as the number of agents grows. This hybrid approach pre-filters candidates using fast retrieval methods before LLM reranking, significantly improving both performance and cost efficiency.

Key Features

Hybrid Search for Agent Selection

• BM25 Lexical Matching: Fast keyword-based search using the rank_bm25 library for efficient initial retrieval
• Semantic Search via pgvector: PostgreSQL-based vector similarity search using Azure OpenAI embeddings (1536 dimensions)
• Configurable Weighting: Tunable BM25/semantic weights (default: 25%/75%) via environment variables
• LLM-based Reranking: Final agent selection using GPT-4o with semantic scores as primary signal

Concurrent Execution & Response Synthesis

• Parallel Agent Execution: Execute multiple agents concurrently using asyncio with ThreadPoolExecutor
• Response Aggregation: LLM-powered synthesis of multi-agent responses into coherent answers
• Configurable Parallelism: Environment-based controls for max parallel agents and execution timeouts
• Primary/Secondary Agent Selection: Intelligent selection of complementary agents for complex queries

Agent Registry with PostgreSQL

• SQLAlchemy ORM Model: AgentRegistry table with metadata, tags, keywords, and vector embeddings
• Auto-generated Embeddings: Description embeddings generated on registration via Azure OpenAI
• TF-IDF Keyword Extraction: Automatic keyword extraction for improved BM25 matching
• CRUD Operations: Full agent lifecycle management through AgentRegistryManager

Background Content Extraction (Celery)

• Celery Task Queue: Asynchronous metadata updates using Redis-backed Celery workers
• TF-IDF Learning Service: Learns new keywords from agent responses to improve future matching
• Lazy PostgreSQL Initialization: Efficient worker process management with skip_embeddings mode

Changes

New Components

• hybrid_search_service.py - Core hybrid search combining BM25 + semantic + LLM reranking
• agent_orchestration_service.py - Parallel execution and response synthesis
• agent_registry_manager.py - Agent registry CRUD operations
• postgres_client.py - PostgreSQL client with pgvector support
• integration/openai_client.py - Azure OpenAI client for embeddings and chat
• agent_registry.py - SQLAlchemy ORM model for agent storage
• model/search.py - Search result dataclasses
• content_update/ - Celery tasks and TF-IDF learning service
• prompts/ - Agent selection, query analysis, and reranking prompts

Configuration

• Added new environment variables for fine-grained control:

Deployment

• Updated Docker configurations (ao-hybrid.Dockerfile)
• Added supporting services (PostgreSQL with pgvector, Redis, Celery workers)
• Updated Makefile with new build targets

Type of Change

• New feature
• Bugfix
• Refactor
• Documentation
• Other

Architecture Diagram

┌─────────────────┐
│  User Query     │
└────────┬────────┘
         │
         ▼
┌─────────────────────────────────────────────────────┐
│              Recipient Chooser                       │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  │
│  │  BM25       │  │  Semantic   │  │  LLM        │  │
│  │  (0.25)     │+ │  pgvector   │→ │  Reranker   │  │
│  │             │  │  (0.75)     │  │  (GPT-4o)   │  │
│  └─────────────┘  └─────────────┘  └─────────────┘  │
└────────────────────────┬────────────────────────────┘
                         │
                         ▼
┌─────────────────────────────────────────────────────┐
│           Agent Orchestration Service                │
│  ┌──────────────────────────────────────────────┐   │
│  │         Parallel Execution (asyncio)          │   │
│  │  ┌─────────┐  ┌─────────┐  ┌─────────┐       │   │
│  │  │ Agent 1 │  │ Agent 2 │  │ Agent N │       │   │
│  │  └────┬────┘  └────┬────┘  └────┬────┘       │   │
│  └───────┼────────────┼────────────┼────────────┘   │
│          └────────────┼────────────┘                │
│                       ▼                             │
│  ┌──────────────────────────────────────────────┐   │
│  │        Response Synthesis (LLM)               │   │
│  └──────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────┘

Additional Comments

• Requires PostgreSQL with pgvector extension enabled
• Celery workers needed for background keyword learning
• Compatible with existing agent implementations
• Backward compatible configuration - features can be individually disabled