diff --git a/.gitignore b/.gitignore new file mode 100644 index 00000000..7a60b85e --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +__pycache__/ +*.pyc diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 00000000..457f44d9 --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,3 @@ +{ + "python.analysis.typeCheckingMode": "basic" +} \ No newline at end of file diff --git a/README.md b/README.md index 6346d623..0288292d 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,10 @@ A maintained fork of **Extended OpenAI Conversation** for **Home Assistant** tha - **Options gear** (model, strategy, effort, max tokens, temp/top‑p, prompt) via `OptionsFlowWithReload` - **Azure OpenAI** support (Base URL + API version) - **Async‑safe client** using HA’s shared HTTPX to avoid blocking SSL CA loads +- **Toolbox parity** with upstream (service execution, automations, history, REST/scrape/script/template/composite when configured) +- **Hosted web search** hook for reasoning models with graceful chat fallback +- **Optional MCP bridge** (lazy import; safe no-op when library absent) +- **Tool execution limits** (per-call timeout, depth/call caps, chat log breadcrumbs for start/result) ## Requirements - Home Assistant **2025.4.0+** recommended @@ -36,9 +40,59 @@ A maintained fork of **Extended OpenAI Conversation** for **Home Assistant** tha Open the integration card → **Configure** (gear): - **Model** (default `gpt-5`) - **Model strategy**: `auto` \| `force_chat_completions` \| `force_responses_api` -- **Use Responses API** (for non‑reasoning models when strategy is `auto`) +- **Use Responses API** (for non-reasoning models when strategy is `auto`) - **Reasoning effort**: `minimal` \| `low` \| `medium` \| `high` - **Max tokens**, **Temperature**, **Top‑p**, **System prompt** +- **Functions YAML**: list of tool specs (defaults cover service/automations/history); extend to add REST/scrape/script/template/composite entries. +- **Tool limits**: max tool calls/iterations, per-call timeout, response size cap. +- **Web search**: enable hosted search, context size (`small`/`medium`/`large` or raw token budget), optional approximate location sharing. +- **MCP bridge**: toggles + timeout/payload caps (requires [`mcp`](https://pypi.org/project/mcp/) to expose external tools). + +### Toolbox +- Default toolbox exposes `execute_service`, `add_automation`, and `get_history` +- Extend via **Functions YAML** in options. Each entry: + +```yaml +- spec: + name: rest_weather + description: Fetch the daily forecast from the weather API. + parameters: + type: object + properties: + city: + type: string + function: + type: rest + resource: https://api.example.com/forecast + method: GET + headers: + Authorization: "{{ secrets.weather_token }}" + payload_template: "{{ {'city': city} | tojson }}" +``` + +- `scrape` tool lazily imports `beautifulsoup4` (installed via manifest); if import fails at runtime the tool returns a friendly error without breaking the integration. +- Tool calls are capped per turn and per loop, with start/result breadcrumbs logged to the Assist chat log for auditability. + +### Web search +- When `Enable hosted web search` is on, reasoning paths send the official `web_search` tool. +- Context size accepts `small`/`medium`/`large` or an integer budget (auto-mapped to low/medium/high). +- Optionally share approximate location (city/region/country/timezone) using HA config data. +- Chat Completions log a notice when the selected model lacks hosted search support and continue without failing. + +### MCP bridge +- Disabled by default; toggle **Enable MCP bridge** to attempt discovery via the optional [`mcp`](https://pypi.org/project/mcp/) SDK. +- When the SDK is unavailable the bridge is a no-op. Future releases can hook into `MCPBridge` for richer behaviour. +- Timeout and payload caps prevent runaway external tool calls. + +## Manual validation checklist +- Base conversation returns `ConversationResult` with language, `response_type`, and `continue_conversation` populated. +- `execute_service` honours Assist exposure; non-exposed entities return a readable denial. +- `get_history` responds with a concise summary (≤10 entries per entity). +- `rest` and `scrape` enforce per-call timeout/output caps; missing `beautifulsoup4` yields a graceful error. +- `composite` chains respect the orchestrator depth limit (safe failure message). +- Web search works end-to-end on reasoning models and logs a skip message on unsupported chat models. +- MCP bridge disabled → no change; when enabled (with SDK) tool discovery/execution routes through the orchestrator. +- Tool start/result breadcrumbs appear in the HA chat log for traceability. ### Recommended (long, smart chats) - Model: `gpt-5` diff --git a/custom_components/extended_openai_conversation/config_flow.py b/custom_components/extended_openai_conversation/config_flow.py index cb1e0dd8..3c8ab447 100644 --- a/custom_components/extended_openai_conversation/config_flow.py +++ b/custom_components/extended_openai_conversation/config_flow.py @@ -5,6 +5,7 @@ from typing import Any import voluptuous as vol +import yaml from homeassistant import config_entries from homeassistant.config_entries import ( ConfigFlow, @@ -26,6 +27,30 @@ CONF_TOP_P, CONF_MAX_TOKENS, CONF_REASONING_EFFORT, + CONF_FUNCTIONS, + CONF_FUNCTIONS_RAW, + CONF_MAX_TOOL_CALLS, + CONF_MAX_TOOL_CHAIN, + CONF_TOOL_TIMEOUT, + CONF_TOOL_MAX_OUTPUT_CHARS, + CONF_ENABLE_WEB_SEARCH, + CONF_WEB_SEARCH_CONTEXT_SIZE, + CONF_INCLUDE_HOME_LOCATION, + CONF_ENABLE_MCP, + CONF_MCP_TIMEOUT, + CONF_MCP_MAX_PAYLOAD, + DEFAULT_FUNCTIONS, + DEFAULT_MAX_TOOL_CALLS, + DEFAULT_MAX_TOOL_CHAIN, + DEFAULT_TOOL_TIMEOUT, + DEFAULT_TOOL_MAX_OUTPUT_CHARS, + DEFAULT_ENABLE_WEB_SEARCH, + DEFAULT_WEB_SEARCH_CONTEXT_SIZE, + DEFAULT_INCLUDE_HOME_LOCATION, + DEFAULT_ENABLE_MCP, + DEFAULT_MCP_TIMEOUT, + DEFAULT_MCP_MAX_PAYLOAD, + WEB_SEARCH_CONTEXT_SIZE_PRESETS, DEFAULT_BASE_URL, DEFAULT_CHAT_MODEL, DEFAULT_MODEL_STRATEGY, @@ -36,6 +61,39 @@ DEFAULT_PROMPT, ) +def _dump_functions_yaml(functions: list[dict[str, Any]] | None) -> str: + if not functions: + return "" + return yaml.safe_dump(functions, sort_keys=False) + + +def _parse_functions_yaml(raw: str) -> list[dict[str, Any]]: + if not raw.strip(): + return [] + + try: + loaded = yaml.safe_load(raw) or [] + except yaml.YAMLError as err: + raise vol.Invalid("invalid_yaml") from err + + if not isinstance(loaded, list): + raise vol.Invalid("invalid_yaml") + + normalized: list[dict[str, Any]] = [] + for item in loaded: + if not isinstance(item, dict): + raise vol.Invalid("invalid_yaml") + + spec = item.get("spec") + runtime = item.get("function") + if not isinstance(spec, dict) or not isinstance(runtime, dict): + raise vol.Invalid("invalid_yaml") + if not spec.get("name") or not runtime.get("type"): + raise vol.Invalid("invalid_yaml") + normalized.append({"spec": spec, "function": runtime}) + + return normalized + class ExtendedOpenAIConfigFlow(ConfigFlow, domain=DOMAIN): """Create the config entry.""" @@ -108,11 +166,47 @@ class EOCOptionsFlow(OptionsFlowWithReload): async def async_step_init( self, user_input: dict[str, Any] | None = None ) -> ConfigFlowResult: + errors: dict[str, str] = {} if user_input is not None: - return self.async_create_entry(data=user_input) + raw_functions = user_input.get(CONF_FUNCTIONS_RAW, "") + try: + parsed_functions = _parse_functions_yaml(raw_functions) + except vol.Invalid: + errors[CONF_FUNCTIONS_RAW] = "invalid_functions_yaml" + else: + user_input[CONF_FUNCTIONS] = parsed_functions + user_input[CONF_FUNCTIONS_RAW] = raw_functions.strip() + + if not errors: + return self.async_create_entry(data=user_input) # Build the form with suggested current values. opts = self.config_entry.options + functions_default = opts.get(CONF_FUNCTIONS) + functions_default_raw = opts.get(CONF_FUNCTIONS_RAW) + if functions_default_raw is None: + functions_default_raw = _dump_functions_yaml(functions_default or DEFAULT_FUNCTIONS) + if user_input and CONF_FUNCTIONS_RAW in user_input: + functions_default_raw = user_input.get(CONF_FUNCTIONS_RAW, functions_default_raw) + + max_tool_calls_default = opts.get(CONF_MAX_TOOL_CALLS, DEFAULT_MAX_TOOL_CALLS) + max_tool_chain_default = opts.get(CONF_MAX_TOOL_CHAIN, DEFAULT_MAX_TOOL_CHAIN) + tool_timeout_default = opts.get(CONF_TOOL_TIMEOUT, DEFAULT_TOOL_TIMEOUT) + tool_max_output_default = opts.get( + CONF_TOOL_MAX_OUTPUT_CHARS, DEFAULT_TOOL_MAX_OUTPUT_CHARS + ) + enable_search_default = opts.get( + CONF_ENABLE_WEB_SEARCH, DEFAULT_ENABLE_WEB_SEARCH + ) + search_context_default = opts.get( + CONF_WEB_SEARCH_CONTEXT_SIZE, DEFAULT_WEB_SEARCH_CONTEXT_SIZE + ) + include_location_default = opts.get( + CONF_INCLUDE_HOME_LOCATION, DEFAULT_INCLUDE_HOME_LOCATION + ) + enable_mcp_default = opts.get(CONF_ENABLE_MCP, DEFAULT_ENABLE_MCP) + mcp_timeout_default = opts.get(CONF_MCP_TIMEOUT, DEFAULT_MCP_TIMEOUT) + mcp_payload_default = opts.get(CONF_MCP_MAX_PAYLOAD, DEFAULT_MCP_MAX_PAYLOAD) schema = vol.Schema( { @@ -128,6 +222,35 @@ async def async_step_init( vol.Optional(CONF_TOP_P, default=opts.get(CONF_TOP_P, DEFAULT_TOP_P)): vol.Coerce(float), vol.Optional(CONF_MAX_TOKENS, default=opts.get(CONF_MAX_TOKENS, DEFAULT_MAX_TOKENS)): vol.Coerce(int), vol.Optional("prompt", default=opts.get("prompt", DEFAULT_PROMPT)): str, + vol.Optional(CONF_FUNCTIONS_RAW, default=functions_default_raw): str, + vol.Optional( + CONF_MAX_TOOL_CALLS, default=max_tool_calls_default + ): vol.All(vol.Coerce(int), vol.Range(min=0, max=10)), + vol.Optional( + CONF_MAX_TOOL_CHAIN, default=max_tool_chain_default + ): vol.All(vol.Coerce(int), vol.Range(min=1, max=8)), + vol.Optional(CONF_TOOL_TIMEOUT, default=tool_timeout_default): vol.All( + vol.Coerce(float), vol.Range(min=1, max=60) + ), + vol.Optional( + CONF_TOOL_MAX_OUTPUT_CHARS, default=tool_max_output_default + ): vol.All(vol.Coerce(int), vol.Range(min=256, max=16384)), + vol.Optional(CONF_ENABLE_WEB_SEARCH, default=enable_search_default): bool, + vol.Optional( + CONF_WEB_SEARCH_CONTEXT_SIZE, + default=search_context_default, + ): vol.Any( + vol.In(list(WEB_SEARCH_CONTEXT_SIZE_PRESETS.keys())), + vol.All(vol.Coerce(int), vol.Range(min=128, max=4096)), + ), + vol.Optional(CONF_INCLUDE_HOME_LOCATION, default=include_location_default): bool, + vol.Optional(CONF_ENABLE_MCP, default=enable_mcp_default): bool, + vol.Optional(CONF_MCP_TIMEOUT, default=mcp_timeout_default): vol.All( + vol.Coerce(float), vol.Range(min=1, max=60) + ), + vol.Optional( + CONF_MCP_MAX_PAYLOAD, default=mcp_payload_default + ): vol.All(vol.Coerce(int), vol.Range(min=512, max=65536)), } ) - return self.async_show_form(step_id="init", data_schema=schema) + return self.async_show_form(step_id="init", data_schema=schema, errors=errors) diff --git a/custom_components/extended_openai_conversation/const.py b/custom_components/extended_openai_conversation/const.py index 57ed1453..afc80eb2 100644 --- a/custom_components/extended_openai_conversation/const.py +++ b/custom_components/extended_openai_conversation/const.py @@ -30,6 +30,33 @@ # Optional scaffolding (off by default) CONF_MEMORY_ENABLED = "memory_enabled" CONF_MEMORY_DEFAULT_NAMESPACE = "memory_default_namespace" +CONF_MEMORY_BASE_URL = "memory_base_url" +CONF_MEMORY_API_KEY = "memory_api_key" +CONF_MEMORY_WRITE_PATH = "memory_write_path" +CONF_MEMORY_SEARCH_PATH = "memory_search_path" + +# Toolbox / tools +CONF_FUNCTIONS = "functions" +CONF_FUNCTIONS_RAW = "functions_yaml" +CONF_MAX_TOOL_CALLS = "max_tool_calls" +CONF_MAX_TOOL_CHAIN = "max_tool_chain" +CONF_TOOL_TIMEOUT = "tool_timeout" +CONF_TOOL_MAX_OUTPUT_CHARS = "tool_max_output_chars" +CONF_ENABLE_WEB_SEARCH = "enable_web_search" +CONF_WEB_SEARCH_CONTEXT_SIZE = "web_search_context_size" +CONF_INCLUDE_HOME_LOCATION = "include_home_location" +CONF_ENABLE_MCP = "enable_mcp" +CONF_MCP_TIMEOUT = "mcp_timeout" +CONF_MCP_MAX_PAYLOAD = "mcp_max_payload" + +WEB_SEARCH_CONTEXT_SIZE_SMALL = "small" +WEB_SEARCH_CONTEXT_SIZE_MEDIUM = "medium" +WEB_SEARCH_CONTEXT_SIZE_LARGE = "large" +WEB_SEARCH_CONTEXT_SIZE_PRESETS = { + WEB_SEARCH_CONTEXT_SIZE_SMALL: 256, + WEB_SEARCH_CONTEXT_SIZE_MEDIUM: 512, + WEB_SEARCH_CONTEXT_SIZE_LARGE: 1024, +} # Optional service SERVICE_QUERY_IMAGE = "query_image" @@ -47,3 +74,228 @@ DEFAULT_PROMPT = "" DEFAULT_MEMORY_ENABLED = False DEFAULT_MEMORY_DEFAULT_NAMESPACE = "default" +DEFAULT_MEMORY_BASE_URL = None +DEFAULT_MEMORY_API_KEY = None +DEFAULT_MEMORY_WRITE_PATH = "/v1/memory/write" +DEFAULT_MEMORY_SEARCH_PATH = "/v1/memory/search" +DEFAULT_FUNCTIONS: list[dict[str, object]] = [ + { + "spec": { + "name": "execute_service", + "description": "Call one or more Home Assistant services on entities exposed to Assist.", + "parameters": { + "type": "object", + "properties": { + "list": { + "type": "array", + "description": "Service calls to execute in order.", + "items": { + "type": "object", + "properties": { + "domain": { + "type": "string", + "description": "Service domain, e.g. light or climate.", + }, + "service": { + "type": "string", + "description": "The service to call within the domain.", + }, + "service_data": { + "type": "object", + "description": "Service data including entity_id for an exposed entity.", + }, + }, + "required": ["domain", "service"], + }, + "minItems": 1, + } + }, + "required": ["list"], + }, + }, + "function": {"type": "native", "name": "execute_service"}, + }, + { + "spec": { + "name": "add_automation", + "description": "Append a YAML automation to automations.yaml and reload automations.", + "parameters": { + "type": "object", + "properties": { + "automation_config": { + "type": "string", + "description": "YAML string describing the automation to add.", + } + }, + "required": ["automation_config"], + }, + }, + "function": {"type": "native", "name": "add_automation"}, + }, + { + "spec": { + "name": "get_history", + "description": "Fetch recent state history for entities exposed to Assist.", + "parameters": { + "type": "object", + "properties": { + "entity_ids": { + "type": "array", + "items": {"type": "string"}, + "description": "Entity IDs to query.", + }, + "start_time": { + "type": "string", + "description": "ISO timestamp for the start of the window.", + }, + "end_time": { + "type": "string", + "description": "ISO timestamp for the end of the window.", + }, + }, + "required": ["entity_ids"], + }, + }, + "function": {"type": "native", "name": "get_history"}, + }, + { + "spec": { + "name": "rest", + "description": "Perform an HTTP request via Home Assistant's REST data helper.", + "parameters": { + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "HTTP or HTTPS endpoint to query.", + }, + "method": { + "type": "string", + "enum": ["GET", "POST"], + "description": "HTTP method (default GET).", + }, + "headers": { + "type": "object", + "additionalProperties": {"type": "string"}, + "description": "Optional request headers.", + }, + "params": { + "type": "object", + "additionalProperties": {"type": "string"}, + "description": "Optional query parameters.", + }, + "timeout": { + "type": "integer", + "description": "Optional timeout in seconds (default 10).", + }, + "body": { + "type": "string", + "description": "Optional request body for POST.", + }, + }, + "required": ["url"], + }, + }, + "function": {"type": "rest"}, + }, + { + "spec": { + "name": "scrape", + "description": "Scrape structured text from a web page using a CSS selector.", + "parameters": { + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "HTTP or HTTPS page to scrape.", + }, + "select": { + "type": "string", + "description": "CSS selector that locates the desired node.", + }, + "attribute": { + "type": "string", + "description": "Optional attribute to read instead of text content.", + }, + "index": { + "type": "integer", + "description": "Zero-based index when the selector matches multiple nodes.", + }, + }, + "required": ["url", "select"], + }, + }, + "function": {"type": "scrape"}, + }, + { + "spec": { + "name": "script", + "description": "Execute a Home Assistant script sequence or script entity.", + "parameters": { + "type": "object", + "properties": { + "sequence": { + "type": "array", + "items": {"type": "object"}, + "description": "Home Assistant script steps to run.", + }, + "entity_id": { + "type": "string", + "description": "Existing script entity to run if no sequence is provided.", + }, + }, + }, + }, + "function": {"type": "script"}, + }, + { + "spec": { + "name": "template", + "description": "Render a Home Assistant template with the provided arguments.", + "parameters": { + "type": "object", + "properties": { + "template": { + "type": "string", + "description": "Template string to render.", + }, + "parse_json": { + "type": "boolean", + "description": "Parse the rendered template as JSON when true.", + }, + }, + "required": ["template"], + }, + }, + "function": {"type": "template"}, + }, + { + "spec": { + "name": "composite", + "description": "Run multiple tools sequentially, sharing variables between them.", + "parameters": { + "type": "object", + "properties": { + "sequence": { + "type": "array", + "items": {"type": "object"}, + "description": "List of tool invocations to execute in order.", + }, + }, + "required": ["sequence"], + }, + }, + "function": {"type": "composite"}, + }, +] +DEFAULT_FUNCTIONS_RAW = "" +DEFAULT_MAX_TOOL_CALLS = 4 +DEFAULT_MAX_TOOL_CHAIN = 3 +DEFAULT_TOOL_TIMEOUT = 12 +DEFAULT_TOOL_MAX_OUTPUT_CHARS = 4000 +DEFAULT_ENABLE_WEB_SEARCH = False +DEFAULT_WEB_SEARCH_CONTEXT_SIZE = WEB_SEARCH_CONTEXT_SIZE_MEDIUM +DEFAULT_INCLUDE_HOME_LOCATION = False +DEFAULT_ENABLE_MCP = False +DEFAULT_MCP_TIMEOUT = 8 +DEFAULT_MCP_MAX_PAYLOAD = 4096 diff --git a/custom_components/extended_openai_conversation/conversation.py b/custom_components/extended_openai_conversation/conversation.py index cd9ee71d..c37071aa 100644 --- a/custom_components/extended_openai_conversation/conversation.py +++ b/custom_components/extended_openai_conversation/conversation.py @@ -2,24 +2,31 @@ from __future__ import annotations +import asyncio +import json import logging -from typing import Any, Optional, Literal +from typing import Any, Literal, Optional +from homeassistant.components import conversation as conv from homeassistant.components.conversation import ( + AssistantContent, + ChatLog, ConversationEntity, ConversationEntityFeature, - ChatLog, ) from homeassistant.helpers import intent from homeassistant.const import CONF_API_KEY +from homeassistant.helpers.llm import ToolInput +from openai._exceptions import OpenAIError # Try to import the real ConversationResult; otherwise provide a compatible shim. try: - from homeassistant.components.conversation.agent import ConversationResult # type: ignore[attr-defined] + # Prefer modern path (HA 2024.8+): models.ConversationResult + from homeassistant.components.conversation.models import ConversationResult # type: ignore[attr-defined] except Exception: try: - from homeassistant.components.conversation import agent as _agent_mod # type: ignore[attr-defined] - ConversationResult = _agent_mod.ConversationResult # type: ignore[assignment] + # Legacy fallback used by older cores + from homeassistant.components.conversation.agent import ConversationResult # type: ignore[attr-defined] except Exception: class ConversationResult: # type: ignore[misc] @@ -57,7 +64,11 @@ def as_dict(self) -> dict[str, Any]: MODEL_STRATEGY_FORCE_RESPONSES, ) from .model_capabilities import detect_model_capabilities -from .responses_adapter import response_text_from_responses_result +from .responses_adapter import ( + response_text_from_responses_result, + extract_function_calls_from_response, +) +from .tools_orchestrator import ToolsOrchestrator, ToolExecutionError _LOGGER = logging.getLogger(__name__) @@ -76,6 +87,8 @@ class ExtendedOpenAIConversationEntity(ConversationEntity): def __init__(self, hass, entry) -> None: self.hass = hass self.entry = entry + self._logged_minimal_effort = False + self._preferred_web_search_type: Optional[str] = None @property def supported_languages(self) -> list[str] | Literal["*"]: @@ -96,6 +109,9 @@ async def _async_handle_message( self, user_input: Any, chat_log: ChatLog ) -> ConversationResult: """Handle a message from Assist.""" + # ConversationEntity contract documented in + # homeassistant.components.conversation (2024.12): _async_handle_message + # receives (ConversationInput, ChatLog) and returns ConversationResult. # Lazy import avoids SDK import at module load from .openai_support import build_async_client @@ -124,8 +140,20 @@ async def _async_handle_message( _LOGGER.debug( "EOC: model=%s caps=%s strategy=%s use_responses=%s", - model, caps, strategy, use_responses + model, + caps, + strategy, + use_responses, + ) + + agent_id = user_input.agent_id or self.entity_id or DOMAIN + orchestrator = ToolsOrchestrator( + self.hass, + options=options, + chat_log=chat_log, + agent_id=agent_id, ) + orchestrator.reset() client = build_async_client( self.hass, @@ -139,78 +167,67 @@ async def _async_handle_message( user_text = user_input.text if use_responses: - # ✅ Correct Responses API schema: - # - System instructions go into "instructions" - # - Input content items use type "input_text" - payload: dict[str, Any] = { - "model": model, - "input": [ - { - "role": "user", - "content": [{"type": "input_text", "text": user_text}], - } - ], - } - if sys_prompt: - payload["instructions"] = sys_prompt - - max_tokens = int(options.get(CONF_MAX_TOKENS) or DEFAULT_MAX_TOKENS) - if max_tokens > 0: - payload["max_output_tokens"] = max_tokens - - if caps.is_reasoning: - effort = options.get(CONF_REASONING_EFFORT) - # OpenAI 1.x expects nested object: {"reasoning": {"effort": "low|medium|high|minimal"}} - payload["reasoning"] = {"effort": effort} - try: - result = await client.responses.create(**payload) # type: ignore[arg-type] - text = response_text_from_responses_result(result) - cont = _should_continue(text) - return _ok( - text=text, - language=user_input.language, - conversation_id=user_input.conversation_id, - cont=cont, - ) - except Exception as err: + responses_result = await self._run_responses_flow( + client=client, + model=model, + sys_prompt=sys_prompt, + user_text=user_text, + options=options, + caps=caps, + user_input=user_input, + orchestrator=orchestrator, + chat_log=chat_log, + agent_id=agent_id, + ) + except ToolExecutionError as err: + _LOGGER.warning("Responses tool failure: %s", err) + return _err(str(err), user_input.language) + except Exception as err: # pragma: no cover - defensive guard _LOGGER.exception("Responses API failure: %s", err) return _err(str(err), user_input.language) - # Fallback: Chat Completions - messages = [] - if sys_prompt: - messages.append({"role": "system", "content": sys_prompt}) - messages.append({"role": "user", "content": user_text}) - - kwargs: dict[str, Any] = {"model": model, "messages": messages} - - max_tokens = int(options.get(CONF_MAX_TOKENS) or DEFAULT_MAX_TOKENS) - if max_tokens > 0: - # Some reasoning-capable chat models use max_completion_tokens - kwargs["max_completion_tokens" if caps.is_reasoning else "max_tokens"] = max_tokens - - if caps.accepts_temperature: - if (t := options.get(CONF_TEMPERATURE)) is not None: - kwargs["temperature"] = float(t) - if (p := options.get(CONF_TOP_P)) is not None: - kwargs["top_p"] = float(p) - - try: - result = await client.chat.completions.create(**kwargs) # type: ignore[arg-type] - msg = result.choices[0].message - text = msg.content or "" + text = response_text_from_responses_result(responses_result) cont = _should_continue(text) + self._log_final_assistant_message(chat_log, agent_id, text) return _ok( text=text, language=user_input.language, conversation_id=user_input.conversation_id, cont=cont, ) - except Exception as err: + + orchestrator.configure_chat_web_search(model) + + try: + text = await self._run_chat_flow( + client=client, + model=model, + sys_prompt=sys_prompt, + user_text=user_text, + options=options, + caps=caps, + user_input=user_input, + orchestrator=orchestrator, + chat_log=chat_log, + agent_id=agent_id, + ) + except ToolExecutionError as err: + _LOGGER.warning("Chat tool failure: %s", err) + return _err(str(err), user_input.language) + except Exception as err: # pragma: no cover _LOGGER.exception("Chat Completions failure: %s", err) return _err(str(err), user_input.language) + cont = _should_continue(text) + self._log_final_assistant_message(chat_log, agent_id, text) + return _ok( + text=text, + language=user_input.language, + conversation_id=user_input.conversation_id, + cont=cont, + ) + def _default_options(self) -> dict[str, Any]: return { CONF_CHAT_MODEL: self.entry.options.get(CONF_CHAT_MODEL) @@ -225,6 +242,468 @@ def _default_options(self) -> dict[str, Any]: "prompt": self.entry.options.get("prompt", DEFAULT_PROMPT), } + async def _run_responses_flow( + self, + *, + client, + model: str, + sys_prompt: str, + user_text: str, + options: dict[str, Any], + caps, + user_input, + orchestrator: ToolsOrchestrator, + chat_log: ChatLog, + agent_id: str, + ): + payload: dict[str, Any] = { + "model": model, + "input": [ + { + "role": "user", + "content": [{"type": "input_text", "text": user_text}], + } + ], + } + + if sys_prompt: + payload["instructions"] = sys_prompt + + max_tokens = int(options.get(CONF_MAX_TOKENS) or DEFAULT_MAX_TOKENS) + if max_tokens > 0: + payload["max_output_tokens"] = max_tokens + + if caps.is_reasoning: + effort = options.get(CONF_REASONING_EFFORT) + if effort: + if ( + effort == "minimal" + and not _model_supports_minimal_reasoning(model) + ): + if not self._logged_minimal_effort: + _LOGGER.debug( + "Reasoning effort 'minimal' not supported for model %s; " + "downgrading to 'low'.", + model or "", + ) + self._logged_minimal_effort = True + effort = "low" + payload["reasoning"] = {"effort": effort} + + tools = orchestrator.conversation_tools_for_responses( + self.hass, + model=model, + tool_type_override=self._preferred_web_search_type, + ) + if tools: + payload["tools"] = tools + + # Responses API expects tool continuations via function_call_output entries: + # https://platform.openai.com/docs/api-reference/responses/create + try: + result = await client.responses.create(**payload) + except OpenAIError as err: + fallback_type = orchestrator.web_search_fallback_type + current_type = orchestrator.web_search_tool_type + if ( + current_type + and fallback_type + and _should_retry_web_search(current_type, str(err)) + ): + # Some tenants still expose 'web_search_preview'; retry gracefully. + _LOGGER.debug( + "Web search tool type '%s' rejected; retrying with '%s'.", + current_type, + fallback_type, + ) + payload["tools"] = self._swap_web_search_tool_type( + payload.get("tools", []), current_type, fallback_type + ) + orchestrator.update_web_search_tool_type(fallback_type) + self._preferred_web_search_type = fallback_type + result = await client.responses.create(**payload) + else: + raise + return await self._handle_responses_tool_calls( + client=client, + model=model, + result=result, + orchestrator=orchestrator, + user_input=user_input, + chat_log=chat_log, + agent_id=agent_id, + ) + + async def _handle_responses_tool_calls( + self, + *, + client, + model: str, + result, + orchestrator: ToolsOrchestrator, + user_input, + chat_log: ChatLog, + agent_id: str, + ): + depth = 0 + + while True: + calls = extract_function_calls_from_response(result) + if not calls: + _LOGGER.debug( + "Responses loop depth %s: no tool calls returned; exiting.", + depth, + ) + return result + + _LOGGER.debug( + "Responses loop depth %s: received %d call(s): %s", + depth, + len(calls), + [self._normalize_tool_call(call)[1] for call in calls], + ) + + outputs: list[dict[str, Any]] = [] + if depth >= orchestrator.max_chain_depth: + _LOGGER.debug( + "Responses tool loop stopped at depth %s (limit %s)", + depth, + orchestrator.max_chain_depth, + ) + for idx, call in enumerate(calls): + call_id = call.get("call_id") or call.get("id") or str(idx) + outputs.append( + { + "type": "function_call_output", + "call_id": call_id, + "output": "Tool execution halted: maximum depth reached.", + } + ) + result = await client.responses.create( + model=model, + previous_response_id=getattr(result, "id", None), + input=outputs, + ) + return result + + self._log_assistant_tool_calls(chat_log, agent_id, None, calls) + + for call in calls: + name = call.get("name") + arguments = call.get("arguments") + call_id = call.get("call_id") or call.get("id") or name + if not name: + continue + _LOGGER.debug( + "Calling tool '%s' via Responses (call_id=%s)", + name, + call_id, + ) + try: + output = await orchestrator.execute_tool_call( + name=name, + arguments=arguments, + user_input=user_input, + call_id=call_id, + ) + except ToolExecutionError as err: + output = f"Tool {name} failed: {err}" + _LOGGER.debug( + "Tool '%s' execution returned error via Responses: %s", + name, + err, + ) + + outputs.append( + { + "type": "function_call_output", + "call_id": call_id or (name or "unknown"), + "output": output, + } + ) + _LOGGER.debug( + "Appending function_call_output for call_id=%s", + call_id or name, + ) + + if not outputs: + _LOGGER.debug( + "Responses loop depth %s produced no outputs; returning.", + depth, + ) + return result + + prev_id = getattr(result, "id", None) + call_ids = [output["call_id"] for output in outputs] + _LOGGER.debug( + "Posting function_call_output for call_id(s) %s with previous_response_id=%s.", + call_ids, + prev_id, + ) + result = await client.responses.create( + model=model, + previous_response_id=prev_id, + input=outputs, + ) + _LOGGER.debug( + "Submitted %d function_call_output payload(s); continuing Responses loop.", + len(outputs), + ) + depth += 1 + + async def _run_chat_flow( + self, + *, + client, + model: str, + sys_prompt: str, + user_text: str, + options: dict[str, Any], + caps, + user_input, + orchestrator: ToolsOrchestrator, + chat_log: ChatLog, + agent_id: str, + ) -> str: + messages: list[dict[str, Any]] = [] + if sys_prompt: + messages.append({"role": "system", "content": sys_prompt}) + messages.append({"role": "user", "content": user_text}) + + kwargs: dict[str, Any] = {"model": model, "messages": messages} + + max_tokens = int(options.get(CONF_MAX_TOKENS) or DEFAULT_MAX_TOKENS) + if max_tokens > 0: + # OpenAI Chat Completions accepts 'max_tokens' for budget control: + # https://platform.openai.com/docs/api-reference/chat/create + kwargs["max_tokens"] = max_tokens + + if caps.accepts_temperature: + if (t := options.get(CONF_TEMPERATURE)) is not None: + kwargs["temperature"] = float(t) + if (p := options.get(CONF_TOP_P)) is not None: + kwargs["top_p"] = float(p) + + tools = orchestrator.conversation_tools_for_chat() + if tools: + kwargs["tools"] = tools + + depth = 0 + while True: + completion = await client.chat.completions.create(**kwargs) + message = completion.choices[0].message + content = message.content or "" + tool_calls = getattr(message, "tool_calls", None) or [] + + if not tool_calls: + _LOGGER.debug( + "Chat loop depth %s: no tool calls; returning assistant message.", + depth, + ) + return content + + _LOGGER.debug( + "Chat loop depth %s: received %d tool call(s): %s", + depth, + len(tool_calls), + [self._normalize_tool_call(call)[1] for call in tool_calls], + ) + + if depth >= orchestrator.max_chain_depth: + suffix = "Tool chain limit reached; unable to continue." + _LOGGER.debug( + "Chat tool loop stopped at depth %s (limit %s)", + depth, + orchestrator.max_chain_depth, + ) + return f"{content}\n{suffix}" if content else suffix + + self._log_assistant_tool_calls(chat_log, agent_id, content, tool_calls) + + assistant_entry = { + "role": "assistant", + "content": content, + "tool_calls": [ + call.model_dump() if hasattr(call, "model_dump") else { + "id": getattr(call, "id", None), + "type": getattr(call, "type", "function"), + "function": { + "name": getattr(getattr(call, "function", None), "name", None), + "arguments": getattr( + getattr(call, "function", None), "arguments", "{}" + ), + }, + } + for call in tool_calls + ], + } + messages.append(assistant_entry) + + for call in tool_calls: + call_id = getattr(call, "id", None) + func = getattr(call, "function", None) + call_name = getattr(func, "name", None) + call_arguments = getattr(func, "arguments", "{}") + if not call_name: + continue + _LOGGER.debug( + "Calling tool '%s' via Chat Completions (call_id=%s)", + call_name, + call_id, + ) + try: + output = await orchestrator.execute_tool_call( + name=call_name, + arguments=call_arguments, + user_input=user_input, + call_id=call_id, + ) + except ToolExecutionError as err: + output = f"Tool {call_name} failed: {err}" + _LOGGER.debug( + "Tool '%s' execution returned error via Chat Completions: %s", + call_name, + err, + ) + + messages.append( + { + "role": "tool", + "tool_call_id": call_id, + "content": output, + } + ) + _LOGGER.debug( + "Appended tool result message for call_id=%s", + call_id, + ) + + kwargs["messages"] = messages + _LOGGER.debug( + "Continuing Chat tool loop with %d accumulated messages.", + len(messages), + ) + depth += 1 + + def _log_assistant_tool_calls( + self, + chat_log: ChatLog | None, + agent_id: str, + content: str | None, + tool_calls: list[Any], + ) -> None: + if not chat_log or not tool_calls: + return + + tool_inputs: list[ToolInput] = [] + for call in tool_calls: + call_id, call_name, args = self._normalize_tool_call(call) + if not call_name: + continue + tool_inputs.append( + ToolInput( + tool_name=call_name, + tool_args=args, + id=call_id or call_name, + external=True, + ) + ) + + if not tool_inputs: + return + + chat_log.async_add_assistant_content_without_tools( + AssistantContent( + agent_id=agent_id, + content=content if content else None, + tool_calls=tool_inputs, + ) + ) + + def _normalize_tool_call( + self, call: Any + ) -> tuple[Optional[str], Optional[str], dict[str, Any]]: + call_id: Optional[str] = None + name: Optional[str] = None + raw_args: Any = None + + if hasattr(call, "id") or hasattr(call, "function"): + call_id = getattr(call, "id", None) + func = getattr(call, "function", None) + name = getattr(func, "name", None) + raw_args = getattr(func, "arguments", None) + elif isinstance(call, dict): + call_id = call.get("id") or call.get("call_id") + if (func := call.get("function")) and isinstance(func, dict): + name = func.get("name") + raw_args = func.get("arguments") + else: + name = call.get("name") + raw_args = call.get("arguments") + + return call_id, name, self._parse_tool_arguments(raw_args) + + @staticmethod + def _parse_tool_arguments(raw: Any) -> dict[str, Any]: + if isinstance(raw, dict): + return raw + if isinstance(raw, str): + try: + parsed = json.loads(raw) + except json.JSONDecodeError: + return {"raw": raw} + if isinstance(parsed, dict): + return parsed + return {"value": parsed} + if raw is None: + return {} + return {"value": raw} + + def _log_final_assistant_message( + self, chat_log: ChatLog | None, agent_id: str, text: str + ) -> None: + if not chat_log: + return + chat_log.async_add_assistant_content_without_tools( + AssistantContent(agent_id=agent_id, content=text or "") + ) + + @staticmethod + def _swap_web_search_tool_type( + tools: list[dict[str, Any]], current_type: str, new_type: str + ) -> list[dict[str, Any]]: + swapped: list[dict[str, Any]] = [] + for tool in tools: + if tool.get("type") == current_type: + updated = dict(tool) + updated["type"] = new_type + swapped.append(updated) + else: + swapped.append(tool) + _LOGGER.debug( + "Registered hosted web search tool using type '%s'.", + new_type, + ) + return swapped + + +def _model_supports_minimal_reasoning(model: Optional[str]) -> bool: + """Return True if the target model advertises 'minimal' reasoning effort. + + GPT-5 family models document the 'minimal' effort; other endpoints generally + only accept 'low'|'medium'|'high' (see OpenAI Responses API docs). + """ + if not model: + return False + name = model.lower() + return name.startswith("gpt-5") + + +def _should_retry_web_search(current_type: str, error_text: str) -> bool: + """Detect tool-type errors so we can retry with the preview variant.""" + lowered = error_text.lower() + return current_type in lowered and "web_search" in lowered + def _ok(*, text: str, language: Optional[str], conversation_id: Optional[str], cont: bool) -> ConversationResult: response = intent.IntentResponse(language=language) diff --git a/custom_components/extended_openai_conversation/exceptions.py b/custom_components/extended_openai_conversation/exceptions.py index 8acf551c..e00cb410 100644 --- a/custom_components/extended_openai_conversation/exceptions.py +++ b/custom_components/extended_openai_conversation/exceptions.py @@ -5,9 +5,9 @@ class EntityNotFound(HomeAssistantError): """When referenced entity not found.""" - def __init__(self, entity_id: str) -> None: + def __init__(self, entity_id) -> None: """Initialize error.""" - super().__init__(self, f"entity {entity_id} not found") + super().__init__(f"entity {entity_id} not found") self.entity_id = entity_id def __str__(self) -> str: @@ -18,9 +18,9 @@ def __str__(self) -> str: class EntityNotExposed(HomeAssistantError): """When referenced entity not exposed.""" - def __init__(self, entity_id: str) -> None: + def __init__(self, entity_id) -> None: """Initialize error.""" - super().__init__(self, f"entity {entity_id} not exposed") + super().__init__(f"entity {entity_id} not exposed") self.entity_id = entity_id def __str__(self) -> str: @@ -34,7 +34,6 @@ class CallServiceError(HomeAssistantError): def __init__(self, domain: str, service: str, data: object) -> None: """Initialize error.""" super().__init__( - self, f"unable to call service {domain}.{service} with data {data}. One of 'entity_id', 'area_id', or 'device_id' is required", ) self.domain = domain @@ -51,7 +50,7 @@ class FunctionNotFound(HomeAssistantError): def __init__(self, function: str) -> None: """Initialize error.""" - super().__init__(self, f"function '{function}' does not exist") + super().__init__(f"function '{function}' does not exist") self.function = function def __str__(self) -> str: @@ -64,7 +63,7 @@ class NativeNotFound(HomeAssistantError): def __init__(self, name: str) -> None: """Initialize error.""" - super().__init__(self, f"native function '{name}' does not exist") + super().__init__(f"native function '{name}' does not exist") self.name = name def __str__(self) -> str: @@ -78,7 +77,6 @@ class FunctionLoadFailed(HomeAssistantError): def __init__(self) -> None: """Initialize error.""" super().__init__( - self, "failed to load functions. Verify functions are valid in a yaml format", ) @@ -93,7 +91,6 @@ class ParseArgumentsFailed(HomeAssistantError): def __init__(self, arguments: str) -> None: """Initialize error.""" super().__init__( - self, f"failed to parse arguments `{arguments}`. Increase maximum token to avoid the issue.", ) self.arguments = arguments @@ -109,7 +106,6 @@ class TokenLengthExceededError(HomeAssistantError): def __init__(self, token: int) -> None: """Initialize error.""" super().__init__( - self, f"token length(`{token}`) exceeded. Increase maximum token to avoid the issue.", ) self.token = token @@ -125,7 +121,6 @@ class InvalidFunction(HomeAssistantError): def __init__(self, function_name: str) -> None: """Initialize error.""" super().__init__( - self, f"failed to validate function `{function_name}`", ) self.function_name = function_name diff --git a/custom_components/extended_openai_conversation/manifest.json b/custom_components/extended_openai_conversation/manifest.json index 5cb59e3f..7964e0a1 100644 --- a/custom_components/extended_openai_conversation/manifest.json +++ b/custom_components/extended_openai_conversation/manifest.json @@ -8,8 +8,8 @@ "iot_class": "cloud_polling", "integration_type": "service", "dependencies": ["conversation"], - "requirements": ["openai>=1.0.0,<2.0.0"], - "version": "1.4.1", + "requirements": ["openai>=1.0.0,<2.0.0", "beautifulsoup4>=4.12"], + "version": "1.5.5b1", "loggers": ["custom_components.extended_openai_conversation"], "platforms": ["conversation"] } diff --git a/custom_components/extended_openai_conversation/responses_adapter.py b/custom_components/extended_openai_conversation/responses_adapter.py index 00f7d858..08635171 100644 --- a/custom_components/extended_openai_conversation/responses_adapter.py +++ b/custom_components/extended_openai_conversation/responses_adapter.py @@ -53,3 +53,40 @@ def response_text_from_responses_result(result: Any) -> str: pass return "" + + +def extract_function_calls_from_response(result: Any) -> list[dict[str, Any]]: + """Return function-call payloads from a Responses API result.""" + + try: + data = result.model_dump() + except AttributeError: + data = result + + if not isinstance(data, dict): + return [] + + calls: list[dict[str, Any]] = [] + + def _maybe_add(item: Any) -> None: + if not item: + return + if hasattr(item, "model_dump"): + item = item.model_dump() + if not isinstance(item, dict): + return + if item.get("type") == "function_call": + calls.append(item) + + for entry in data.get("output", []): + _maybe_add(entry) + if not isinstance(entry, dict) and not hasattr(entry, "model_dump"): + continue + if hasattr(entry, "model_dump"): + entry = entry.model_dump() + if not isinstance(entry, dict): + continue + for part in entry.get("content", []): + _maybe_add(part) + + return calls diff --git a/custom_components/extended_openai_conversation/services.py b/custom_components/extended_openai_conversation/services.py index 1a09a6bb..80bb114d 100644 --- a/custom_components/extended_openai_conversation/services.py +++ b/custom_components/extended_openai_conversation/services.py @@ -4,7 +4,6 @@ from pathlib import Path from urllib.parse import urlparse -from openai import AsyncOpenAI from openai._exceptions import OpenAIError from openai.types.chat.chat_completion_content_part_image_param import ( ChatCompletionContentPartImageParam, @@ -21,7 +20,15 @@ from homeassistant.helpers import config_validation as cv, selector from homeassistant.helpers.typing import ConfigType -from .const import DOMAIN, SERVICE_QUERY_IMAGE +from .const import ( + DOMAIN, + SERVICE_QUERY_IMAGE, + CONF_API_KEY, + CONF_BASE_URL, + CONF_API_VERSION, + CONF_ORGANIZATION, +) +from .openai_support import build_async_client QUERY_IMAGE_SCHEMA = vol.Schema( { @@ -59,10 +66,21 @@ async def query_image(call: ServiceCall) -> ServiceResponse: } ] _LOGGER.debug("Prompt for %s: %s", model, messages) + entry_id = call.data["config_entry"] + entry = hass.config_entries.async_get_entry(entry_id) + if entry is None: + raise HomeAssistantError(f"Config entry {entry_id} not found") + + data = entry.data + client = build_async_client( + hass, + api_key=data[CONF_API_KEY], + base_url=data.get(CONF_BASE_URL), + api_version=data.get(CONF_API_VERSION), + organization=data.get(CONF_ORGANIZATION), + ) - response = await AsyncOpenAI( - api_key=hass.data[DOMAIN][call.data["config_entry"]]["api_key"] - ).chat.completions.create( + response = await client.chat.completions.create( model=model, messages=messages, max_tokens=call.data["max_tokens"], diff --git a/custom_components/extended_openai_conversation/strings.json b/custom_components/extended_openai_conversation/strings.json index e37f722f..ffa7a934 100644 --- a/custom_components/extended_openai_conversation/strings.json +++ b/custom_components/extended_openai_conversation/strings.json @@ -39,9 +39,26 @@ "temperature": "Temperature (non‑reasoning only)", "top_p": "Top‑p (non‑reasoning only)", "max_tokens": "Max output tokens", - "prompt": "System prompt (optional)" + "prompt": "System prompt (optional)", + "functions_yaml": "Functions YAML", + "max_tool_calls": "Max tool calls per turn", + "max_tool_chain": "Max tool iterations", + "tool_timeout": "Tool timeout (seconds)", + "tool_max_output_chars": "Tool output cap (characters)", + "enable_web_search": "Enable hosted web search", + "web_search_context_size": "Web search context size", + "include_home_location": "Share approximate Home Assistant location", + "enable_mcp": "Enable MCP bridge", + "mcp_timeout": "MCP tool timeout (seconds)", + "mcp_max_payload": "MCP payload cap (bytes)" } } } + }, + "errors": { + "rest_url_required": "REST tool requires a 'url' (http/https)", + "scrape_requirements": "Scrape tool requires a 'url' and 'select'", + "template_required": "Template tool requires a 'template' or 'value_template'", + "exposure_denied": "Some targets are hidden from Assist: {entities}" } } diff --git a/custom_components/extended_openai_conversation/helpers.py b/custom_components/extended_openai_conversation/tools_builtin.py similarity index 56% rename from custom_components/extended_openai_conversation/helpers.py rename to custom_components/extended_openai_conversation/tools_builtin.py index b39f67f3..b6793964 100644 --- a/custom_components/extended_openai_conversation/helpers.py +++ b/custom_components/extended_openai_conversation/tools_builtin.py @@ -6,11 +6,9 @@ import re import sqlite3 import time -from typing import Any +from typing import Any, TYPE_CHECKING from urllib import parse -from bs4 import BeautifulSoup - try: from openai import AsyncAzureOpenAI, AsyncOpenAI except Exception: # pragma: no cover @@ -19,6 +17,9 @@ import voluptuous as vol import yaml +if TYPE_CHECKING: + from bs4 import BeautifulSoup + from homeassistant.components import ( automation, conversation, @@ -32,8 +33,10 @@ from homeassistant.config import AUTOMATION_CONFIG_PATH from homeassistant.const import ( CONF_ATTRIBUTE, + CONF_HEADERS, CONF_METHOD, CONF_NAME, + CONF_PARAMS, CONF_PAYLOAD, CONF_RESOURCE, CONF_RESOURCE_TEMPLATE, @@ -65,6 +68,23 @@ AZURE_DOMAIN_PATTERN = r"\.(openai\.azure\.com|azure-api\.net)" +HISTORY_SUMMARY_LIMIT = 10 + +# REST helper mirrors https://www.home-assistant.io/integrations/rest/ defaults. +REST_ALLOWED_METHODS = {"GET", "POST", "PUT", "PATCH", "DELETE"} +REST_BODY_MAX_CHARS = 8192 +SCRAPE_SAFE_MAX_INDEX = 50 + + +def _lazy_import_bs4(): + try: + from bs4 import BeautifulSoup # type: ignore + except ImportError as exc: # pragma: no cover - optional dep + raise HomeAssistantError( + "beautifulsoup4 is required for the scrape tool but is not installed." + ) from exc + return BeautifulSoup + def get_function_executor(value: str): function_executor = FUNCTION_EXECUTORS.get(value) @@ -188,9 +208,16 @@ def to_arguments(self, arguments): def validate_entity_ids(self, hass: HomeAssistant, entity_ids, exposed_entities): if any(hass.states.get(entity_id) is None for entity_id in entity_ids): raise EntityNotFound(entity_ids) - exposed_entity_ids = map(lambda e: e["entity_id"], exposed_entities) - if not set(entity_ids).issubset(exposed_entity_ids): - raise EntityNotExposed(entity_ids) + exposed_entity_ids = {exposed["entity_id"] for exposed in exposed_entities} + missing = set(entity_ids) - exposed_entity_ids + if missing: + # Exposure follows Assist's 'conversation' assistant toggle (see + # homeassistant.components.homeassistant.exposed_entities.async_should_expose). + _LOGGER.debug( + "Exposure denied for assistant='conversation', entities=%s", + sorted(missing), + ) + raise EntityNotExposed(sorted(missing)) @abstractmethod async def execute( @@ -274,7 +301,71 @@ async def execute_service_single( raise CallServiceError(domain, service, service_data) if not hass.services.has_service(domain, service): raise ServiceNotFound(domain, service) + + # If explicit entity_id is provided, validate exposure as usual and drop area/device self.validate_entity_ids(hass, entity_id or [], exposed_entities) + if entity_id: + # Avoid bypass via extra selectors + service_data.pop("area_id", None) + service_data.pop("device_id", None) + + # If no entity_id but area_id/device_id is provided, resolve targets and enforce exposure + if not entity_id and (area_id or device_id): + from homeassistant.helpers import area_registry as ar, device_registry as dr, entity_registry as er # lazy import + from homeassistant.components.homeassistant.exposed_entities import async_should_expose + + def _as_list(x): + if x is None: + return [] + if isinstance(x, list): + return x + return [x] + + area_ids = [a for a in _as_list(area_id) if a] + device_ids = [d for d in _as_list(device_id) if d] + + ent_reg = er.async_get(hass) + dev_reg = dr.async_get(hass) + targets: set[str] = set() + + # Resolve by device_id + if device_ids: + for entry in ent_reg.entities.values(): + if entry.device_id and entry.device_id in device_ids: + targets.add(entry.entity_id) + + # Resolve by area_id (direct entity area or via device area) + if area_ids: + # Map device_id → area_id for quick lookups + device_area = {dev.id: dev.area_id for dev in dev_reg.devices.values()} + for entry in ent_reg.entities.values(): + if entry.area_id and entry.area_id in area_ids: + targets.add(entry.entity_id) + continue + if entry.device_id and device_area.get(entry.device_id) in area_ids: + targets.add(entry.entity_id) + + # Filter to currently loaded entities only (avoid phantom registry entries) + targets = {eid for eid in targets if hass.states.get(eid) is not None} + + hidden = [eid for eid in sorted(targets) if not async_should_expose(hass, "conversation", eid)] + if hidden: + _LOGGER.debug( + "Exposure denied for assistant='conversation', hidden targets=%s", + hidden, + ) + return {"error": f"Some targets are hidden from Assist: {', '.join(hidden)}"} + + if not targets: + return {"error": "No exposed targets found for the provided area/device"} + + if targets: + # Limit the actual call to exposed targets only + service_data = dict(service_data) + service_data["entity_id"] = list(sorted(targets)) + # Avoid passing area/device ids to the call to prevent bypass + service_data.pop("area_id", None) + service_data.pop("device_id", None) try: await hass.services.async_call( @@ -382,7 +473,30 @@ async def get_history( no_attributes, ) - return [[self.as_dict(item) for item in sublist] for sublist in result.values()] + summary: list[dict[str, Any]] = [] + for entity_id, states in result.items(): + samples: list[dict[str, Any]] = [] + trimmed = list(states)[-HISTORY_SUMMARY_LIMIT:] + for state in trimmed: + if isinstance(state, State): + samples.append( + { + "last_changed": state.last_changed.isoformat(), + "state": state.state, + "attributes": None if no_attributes else dict(state.attributes), + } + ) + elif isinstance(state, dict): + samples.append( + { + "last_changed": state.get("last_changed"), + "state": state.get("state"), + "attributes": None if no_attributes else state.get("attributes"), + } + ) + summary.append({"entity_id": entity_id, "samples": samples}) + + return summary async def get_energy( self, @@ -450,6 +564,14 @@ def __init__(self) -> None: """initialize script function""" super().__init__(SCRIPT_ENTITY_SCHEMA) + def to_arguments(self, arguments): + if "sequence" in arguments: + return super().to_arguments(arguments) + entity_id = arguments.get("entity_id") + if isinstance(entity_id, str) and entity_id.startswith("script."): + return {"type": arguments["type"], "entity_id": entity_id} + raise InvalidFunction("script") + async def execute( self, hass: HomeAssistant, @@ -458,19 +580,39 @@ async def execute( user_input: conversation.ConversationInput, exposed_entities, ): - script = Script( - hass, - function["sequence"], - "extended_openai_conversation", - DOMAIN, - running_description="[extended_openai_conversation] function", - logger=_LOGGER, - ) + sequence = function.get("sequence") or arguments.get("sequence") + if sequence: + if not isinstance(sequence, list): + raise HomeAssistantError("Script sequence must be a list of steps.") + script = Script( + hass, + sequence, + "extended_openai_conversation", + DOMAIN, + running_description="[extended_openai_conversation] function", + logger=_LOGGER, + ) + result = await script.async_run( + run_variables=arguments, context=user_input.context + ) + return result.variables.get("_function_result", "Success") - result = await script.async_run( - run_variables=arguments, context=user_input.context + entity_id = arguments.get("entity_id") + if isinstance(entity_id, str) and entity_id.startswith("script."): + domain, service = entity_id.split(".", 1) + # Execute script entity as documented in + # https://www.home-assistant.io/integrations/script/. + await hass.services.async_call( + domain, + service, + {"entity_id": entity_id}, + context=user_input.context, + ) + return "Success" + + raise HomeAssistantError( + "Script tool requires either a 'sequence' or script 'entity_id'." ) - return result.variables.get("_function_result", "Success") class TemplateFunctionExecutor(FunctionExecutor): @@ -481,10 +623,27 @@ def __init__(self) -> None: { vol.Required("value_template"): cv.template, vol.Optional("parse_result"): bool, + # Accept variables mapping for render context + vol.Optional("vars", default={}): dict, } ) ) + def to_arguments(self, arguments): + """Normalize friendly keys before schema validation. + + Accepts 'template' → maps to 'value_template' and preserves optional 'vars'. + Verified against HA template helper contract (cv.template + Template render). + """ + mapped = dict(arguments) + tmpl = mapped.get("value_template") + if tmpl is None and isinstance(mapped.get("template"), str): + mapped["value_template"] = mapped.pop("template") + # Ensure vars is a dict if provided + if "vars" in mapped and not isinstance(mapped["vars"], dict): + raise vol.Invalid("vars must be a mapping") + return self.data_schema(mapped) + async def execute( self, hass: HomeAssistant, @@ -493,9 +652,28 @@ async def execute( user_input: conversation.ConversationInput, exposed_entities, ): - return function["value_template"].async_render( - arguments, - parse_result=function.get("parse_result", False), + value_template: Template | None = function.get("value_template") or arguments.get("value_template") + if value_template is None: + template_str = arguments.get("template") + if not isinstance(template_str, str) or not template_str.strip(): + raise HomeAssistantError("Template tool requires a 'template' or 'value_template'") + value_template = Template(template_str, hass) + + parse_result = function.get("parse_result", arguments.get("parse_json", False)) + # Merge provided vars mapping (function-level or call-level) into the context + ctx_vars = {} + if isinstance(function.get("vars"), dict): + ctx_vars.update(function["vars"]) + if isinstance(arguments.get("vars"), dict): + ctx_vars.update(arguments["vars"]) + # Render using HA template environment (async, non-blocking). + # Pass all arguments as context plus optional 'vars'. + merged_context = dict(arguments) + if ctx_vars: + merged_context.update(ctx_vars) + return value_template.async_render( + merged_context, + parse_result=bool(parse_result), ) @@ -511,6 +689,56 @@ def __init__(self) -> None: ) ) + def to_arguments(self, arguments): + """Normalize friendly keys (url, method, headers, params, timeout, body). + + Map to REST data schema (resource, method, headers, params, timeout, payload). + Enforce http/https and request method allowlist including DELETE. + See HA REST integration docs for RESOURCE_SCHEMA fields. + """ + mapped = dict(arguments) + # url → resource + url = mapped.pop("url", mapped.get(CONF_RESOURCE)) + if url is None: + raise vol.Invalid("REST tool requires a 'url' (http/https)") + if not isinstance(url, str) or not url.lower().startswith(("http://", "https://")): + raise vol.Invalid("REST tool requires a 'url' (http/https)") + mapped[CONF_RESOURCE] = url + + # method allowlist (include DELETE) + method = str(mapped.pop("method", mapped.get(CONF_METHOD, "GET"))).upper() + # HA rest.RESOURCE_SCHEMA restricts methods to rest.const.METHODS (POST/GET) + # https://github.com/home-assistant/core/blob/dev/homeassistant/components/rest/schema.py + if method not in {"GET", "POST"}: + raise vol.Invalid("REST method not allowed") + mapped[CONF_METHOD] = method + + # headers + if "headers" in mapped: + if not isinstance(mapped["headers"], dict): + raise vol.Invalid("headers must be an object") + mapped[CONF_HEADERS] = mapped.pop("headers") + + # params + if "params" in mapped: + if not isinstance(mapped["params"], dict): + raise vol.Invalid("params must be an object") + mapped[CONF_PARAMS] = {str(k): str(v) for k, v in mapped["params"].items()} + mapped.pop("params", None) + + # timeout + if "timeout" in mapped: + try: + mapped[CONF_TIMEOUT] = int(mapped.pop("timeout")) + except Exception as exc: + raise vol.Invalid("timeout must be an integer") from exc + + # body → payload (cap also enforced in execute) + if "body" in mapped and CONF_PAYLOAD not in mapped: + mapped[CONF_PAYLOAD] = str(mapped.pop("body")) + + return self.data_schema(mapped) + async def execute( self, hass: HomeAssistant, @@ -519,7 +747,66 @@ async def execute( user_input: conversation.ConversationInput, exposed_entities, ): - config = function + config = dict(function) + + # Merge normalized arguments from to_arguments into config + for key in (CONF_RESOURCE, CONF_METHOD, CONF_HEADERS, CONF_PARAMS, CONF_TIMEOUT, CONF_VERIFY_SSL, CONF_PAYLOAD): + if key in arguments: + config[key] = arguments[key] + + request_url = arguments.get("url") or arguments.get(CONF_RESOURCE) + if request_url: + if not isinstance(request_url, str) or not request_url.lower().startswith( + ("http://", "https://") + ): + raise HomeAssistantError("REST tool requires an http(s) URL.") + config[CONF_RESOURCE] = request_url + if CONF_RESOURCE not in config: + raise HomeAssistantError("REST tool requires a 'url' argument.") + + method = str(arguments.get("method", config.get(CONF_METHOD, "GET"))).upper() + if method not in REST_ALLOWED_METHODS: + raise HomeAssistantError( + f"REST method '{method}' is not allowed; " + f"allowed methods: {', '.join(sorted(REST_ALLOWED_METHODS))}" + ) + config[CONF_METHOD] = method + + # Prefer normalized CONF_HEADERS set by to_arguments + headers = arguments.get(CONF_HEADERS) or arguments.get("headers") + if headers is not None: + if not isinstance(headers, dict): + raise HomeAssistantError("REST headers must be an object of strings.") + safe_headers: dict[str, str] = {str(k)[:64]: str(v)[:256] for k, v in headers.items()} + config[CONF_HEADERS] = safe_headers + + # params + if (CONF_PARAMS in arguments) or ("params" in arguments): + params = arguments.get(CONF_PARAMS, arguments.get("params")) + if not isinstance(params, dict): + raise HomeAssistantError("REST params must be an object of strings.") + config[CONF_PARAMS] = {str(k): str(v) for k, v in params.items()} + + if (payload := arguments.get("payload")) is not None: + payload_str = str(payload) + if len(payload_str) > REST_BODY_MAX_CHARS: + raise HomeAssistantError("REST payload exceeds 8KB limit.") + config[CONF_PAYLOAD] = payload_str + + # timeout + if (CONF_TIMEOUT in arguments) or ("timeout" in arguments): + try: + config[CONF_TIMEOUT] = int(arguments.get(CONF_TIMEOUT, arguments.get("timeout"))) + except Exception as exc: + raise HomeAssistantError("timeout must be an integer") from exc + else: + config.setdefault(CONF_TIMEOUT, min(15, config.get(CONF_TIMEOUT, 10))) + config.setdefault(CONF_VERIFY_SSL, True) + + # REST helper mirrors https://www.home-assistant.io/integrations/rest/ + # behaviour: build data coordinator and read text/JSON. + # Scrape helper aligns with https://www.home-assistant.io/integrations/scrape/ + # by delegating to the shared REST data coordinator. rest_data = _get_rest_data(hass, config, arguments) await rest_data.async_update() @@ -546,6 +833,43 @@ def __init__(self) -> None: ) ) + def to_arguments(self, arguments): + """Normalize friendly keys (url, select, attr, index) to COMBINED_SCHEMA. + + Friendly keys are mapped before validation so LLM calls don't fail early. + Verified against HA scrape integration (selector/attribute/index semantics). + """ + mapped = dict(arguments) + # url → resource + url = mapped.pop("url", mapped.get(CONF_RESOURCE)) + if url is None: + raise vol.Invalid("Scrape tool requires a 'url'") + if not isinstance(url, str) or not url.lower().startswith(("http://", "https://")): + raise vol.Invalid("Scrape tool requires a 'url' (http/https)") + mapped[CONF_RESOURCE] = url + + # Build sensor list from friendly keys if absent + if "sensor" not in mapped: + select = mapped.pop("select", None) + if not isinstance(select, str) or not select.strip(): + raise vol.Invalid("Scrape tool requires a 'select' CSS selector") + idx = mapped.pop("index", 0) + try: + idx = int(idx) + except Exception as exc: + raise vol.Invalid("index must be an integer") from exc + idx = max(0, min(idx, SCRAPE_SAFE_MAX_INDEX)) + sensor_cfg = { + scrape.const.CONF_SELECT: select.strip(), + scrape.const.CONF_INDEX: idx, + } + attr = mapped.pop("attr", mapped.pop("attribute", None)) + if attr is not None: + sensor_cfg[CONF_ATTRIBUTE] = str(attr) + mapped["sensor"] = [sensor_cfg] + + return self.data_schema(mapped) + async def execute( self, hass: HomeAssistant, @@ -554,7 +878,59 @@ async def execute( user_input: conversation.ConversationInput, exposed_entities, ): - config = function + _lazy_import_bs4() + config = dict(function) + + request_url = arguments.get("url") or arguments.get(CONF_RESOURCE) + if request_url: + if not isinstance(request_url, str) or not request_url.lower().startswith( + ("http://", "https://") + ): + raise HomeAssistantError("Scrape tool requires an http(s) URL.") + config[CONF_RESOURCE] = request_url + if CONF_RESOURCE not in config: + raise HomeAssistantError("Scrape tool requires a 'url' argument.") + + sensor_configs = [dict(sensor) for sensor in config.get("sensor", [])] + # Prefer normalized sensors from to_arguments if present + if not sensor_configs and isinstance(arguments.get("sensor"), list): + sensor_configs = [dict(s) for s in arguments["sensor"]] + def _coerce_index(value: Any) -> int: + if value in (None, ""): + return 0 + try: + parsed = int(value) + except (TypeError, ValueError) as exc: + raise HomeAssistantError("Scrape index must be an integer.") from exc + return max(0, min(parsed, SCRAPE_SAFE_MAX_INDEX)) + + if not sensor_configs: + select = arguments.get("select") + if not isinstance(select, str) or not select.strip(): + raise HomeAssistantError("Scrape tool requires a 'select' CSS selector.") + sensor_config: dict[str, Any] = { + scrape.const.CONF_SELECT: select.strip(), + scrape.const.CONF_INDEX: _coerce_index(arguments.get("index")), + } + if attribute := arguments.get("attribute"): + sensor_config[CONF_ATTRIBUTE] = str(attribute) + sensor_configs = [sensor_config] + else: + select = arguments.get("select") + index = arguments.get("index") + attribute = arguments.get("attribute") + target_sensor = sensor_configs[0] + if select: + target_sensor[scrape.const.CONF_SELECT] = str(select) + if index is not None: + target_sensor[scrape.const.CONF_INDEX] = _coerce_index(index) + if attribute is not None: + if attribute == "": + target_sensor.pop(CONF_ATTRIBUTE, None) + else: + target_sensor[CONF_ATTRIBUTE] = str(attribute) + config["sensor"] = sensor_configs + rest_data = _get_rest_data(hass, config, arguments) coordinator = scrape.coordinator.ScrapeCoordinator( hass, @@ -586,7 +962,7 @@ async def execute( def _async_update_from_rest_data( self, - data: BeautifulSoup, + data: Any, sensor_config: dict[str, Any], arguments: dict[str, Any], ) -> None: @@ -601,7 +977,7 @@ def _async_update_from_rest_data( return value - def _extract_value(self, data: BeautifulSoup, sensor_config: dict[str, Any]) -> Any: + def _extract_value(self, data: Any, sensor_config: dict[str, Any]) -> Any: """Parse the html extraction in the executor.""" value: str | list[str] | None select = sensor_config[scrape.const.CONF_SELECT] @@ -658,7 +1034,11 @@ async def execute( exposed_entities, ): config = function - sequence = config["sequence"] + sequence = config.get("sequence") or arguments.get("sequence") + if not isinstance(sequence, list) or not sequence: + raise HomeAssistantError( + "Composite tool requires a non-empty 'sequence' list." + ) for executor_config in sequence: function_executor = get_function_executor(executor_config["type"]) diff --git a/custom_components/extended_openai_conversation/tools_mcp_bridge.py b/custom_components/extended_openai_conversation/tools_mcp_bridge.py new file mode 100644 index 00000000..3311daee --- /dev/null +++ b/custom_components/extended_openai_conversation/tools_mcp_bridge.py @@ -0,0 +1,92 @@ +"""Optional MCP bridge for exposing external tools.""" + +from __future__ import annotations + +import importlib +import logging +from typing import Any, Iterable + +from .const import ( + CONF_ENABLE_MCP, + CONF_MCP_MAX_PAYLOAD, + CONF_MCP_TIMEOUT, + DEFAULT_ENABLE_MCP, + DEFAULT_MCP_MAX_PAYLOAD, + DEFAULT_MCP_TIMEOUT, +) + +LOGGER = logging.getLogger(__name__) + + +class MCPBridge: + """Lazy MCP client facade.""" + + def __init__(self, options: dict[str, Any]) -> None: + self._enabled = options.get(CONF_ENABLE_MCP, DEFAULT_ENABLE_MCP) + self._timeout = options.get(CONF_MCP_TIMEOUT, DEFAULT_MCP_TIMEOUT) + self._payload_cap = options.get(CONF_MCP_MAX_PAYLOAD, DEFAULT_MCP_MAX_PAYLOAD) + self._client_module = None + self._load_error: str | None = None + + if not self._enabled: + return + + try: + self._client_module = importlib.import_module("mcp") + except Exception as err: # pragma: no cover - optional dependency + self._load_error = str(err) + LOGGER.debug("MCP bridge disabled: %s", err) + self._enabled = False + + @property + def available(self) -> bool: + return self._enabled and self._client_module is not None + + @property + def payload_cap(self) -> int: + return self._payload_cap + + @property + def timeout(self) -> float: + return self._timeout + + def describe_tools(self) -> list[dict[str, Any]]: + """Return MCP tool specs ready for registration.""" + + if not self.available: + return [] + + # Full MCP discovery is not yet implemented. This placeholder keeps + # the integration safe until the optional dependency is configured. + LOGGER.info( + "MCP support is enabled but automatic MCP tool discovery is not " + "implemented in this build. No MCP tools will be exposed." + ) + return [] + + async def async_call_tool( + self, + name: str, + arguments: dict[str, Any], + ) -> dict[str, Any]: + """Execute an MCP tool call. Currently returns informative error.""" + + if not self.available: + return { + "error": "mcp_unavailable", + "message": self._load_error + or "MCP client library not installed. Install the 'mcp' package to enable MCP tools.", + } + + return { + "error": "mcp_not_implemented", + "message": ( + "MCP support is enabled but runtime execution is not yet implemented." + ), + } + + +def create_bridge(options: dict[str, Any]) -> MCPBridge: + """Factory used by the orchestrator.""" + + return MCPBridge(options) diff --git a/custom_components/extended_openai_conversation/tools_orchestrator.py b/custom_components/extended_openai_conversation/tools_orchestrator.py new file mode 100644 index 00000000..aeb2f2da --- /dev/null +++ b/custom_components/extended_openai_conversation/tools_orchestrator.py @@ -0,0 +1,421 @@ +"""Orchestrates tool execution for both Chat Completions and Responses API.""" + +from __future__ import annotations + +import asyncio +import copy +import json +import logging +import time +from typing import Any + +from homeassistant.components.conversation.chat_log import ChatLog, ToolResultContent +from homeassistant.components.homeassistant.exposed_entities import async_should_expose +from homeassistant.core import HomeAssistant +from homeassistant.exceptions import HomeAssistantError + +from .const import ( + CONF_FUNCTIONS, + CONF_MAX_TOOL_CALLS, + CONF_MAX_TOOL_CHAIN, + CONF_TOOL_TIMEOUT, + CONF_TOOL_MAX_OUTPUT_CHARS, + CONF_MEMORY_ENABLED, + DEFAULT_FUNCTIONS, + DEFAULT_MAX_TOOL_CALLS, + DEFAULT_MAX_TOOL_CHAIN, + DEFAULT_TOOL_TIMEOUT, + DEFAULT_TOOL_MAX_OUTPUT_CHARS, + DEFAULT_MEMORY_ENABLED, +) +from .exceptions import InvalidFunction, FunctionNotFound, NativeNotFound +from .memory_tools import ( + MEMORY_TOOL_SPECS, + dispatch_memory_tool, + get_memory_service_config, +) +from .tools_builtin import FUNCTION_EXECUTORS, get_function_executor +from .tools_mcp_bridge import MCPBridge, create_bridge +from .tools_web_search import ( + build_responses_web_search_tool, + configure_chat_completion_web_search, +) + +LOGGER = logging.getLogger(__name__) + +class ToolExecutionError(Exception): + """Raised when a tool fails to execute cleanly.""" + + +class ToolsOrchestrator: + """Coordinate execution of built-in, memory, and optional MCP tools.""" + + def __init__( + self, + hass: HomeAssistant, + *, + options: dict[str, Any], + chat_log: ChatLog | None, + agent_id: str, + ) -> None: + self.hass = hass + self.options = options + self.chat_log = chat_log + self.agent_id = agent_id + + self._max_calls: int = options.get(CONF_MAX_TOOL_CALLS, DEFAULT_MAX_TOOL_CALLS) + self._max_chain_depth: int = options.get( + CONF_MAX_TOOL_CHAIN, DEFAULT_MAX_TOOL_CHAIN + ) + self._call_timeout: float = float( + options.get(CONF_TOOL_TIMEOUT, DEFAULT_TOOL_TIMEOUT) + ) + self._result_cap: int = int( + options.get(CONF_TOOL_MAX_OUTPUT_CHARS, DEFAULT_TOOL_MAX_OUTPUT_CHARS) + ) + + self._call_count = 0 + self._chain_depth = 0 + self._started = time.monotonic() + self._exposed_entities: list[dict[str, Any]] | None = None + self._exposed_entities_expires: float | None = None + + self._tool_specs: list[dict[str, Any]] = [] + self._runtime_map: dict[str, dict[str, Any]] = {} + + self._web_search_tool_type: str | None = None + self._web_search_fallback_type: str | None = None + + self._load_builtin_functions(options.get(CONF_FUNCTIONS, DEFAULT_FUNCTIONS)) + self._memory_enabled = options.get(CONF_MEMORY_ENABLED, DEFAULT_MEMORY_ENABLED) + self._memory_config = ( + get_memory_service_config(options) if self._memory_enabled else None + ) + if self._memory_enabled: + self._register_memory_tools() + + self._mcp_bridge: MCPBridge = create_bridge(options) + if self._mcp_bridge.available: + self._register_mcp_tools() + + # --------------------------------------------------------------------- + # Tool specification helpers + # --------------------------------------------------------------------- + def _load_builtin_functions(self, functions: list[dict[str, Any]] | None) -> None: + entries: list[dict[str, Any]] = copy.deepcopy(DEFAULT_FUNCTIONS) + if functions: + entries.extend(functions) + + for entry in entries: + if not isinstance(entry, dict): + continue + spec = entry.get("spec") + runtime = entry.get("function") + if not isinstance(spec, dict) or not isinstance(runtime, dict): + continue + name = spec.get("name") + if not name: + continue + self._tool_specs = [ + existing for existing in self._tool_specs if existing.get("name") != name + ] + self._tool_specs.append(dict(spec)) + self._runtime_map[name] = dict(runtime) + + def _register_memory_tools(self) -> None: + for spec in MEMORY_TOOL_SPECS: + name = spec.get("name") + if not name: + continue + self._tool_specs.append(dict(spec)) + self._runtime_map[name] = {"type": "memory", "name": name} + + def _register_mcp_tools(self) -> None: + for spec in self._mcp_bridge.describe_tools(): + name = spec.get("name") + if not name: + continue + self._tool_specs.append(spec) + self._runtime_map[name] = {"type": "mcp", "name": name} + + # ------------------------------------------------------------------ + # Tool definitions exposed to OpenAI APIs + # ------------------------------------------------------------------ + def conversation_tools_for_chat(self) -> list[dict[str, Any]]: + return [ + {"type": "function", "function": dict(spec)} for spec in self._tool_specs + ] + + def conversation_tools_for_responses( + self, + hass: HomeAssistant, + *, + model: str | None, + tool_type_override: str | None = None, + ) -> list[dict[str, Any]]: + tools = [ + {"type": "function", "function": dict(spec)} + for spec in self._tool_specs + ] + self._web_search_tool_type = None + self._web_search_fallback_type = None + if tool_spec := build_responses_web_search_tool( + hass, + self.options, + model=model, + tool_type_override=tool_type_override, + ): + payload, fallback_type = tool_spec + self._web_search_tool_type = payload.get("type") + self._web_search_fallback_type = fallback_type + tools.append(payload) + return tools + + def configure_chat_web_search(self, model: str | None) -> None: + configure_chat_completion_web_search(options=self.options, model=model) + + # ------------------------------------------------------------------ + # Execution helpers + # ------------------------------------------------------------------ + def reset(self) -> None: + self._call_count = 0 + self._chain_depth = 0 + self._started = time.monotonic() + self._exposed_entities = None + self._exposed_entities_expires = None + self._web_search_tool_type = None + self._web_search_fallback_type = None + # Note: Options save/reconfigure triggers a reload of the entity which + # constructs a new orchestrator. This reset (and the reload) ensure the + # exposure cache reflects current UI settings immediately. + + @property + def max_chain_depth(self) -> int: + return self._max_chain_depth + + def _ensure_call_budget(self) -> None: + if self._max_calls and self._call_count >= self._max_calls: + raise ToolExecutionError("Maximum tool calls exceeded") + + def _ensure_time_budget(self) -> None: + if self._call_timeout <= 0: + return + max_runtime = self._call_timeout * max(1, self._max_chain_depth) + elapsed = time.monotonic() - self._started + if elapsed > max_runtime: + LOGGER.debug( + "Tool chain time budget exceeded after %.2fs (limit %.2fs)", + elapsed, + max_runtime, + ) + raise ToolExecutionError("Tool chain time budget exceeded") + + def _get_exposed_entities(self) -> list[dict[str, Any]]: + now = time.monotonic() + if ( + self._exposed_entities is not None + and self._exposed_entities_expires is not None + and now < self._exposed_entities_expires + ): + return self._exposed_entities + + exposed: list[dict[str, Any]] = [] + for state in self.hass.states.async_all(): + # Assist exposure uses the 'conversation' assistant id; see + # homeassistant.components.homeassistant.exposed_entities.async_should_expose. + if async_should_expose(self.hass, "conversation", state.entity_id): + exposed.append( + { + "entity_id": state.entity_id, + "name": state.name, + "state": state.state, + } + ) + self._exposed_entities = exposed + # Expose settings may change from the UI at runtime; refresh every 5 minutes. + self._exposed_entities_expires = now + 300 + return exposed + + def _stringify_result(self, result: Any) -> str: + if isinstance(result, str): + text = result + else: + try: + text = json.dumps(result, default=str, ensure_ascii=False) + except TypeError: + text = str(result) + + text = text.strip() + if self._result_cap and len(text) > self._result_cap: + truncated = text[: self._result_cap - 16].rstrip() + text = f"{truncated}... (truncated)" + return text + + async def execute_tool_call( + self, + *, + name: str, + arguments: str | dict[str, Any] | None, + user_input, + call_id: str | None, + ) -> str: + self._ensure_call_budget() + self._ensure_time_budget() + self._call_count += 1 + runtime = self._runtime_map.get(name) + if runtime is None: + raise ToolExecutionError(f"Unknown tool '{name}'") + + try: + payload = self._prepare_arguments(runtime, arguments) + except (json.JSONDecodeError, InvalidFunction) as err: + raise ToolExecutionError(f"Invalid arguments for {name}: {err}") from err + + LOGGER.debug( + "Tool %s starting (call_id=%s, invocation=%s/%s)", + name, + call_id or "", + self._call_count, + self._max_calls if self._max_calls else "unbounded", + ) + started = time.perf_counter() + try: + result = await asyncio.wait_for( + self._invoke(runtime, payload, user_input), + timeout=self._call_timeout, + ) + except asyncio.TimeoutError as err: + elapsed_ms = (time.perf_counter() - started) * 1000 + message = f"Tool {name} timed out" + self._record_tool_result( + call_id=call_id or name or "tool", + tool_name=name, + text=message, + elapsed_ms=elapsed_ms, + ) + raise ToolExecutionError(message) from err + except (HomeAssistantError, ToolExecutionError) as err: + elapsed_ms = (time.perf_counter() - started) * 1000 + message = str(err) or f"Tool {name} failed" + self._record_tool_result( + call_id=call_id or name or "tool", + tool_name=name, + text=message, + elapsed_ms=elapsed_ms, + ) + raise ToolExecutionError(message) from err + except Exception as err: # pragma: no cover - defensive guard + elapsed_ms = (time.perf_counter() - started) * 1000 + message = f"Tool {name} failed: {err}" + self._record_tool_result( + call_id=call_id or name or "tool", + tool_name=name, + text=message, + elapsed_ms=elapsed_ms, + ) + raise ToolExecutionError(message) from err + + elapsed_ms = (time.perf_counter() - started) * 1000 + text = self._stringify_result(result) + LOGGER.debug("Tool %s completed in %.1f ms -> %s", name, elapsed_ms, text) + self._record_tool_result( + call_id=call_id or name or "tool", + tool_name=name, + text=text, + elapsed_ms=elapsed_ms, + ) + return text + + def _prepare_arguments( + self, runtime: dict[str, Any], arguments: str | dict[str, Any] | None + ) -> dict[str, Any]: + if isinstance(arguments, str) and arguments.strip(): + data = json.loads(arguments) + elif isinstance(arguments, dict): + data = dict(arguments) + else: + data = {} + + runtime_type = runtime.get("type") + if runtime_type in {"memory", "mcp"}: + return data + + executor = get_function_executor(runtime_type) + validated = executor.to_arguments({"type": runtime_type, **data}) + validated.pop("type", None) + return validated + + async def _invoke( + self, runtime: dict[str, Any], arguments: dict[str, Any], user_input + ) -> Any: + runtime_type = runtime.get("type") + if runtime_type == "memory": + if not self._memory_config: + raise ToolExecutionError("Memory service not configured") + name = runtime.get("name") + return await dispatch_memory_tool( + self.hass, self._memory_config, name, arguments + ) + + if runtime_type == "mcp": + return await self._mcp_bridge.async_call_tool( + runtime.get("name"), arguments + ) + + executor = FUNCTION_EXECUTORS.get(runtime_type) + if executor is None: + raise ToolExecutionError(f"Unsupported tool type {runtime_type}") + + try: + return await executor.execute( + self.hass, + runtime, + arguments, + user_input, + self._get_exposed_entities(), + ) + except NativeNotFound as err: + raise ToolExecutionError(str(err)) from err + except FunctionNotFound as err: + raise ToolExecutionError(str(err)) from err + + def _record_tool_result( + self, + *, + call_id: str, + tool_name: str, + text: str, + elapsed_ms: float, + ) -> None: + if not self.chat_log: + return + + payload: dict[str, Any] = { + "content": text, + "elapsed_ms": round(elapsed_ms, 2), + } + try: + self.chat_log.async_add_assistant_content_without_tools( + ToolResultContent( + agent_id=self.agent_id, + tool_call_id=call_id, + tool_name=tool_name, + tool_result=payload, + ) + ) + except Exception as err: # pragma: no cover - defensive guard + LOGGER.debug("Unable to append tool result to chat log: %s", err) + + @property + def web_search_tool_type(self) -> str | None: + return self._web_search_tool_type + + @property + def web_search_fallback_type(self) -> str | None: + return self._web_search_fallback_type + + def update_web_search_tool_type(self, new_type: str | None) -> None: + if new_type: + self._web_search_tool_type = new_type + # Only allow a single fallback step to avoid loops. + self._web_search_fallback_type = None diff --git a/custom_components/extended_openai_conversation/tools_web_search.py b/custom_components/extended_openai_conversation/tools_web_search.py new file mode 100644 index 00000000..83382adf --- /dev/null +++ b/custom_components/extended_openai_conversation/tools_web_search.py @@ -0,0 +1,145 @@ +"""Hosted web search helpers for Extended OpenAI Conversation.""" + +from __future__ import annotations + +import logging +from typing import Any, Dict, NamedTuple + +from homeassistant.core import HomeAssistant + +from .const import ( + CONF_ENABLE_WEB_SEARCH, + CONF_WEB_SEARCH_CONTEXT_SIZE, + CONF_INCLUDE_HOME_LOCATION, + DEFAULT_ENABLE_WEB_SEARCH, + DEFAULT_WEB_SEARCH_CONTEXT_SIZE, +) +from .model_capabilities import detect_model_capabilities + +LOGGER = logging.getLogger(__name__) + + +class WebSearchToolSpec(NamedTuple): + payload: Dict[str, Any] + fallback_type: str | None + + +_CONTEXT_ALIASES = { + "small": "low", + "low": "low", + "medium": "medium", + "med": "medium", + "mid": "medium", + "large": "high", + "high": "high", +} + + +def _normalize_context_size(value: Any) -> str: + """Map user option to OpenAI context size token.""" + + if isinstance(value, str): + key = value.strip().lower() + if key in _CONTEXT_ALIASES: + return _CONTEXT_ALIASES[key] + + if isinstance(value, (int, float)): + if value <= 384: + return "low" + if value <= 768: + return "medium" + return "high" + + return "medium" + + +def _build_user_location(hass: HomeAssistant) -> Dict[str, str] | None: + """Return approximate location payload if HA config provides details.""" + + location: Dict[str, str] = {"type": "approximate"} + + city = getattr(hass.config, "location_name", None) + if city and city.lower() not in {"home", "house"}: + location["city"] = city + + region = getattr(hass.config, "state", None) + if region: + location["region"] = region + + country = getattr(hass.config, "country", None) + if country: + location["country"] = country + + timezone = getattr(hass.config, "time_zone", None) + if timezone: + location["timezone"] = timezone + + if len(location) > 1: + return location + return None + + +def _model_supports_hosted_search(model: str | None) -> bool: + if not model: + return False + caps = detect_model_capabilities(model) + return caps.is_reasoning + + +def build_responses_web_search_tool( + hass: HomeAssistant, + options: dict[str, Any], + *, + model: str | None, + tool_type_override: str | None = None, +) -> WebSearchToolSpec | None: + """Return Responses API tool payload for hosted web search if enabled.""" + + if not options.get(CONF_ENABLE_WEB_SEARCH, DEFAULT_ENABLE_WEB_SEARCH): + return None + if not _model_supports_hosted_search(model): + LOGGER.debug( + "Web search disabled for model=%s (unsupported)", + model or "", + ) + return None + + context_size = _normalize_context_size( + options.get(CONF_WEB_SEARCH_CONTEXT_SIZE, DEFAULT_WEB_SEARCH_CONTEXT_SIZE) + ) + tool_type = tool_type_override or "web_search" + # OpenAI may expose the legacy 'web_search_preview' type for older tenants; + # prefer 'web_search' and fall back if the API rejects it (see + # https://platform.openai.com/docs/assistants/tools/web-search). + fallback_type = None if tool_type != "web_search" else "web_search_preview" + tool: Dict[str, Any] = { + "type": tool_type, + "search_context_size": context_size, + } + + if options.get(CONF_INCLUDE_HOME_LOCATION): + if location := _build_user_location(hass): + tool["user_location"] = location + + LOGGER.debug( + "Hosted web search enabled for model=%s using tool type '%s'.", + model or "", + tool_type, + ) + return WebSearchToolSpec(tool, fallback_type) + + +def configure_chat_completion_web_search( + *, + options: dict[str, Any], + model: str | None, +) -> None: + """Chat Completions do not yet expose hosted web search; log and skip.""" + + if not options.get(CONF_ENABLE_WEB_SEARCH, DEFAULT_ENABLE_WEB_SEARCH): + return + + LOGGER.debug( + "Web search disabled for model=%s (unsupported via Chat Completions)", + model or "", + ) diff --git a/custom_components/extended_openai_conversation/translations/en.json b/custom_components/extended_openai_conversation/translations/en.json index 56ec6132..5b674c0e 100644 --- a/custom_components/extended_openai_conversation/translations/en.json +++ b/custom_components/extended_openai_conversation/translations/en.json @@ -38,9 +38,29 @@ "temperature": "Temperature (chat models only)", "top_p": "Top‑p (chat models only)", "max_tokens": "Max tokens", - "prompt": "System prompt" + "prompt": "System prompt", + "functions_yaml": "Functions YAML", + "max_tool_calls": "Max tool calls per turn", + "max_tool_chain": "Max tool iterations", + "tool_timeout": "Tool timeout (seconds)", + "tool_max_output_chars": "Tool output cap (characters)", + "enable_web_search": "Enable hosted web search", + "web_search_context_size": "Web search context size", + "include_home_location": "Share approximate Home Assistant location", + "enable_mcp": "Enable MCP bridge", + "mcp_timeout": "MCP tool timeout (seconds)", + "mcp_max_payload": "MCP payload cap (bytes)" + }, + "error": { + "invalid_functions_yaml": "Functions YAML is invalid. Provide a list of specs with 'spec' and 'function'." } } } + }, + "errors": { + "rest_url_required": "REST tool requires a 'url' (http/https)", + "scrape_requirements": "Scrape tool requires a 'url' and 'select'", + "template_required": "Template tool requires a 'template' or 'value_template'", + "exposure_denied": "Some targets are hidden from Assist: {entities}" } } diff --git a/docs/PLAN_TOOLBOX.md b/docs/PLAN_TOOLBOX.md new file mode 100644 index 00000000..79c97939 --- /dev/null +++ b/docs/PLAN_TOOLBOX.md @@ -0,0 +1,32 @@ +## Toolbox & Options Parity Plan + +### Scope Highlights +- Build a unified tools orchestrator ensuring identical behavior for chat vs reasoning paths, including depth/time limits and chat-log breadcrumbs. +- Implement built-in tools (`execute_service`, `add_automation`, `get_history`, `rest`, `scrape`, `composite`, `script`, `template`) with entity exposure enforcement, payload caps, and lazy imports (BeautifulSoup optional). +- Re-introduce Functions editor in Options flow with strong YAML validation, plus toggles for hosted web search and MCP bridging when available. +- Extend conversation pipeline to honor tool continuations for Responses API (`function_call`/`function_call_output`) and classic Chat Completions tool loops. +- Add hosted web search adapter with context sizing, location enrichment, and graceful no-op on unsupported chat models/backend. +- Optionally bridge MCP tools (namespaced) via lazy import, honoring timeouts and payload safeguards without impacting baseline when unavailable. + +### Key Files / Modules +- `custom_components/extended_openai_conversation/`: `tools_orchestrator.py`, `tools_builtin.py`, `tools_web_search.py`, `tools_mcp_bridge.py`, `conversation.py`, `responses_adapter.py`, `config_flow.py`, `const.py`, `openai_support.py`, `services.py`, `services.yaml`. +- `manifest.json`, `translations/en.json`, `strings.json` (if legacy strings still referenced), `README.md`, `docs/` (this plan + potential future notes). + +### Risks & Mitigations +- **Recursive tool loops**: enforce configurable depth/time limits and emit safe errors when exceeded. +- **Entity access control drift**: centralize exposure checks using HA permissions helpers; add clear user-facing error strings. +- **YAML validation regressions**: wrap parsing in try/except with schema verification and surface translations for misconfiguration. +- **Optional deps (bs4, MCP)**: lazy import and fall back with descriptive errors; include conditional manifest dependency for BeautifulSoup. +- **Hosted search availability mismatch**: introspect model/backend support; log info-level skip rather than raising. +- **Concurrency/event loop**: ensure all I/O uses async/httpx via HA shared client; offload blocking work with executor where required. + +### Manual Validation Checklist +- Base conversation flow returns `ConversationResult` with language + response_type + `continue_conversation`. +- `execute_service` respects exposed entities and denies others with clear messaging. +- `get_history` returns bounded, summarized results. +- `rest` and `scrape` apply timeouts/size caps; `scrape` gracefully fails when `beautifulsoup4` missing. +- `composite` chains calls, enforces loop depth/time caps. +- Web search succeeds on reasoning model and no-ops with log on unsupported chat path; location payload included when enabled. +- MCP bridge disabled by default; when enabled with server it exposes namespaced tools and respects limits. +- Functions editor rejects malformed YAML without crashing Options flow. +- `query_image` service registered and uses consistent credential sourcing.