feat: SmartHopper 2.0.0 - Google Gemini Provider, Batch API, Vision Input, and Breaking Changes#420
feat: SmartHopper 2.0.0 - Google Gemini Provider, Batch API, Vision Input, and Breaking Changes#420
Conversation
…xt2json tool - Add `text2json` AI tool for generating structured JSON from prompts conforming to JSON Schema - Add `AIText2JsonComponent` in `SmartHopper > JSON` category with Prompt, Instructions, and Schema inputs - Add JSON utility components requiring no AI: - `JsonSchemaComponent`: builds JSON Schema from property definitions with dot-notation nesting support - `JsonObjectComponent`: creates JSON objects from key-value pairs
…ts-toolkit/SmartHopper into feature/2.0.0-text2json
…mposition - Add `JsonSchemaPropComponent` (`JsonSchemaProp`): builds scalar property definitions from Name, Type, and Description inputs - Add `JsonSchemaPropObjectComponent` (`JsonSchemaPropObj`): builds object properties by prefixing sub-properties with dot-notation - Add `JsonSchemaPropArrayComponent` (`JsonSchemaPropArr`): builds array properties with configurable Items Type using `array[itemsType]` encoding - Update `JsonSchemaComponent`
…t reconstruction - Add `OnBatchCompleted` override to decode batch results and reconstruct output trees - Add batch submission calls in worker `DoWorkAsync` after processing all items - Add reconstructed tree handling in worker `SetOutput` to use batch results when available - Implement provider-based result decoding using `AIInteractionText` extraction - Add batch support to `AIImg2TextComponent`, `AIText2JsonComponent`, `AIFile
… prevent stale parameter usage
…nent When a batch job completes but individual items fail (e.g. OpenAI `BadRequest` due to `max_tokens` too large), the error was silently discarded and the component showed "Done".
…ns via img2text
- **AIFile2MdComponent**: File conversion and image extraction now run locally via `file2md` with `describeImages=false`. Each extracted image is then described via `CallAiToolAsync("img2text", ...)` which is batch-interceptable. `OnBatchCompleted` reassembles final markdown from locally-stored per-file context and batch image description results using `ImageSentinelContext` dictionary.
…re queuing - Add `IsValid()` validation call on batch requests before adding to queue - Surface validation warnings to component using `SurfaceMessagesFromReturn` - Surface validation errors but allow queuing to proceed (non-blocking) - Add debug logging for validation message counts
…ding images section - Add `InsertImagePlaceholders` to inject `[image N]` placeholders into markdown during file2md conversion - Add `ImageIndex` property to `ImageSentinelContext` for 1-based placeholder tracking - Replace image section assembly with placeholder substitution in both batch and non-batch modes - Add `_sentinelContextsInitialized` flag to prevent clearing sentinel contexts during batch polling re-runs
… days - Update milestone management guide to reflect 15-day waiting period between beta and rc releases - Update GitHub Actions default `days-lookback` parameter from 30 to 15 days - Update promotion PR body template to reference 15-day period - Update release workflow documentation to reflect 15-day requirements for release age and last closed issue - Update release-promotion workflow comments and validation table to show 15-day minimum
…ng brace in AIFile2MdComponent - Add missing closing brace in `AIFile2MdComponent.OnBatchCompleted` after markdown assembly - Consolidate warning and error validation message handling into single loop in `AIStatefulAsyncComponentBase` - Use `AddRuntimeMessage(severity, origin, message)` instead of passing message object directly - Update debug logging to reflect combined warning/error count
…nd skip output during active batch - Move AIReturnSnapshot and batch state clearing from BeforeSolveInstance to new OnEnteringProcessingState hook - Remove metricsInitializedForRun flag in favor of state transition-based clearing - Add OnEnteringProcessingState virtual method to StatefulComponentBase called during Processing state entry - Skip SetMetricsOutput in OnSolveInstancePostSolve when batch submission is still active
… convention - Rename `WebToMdComponent` to `Web2MdComponent` and update nickname from "WebToMd" to "Web2Md" - Rename icon resource from `webtomd` to `Web2Md` - Rename `WebToMdAsync` method to `Web2MdAsync` in web2md AI tool - Update CHANGELOG.md and DEV.md references from WebToMdComponent to Web2MdComponent - Update debug logging prefix from "WebToMd" to "Web2Md"
…alongside McNeel and Ladybug forum support - Add generic `DiscourseSearch`, `DiscoursePostGet`, `DiscoursePostOpen`, `DiscoursePostDeconstruct` components - Add generic `AIDiscoursePostSummarize` and `AIDiscourseTopicSummarize` components
…ings instead of SurfaceMessagesFromReturn
…e-line format in DiscourseToolsBase - Replace verbatim multi-line JSON schema strings with single-line interpolated strings - Rename `baseUrlSchemaProperty` to `baseUrlProperty` for consistency - Simplify schema formatting while maintaining identical structure and functionality
…ng with improved logging - Add try-catch block around CallAiToolAsync to handle exceptions during batch item processing - Add null check for toolResult with warning message when tool returns null - Add error messages with batch item index when exceptions occur - Update debug logging to use "batch item" terminology instead of "index" for consistency - Add empty string outputs for failed batch items to maintain output alignment
…centralizing timeout configuration - Add `CreateBatchHttpClient()` protected helper method to `AIProvider` base class with 5-minute default timeout (300 seconds) - Refactor `OpenAIProvider`, `AnthropicProvider`, and `MistralAIProvider` to use centralized helper instead of default HttpClient - Add timeout clamping (1-600 seconds) and debug logging for batch HTTP client creation - Update CHANGELOG.md to document batch processing
…with configurable UI controls - Add `HttpTimeoutSeconds` (default 120s) and `BatchHttpTimeoutSeconds` (default 300s) global settings in ProvidersSettingsPage under "Network Settings" section - Add NumericStepper controls (1-600 seconds range) with descriptive labels and help text for both timeout types - Update `AIProvider.CreateBatchHttpClient()` to read `BatchHttpTimeoutSeconds` from provider settings instead of hardcoded 300s default
…per-file storage with ordered image slots - Replace `ImageSentinelContext` with `FileBatchContext` containing base markdown and ordered `ImageSlot` list - Rename `_sentinelContexts` to `_fileContexts` and key by representative sentinel ID (first image) instead of per-image - Add `ImageSlot` class to store per-image metadata (index, sentinel ID, mode, context, MIME type, base64)
…stead of per-provider settings
- Update `CreateBatchHttpClient()` to read `BatchHttpTimeoutSeconds` from global settings instead of provider-specific settings
- Update `CreateHttpClient()` to read `HttpTimeoutSeconds` from global settings instead of provider-specific settings
- Use `SmartHopperSettings.Instance.GetSetting("Global", ...)` for both timeout configurations
- Maintain same default values (120s for HTTP, 300s for batch
…nel tree output during active batch operations - Add debug logging throughout File2Md and file2md batch processing to track tool execution flow - Override `RestorePersistentOutputs()` in AIStatefulAsyncComponentBase to skip sentinel tree output during active batch submission - Update AIFile2MdComponent to extract image descriptions from AIInteractionToolResult instead of AIInteractionText
…ription extraction across execute and batch paths - Add `Write`/`Read` methods to serialize `_fileContexts` (base markdown + image slot metadata) so batch results survive Grasshopper file save/reload - Add `_batchContextLost` flag to prevent `GatherInput` from resetting `_fileContextsInitialized` during active batch, fixing same-session context overwrite bug - Update `OnBatchCompleted` to extract image descriptions from `AIInteractionText.Content` instead of
…s persistence after batch completion - Add `HasActiveBatchSubmission` protected property to distinguish poll cycles from new runs in derived classes - Call `SetMetricsOutput(null)` in `OnBatchCompleted` to persist aggregated metrics for `RestorePersistentOutputs` (since `OnSolveInstancePostSolve` is skipped during batch) - Update `AIMetrics.CombineWith()` to skip overwriting Provider/Model with "Unknown" or empty values during
- Replace ASCII encoding with UTF-8 BOM () in file headers across 5 test files - Change HtmlReadabilityHelper.ExtractMainContent reflection binding from NonPublic to Public in 7 test methods - Remove GH_ExtractedImage tests (204 lines) from ExtractedImageTests.cs, keeping only ExtractedImagePocoTests - Remove GH_Structure tests (164 lines) from DataTreeProcessorMixedTypeTests.cs, keeping only ProcessingTopology and ProcessingOptions
…ation and GH_Path/GH_Structure test implementations - Add GrasshopperTestComponents.md with 198 lines documenting test component pattern, structure, implementation guidelines, and migration from xUnit - Document test component characteristics: septenary exposure, async worker pattern, success/messages outputs, comprehensive logging - Add comparison table between xUnit tests and test components for Grasshopper-dependent testing
…maProp components - Add validation to reject colons (:) in name and description fields for both JsonSchemaObjectComponent and JsonSchemaPropComponent - Add error messages "Name cannot contain colons (:)." and "Description cannot contain colons (:)." when validation fails - Insert validation checks after empty string validation and before name trimming in both components
…provider-specific tests and hardcoded test data approach - Add 30 provider-specific test components across 6 providers (OpenAI, MistralAI, DeepSeek, Gemini, Anthropic, OpenRouter) - Document 5 test cases per provider: Encode, Decode, Standard Call, Batch Call, Tools/Function Calling - Clarify all test data is hardcoded internally - users only toggle Run=true - Reduce core functionality tests from 25 to 20 (removed unsuitable tests requiring
- Change "Stack to describing" to "Stick to describing" in DefaultImageDescriptionPrompt constant
- Add gemini_icon.png to Resources folder - Register gemini_icon in Resources.resx as embedded resource - Update GeminiProvider.Icon property to return Properties.Resources.gemini_icon instead of null
… source files - Remove trailing spaces from markdown files (DataTreeProcessingSchema.md, PR-Testing-Plan.md) - Normalize file encoding from UTF-8 with BOM to UTF-8 in AIStatefulAsyncComponentBase.cs - Add null-coalescing operator to Messages property initialization in AIBatchTypes.cs
…re and add OpenAI provider project - Add SmartHopper.Providers.OpenAI project to solution with all build configurations - Update Anthropic test components to use new AICall.Core namespace structure (Base, Interactions, Requests, Returns) - Replace AIInteraction with specific interaction types (AIInteractionText, AIInteractionToolCall, AIInteractionToolResult) - Change AIInteraction.Role to AIInteraction.Agent across all test components - Replace
…iation patterns - Add settings.png and settingsextra.png icon resources for AISettings and AIExtraSettings components - Update AISettingsComponent and AIExtraSettingsComponent to use new icon resources instead of null - Fix Discourse and Ladybug forum topic summarize components to use correct icons (discoursetopicsummarize, ladybugtopicsummarize) - Refactor GeminiProvider to use AIProvider<GeminiProvider>.Instance singleton pattern,
…omponent icons - Add jsonarray.png and jsonobj.png icon resources for JsonArray and JsonObject components - Update JsonArrayComponent and JsonObjectComponent to use new icons instead of textgenerate - Update forum component icons (Discourse, Ladybug, McNeel) for search, get, open, and summarize operations - Add sealed modifier to GeminiProvider partial class declarations - Add exception handling and debug logging to GeminiProvider
…sage metrics extraction, and improve image handling - Add service_tier extra setting for per-request tier override (standard/flex/priority) - Add batch_priority extra setting for batch request prioritization - Refactor batch API to use correct Gemini format with InlinedRequest structure and Operation polling - Add ExtractUsageMetadata method to capture promptTokenCount, candidatesTokenCount, and thoughtsTokenCount
…roviders and fix stopwatch cleanup - Add structured AIReturn error handling in AIProvider.CallApi for HTTP failures instead of throwing exceptions - Add context-specific error messages for common HTTP status codes (503, 429, 401/403, 408, 413, 500, 502, 504) - Include provider-specific guidance (Flex tier for 503, retry delays for 429, API key checks for auth errors) - Tag all HTTP errors with Provider origin for consistent UI
… text2json prompt - Comment out colon validation in JsonSchemaObjectComponent and JsonSchemaPropComponent description fields - Add instruction to text2json default prompt to not return a copy of the JSON Schema in the response
…lution logic - Change AIRequestBase.TimeoutSeconds to nullable int? to allow null/empty values - Update RequestTimeoutPolicy to resolve timeout from settings when null (reads "TimeoutSeconds" with fallback to "HttpTimeoutSeconds" for backward compatibility, uses 300s default) - Add Timeout input parameter to AISettingsComponent (positioned before Extras) - Add ConfigureRequestTimeout() helper method to AIStatefulAsyncComponentBase for centralized timeout configuration - Remove duplicate timeout resolution logic from AI
… into feature/2.0.0-text2json
…r credits - Update Rhinoceros icon URL to direct icon link instead of search term - Reorder attribution section to show icon credits before logo design thanks label
…imeSpan.FromSeconds - Cast timeout calculation result to double in AIToolCall.ExecuteAsync to ensure correct TimeSpan.FromSeconds overload is used
… nullable TimeoutSeconds - Add explicit double cast to TimeoutSeconds fallback expressions in all providers to ensure correct TimeSpan.FromSeconds overload - Extract clamped timeout calculation to separate variable in AIToolCall.ExecuteAsync for clarity - Update Gemini streaming to use .Value accessor for nullable TimeoutSeconds - Handle nullable TimeoutSeconds with null-coalescing in AIToolCall timeout resolution
…ze JSON path error formatting - Add bracket notation support (e.g., "results[0].name", "items[5]") to JsonGetValueComponent alongside existing dot notation - Implement ParsePathSegments() method to parse mixed dot and bracket notation paths into PathSegment objects - Add PathSegment class to represent property names and array indices in JSON paths - Create JsonPathHelper utility class with FormatJsonPathError, FormatJsonPathWarning, For
…raction and minification - Create JsonFormatHelper utility class in SmartHopper.Infrastructure.Utilities for consistent JSON formatting across all components - Add core methods: JsonToString() (JToken/string to minified JSON), StringToJson() (string to JToken), IsValidJson() (validation with optional parsing) - Implement automatic markdown code block extraction (```json, ```txt, ```text, ```) in string-based methods before processing
…sition in WebChatObserver - Add comment explaining that turn completion marks exit from processing state and readiness for next user message
…onent class documentation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…rs for all test provider components - Update ComponentGuid for TestAnthropicBatchCallComponent, TestAnthropicDecodeComponent, TestAnthropicEncodeComponent, TestAnthropicStandardCallComponent, TestAnthropicToolsComponent - Update ComponentGuid for TestGeminiDecodeComponent, TestGeminiEncodeComponent, TestGeminiFunctionCallingComponent, TestGeminiStandardCallComponent, TestGeminiVisionComponent - Replace duplicate/placeholder GUIDs with properly
… requests for batch status and cancel operations - Add SendBatchRequestAsync helper method for direct HTTP GET/POST requests to batch endpoints - Replace Call/Decode pipeline usage in GetBatchStatusAsync with direct HTTP GET to avoid unnecessary response parsing through generateContent decoder - Replace Call/Decode pipeline usage in CancelBatchAsync with direct HTTP POST for consistency - Add explanatory comment that batch Operation
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…gs and register textlist2boolean tool in AIList2BooleanComponent - Remove redundant bool.TryParse attempt on non-parseable strings in both AIText2BooleanComponent and AIList2BooleanComponent - Directly treat non-parseable strings as fallback cases (null result, usedFallback=true) instead of attempting secondary parse - Change UsingAiTools from virtual to override in AIList2BooleanComponent and register "textlist2boolean" tool
There was a problem hiding this comment.
📝 Info: Discourse tools consolidation — 6 classes replaced by 3 via DiscourseToolsBase refactor
The PR consolidates separate post/topic tool classes (mcneelpost2text, mcneeltopic2text, ladybugpost2text, ladybugtopic2text, discoursepost2text, discoursetopic2text) into 3 classes (discourse_mcneel_tools, discourse_ladybug_tools, discourse_tools) that inherit from the refactored DiscourseToolsBase. The base class now uses PresetBaseUrl (nullable) instead of abstract BaseUrl, and the generic discourse_tools variant requires base_url as a parameter. The old classes that filtered tools (e.g., only post-related or topic-related) are gone — each new class now exposes ALL tools (search, get_post, get_topic, summarize_post, summarize_topic). This means McNeel and Ladybug forum tool sets doubled in size. The GetTools() override that filtered is removed. Verified tool names remain backward-compatible (mcneel_forum_search, ladybug_forum_post_get, etc.).
Was this helpful? React with 👍 or 👎 to provide feedback.
| // Forward it so ReconstructOutputTree can replace it after the batch completes. | ||
| var resultValue = toolResult["result"]?.ToString(); | ||
| if (resultValue != null && resultValue.StartsWith("##SH_BATCH:", StringComparison.Ordinal)) | ||
| { | ||
| return result; | ||
| outputs["Result"].Add(new GH_String(resultValue)); | ||
| continue; | ||
| } | ||
|
|
||
| // Non-batch: get the result (could be boolean string or fallback value) | ||
| // The result from the tool is already processed | ||
| outputs["Result"].Add(new GH_String(resultValue ?? string.Empty)); | ||
| } | ||
|
|
||
| return null; | ||
| return outputs; |
There was a problem hiding this comment.
🔴 "Used Fallback" output is always false in non-batch mode when fallback IS used
In AIText2BooleanComponent and AIList2BooleanComponent, when the AI returns an unparseable response and the text2boolean/textlist2boolean tool applies the fallback value, the component's ProcessData only extracts toolResult["result"] as a string (which is now the boolean fallback value, e.g., "True"). Later, ConvertStringTreeToBoolean sees this valid boolean string and marks usedFallback = false. The tool DID set toolResult["usedFallback"] = true, but the component never reads that field. So the "Used Fallback" output always reports false even when the fallback was actually used, making it impossible for users to distinguish "AI said true" from "AI returned gibberish, fallback was applied".
Trace through the code
- AI returns unparseable text (e.g., "I think yes")
- Tool at
text2boolean.cs:186-206: setsresult = fallback.Value,usedFallback = true - Component ProcessData at
AIText2BooleanComponent.cs:388:resultValue = toolResult["result"]?.ToString()→"True" ConvertStringTreeToBooleanat line 288:bool.TryParse("True")succeeds →usedFallbackBranch.Add(new GH_Boolean(false))
Prompt for agents
The component's ProcessData method only extracts toolResult["result"] but ignores toolResult["usedFallback"]. The outputs dictionary should carry both values so ConvertStringTreeToBoolean (or a replacement) can correctly report whether the fallback was used.
Approach: In ProcessData, extract both "result" and "usedFallback" from the tool result. Instead of storing a single "Result" string that gets re-parsed, store a tuple or use two separate output lists (one for the boolean result, one for the usedFallback flag). Then ConvertStringTreeToBoolean can use the tool's usedFallback directly instead of re-inferring it from string parseability.
The same fix is needed in AIList2BooleanComponent.cs ProcessData (around line 395-405).
Was this helpful? React with 👍 or 👎 to provide feedback.
| private static (GH_Boolean value, bool usedFallback) ParseBooleanWithFallback( | ||
| JObject resultBody, | ||
| System.Func<JObject, System.Collections.Generic.List<IAIInteraction>> decode) | ||
| { | ||
| if (resultBody == null) | ||
| { | ||
| return (null, true); | ||
| } | ||
|
|
||
| var interactions = decode(resultBody); | ||
| var lastText = interactions | ||
| ?.OfType<AIInteractionText>() | ||
| .LastOrDefault(i => i.Agent == AIAgent.Assistant); | ||
|
|
||
| if (lastText == null) | ||
| { | ||
| return (null, true); | ||
| } | ||
|
|
||
| if (bool.TryParse(lastText.Content?.Trim(), out bool value)) | ||
| { | ||
| return (new GH_Boolean(value), false); | ||
| } | ||
|
|
||
| return (null, true); |
There was a problem hiding this comment.
🔴 Batch mode never applies fallback value for boolean components — result is null instead of fallback
In AIText2BooleanComponent.OnBatchCompleted and AIList2BooleanComponent.OnBatchCompleted, the batch path uses ParseBooleanWithFallback which decodes the raw provider response and attempts bool.TryParse. If the AI response isn't a clean boolean, it returns (null, true) — the fallback value from the component's "Fallback" input is never consulted. In non-batch mode, the text2boolean tool applies the fallback inside its execute function (text2boolean.cs:186-206). But in batch mode, the tool's execute function doesn't run — only BuildEvaluateRequest runs, and the raw provider response is decoded directly in OnBatchCompleted. The user's fallback value is silently lost, resulting in null output items instead of the configured fallback boolean.
Prompt for agents
ParseBooleanWithFallback in AIText2BooleanComponent (and the identical copy in AIList2BooleanComponent) needs access to the component's fallback input value so it can apply it when bool.TryParse fails. Currently ParseBooleanWithFallback is a static method that only receives the result body and decode function.
Approach: Either (1) make ParseBooleanWithFallback accept a bool? fallbackValue parameter, and when parsing fails, return (new GH_Boolean(fallbackValue.Value), true) instead of (null, true), or (2) store the fallback value in a component-level field during GatherInput and access it from OnBatchCompleted.
The same fix is needed in AIList2BooleanComponent.cs (around lines 147-170).
Was this helpful? React with 👍 or 👎 to provide feedback.
| this.ProcessBatchResults<GH_Boolean>( | ||
| "Result", | ||
| sentinel, | ||
| results, | ||
| (customId, resultBody) => | ||
| { | ||
| // Decode and parse ONCE, cache both values | ||
| var (value, usedFallback) = ParseBooleanWithFallback(resultBody, provider.Decode); | ||
| this._batchParseCache[customId] = (value, usedFallback); | ||
| return value; | ||
| }, | ||
| messages); | ||
|
|
||
| // Process Used Fallback output using cached values (no re-parse!) | ||
| // Use the same sentinel as Result since both trees have identical structure | ||
| this.ProcessBatchResults<GH_Boolean>( | ||
| "Used Fallback", | ||
| sentinel, | ||
| results, | ||
| (customId, resultBody) => | ||
| { | ||
| // Retrieve from cache - no parsing needed | ||
| if (this._batchParseCache.TryGetValue(customId, out var cached)) | ||
| { | ||
| return new GH_Boolean(cached.usedFallback); | ||
| } | ||
| return null; | ||
| }, | ||
| messages); | ||
|
|
||
| // Clear cache | ||
| this._batchParseCache = null; |
There was a problem hiding this comment.
🟡 Double ProcessBatchResults causes every batch result to be provider-Decoded 3 times
In AIText2BooleanComponent.OnBatchCompleted (lines 108-141) and AIList2BooleanComponent.OnBatchCompleted, ProcessBatchResults is called twice — once for "Result" and once for "Used Fallback". Each ProcessBatchResults call internally calls provider.Decode(resultBody) at AIStatefulAsyncComponentBase.cs:1571 for every sentinel. Additionally, the first call's user decode lambda (ParseBooleanWithFallback) calls provider.Decode again. This means each batch result body is decoded 3 times total (2× in the first call + 1× in the second call), tripling the decode work. The second call also overwrites _persistedMetrics and AIReturnSnapshot set by the first call.
Prompt for agents
Instead of calling ProcessBatchResults twice, call it once for the primary output ("Result") and use TransformOutputs or manual iteration to populate the "Used Fallback" tree in the same pass. This avoids redundant provider.Decode calls and the double FinishResults/SetMetricsOutput emissions.
Alternatively, the decode cache pattern could be extended to also cache the decoded interactions so the second ProcessBatchResults call skips the internal provider.Decode step. But the cleanest fix is a single ProcessBatchResults call that populates both outputs.
The same pattern exists in AIList2BooleanComponent.OnBatchCompleted (lines ~110-144).
Was this helpful? React with 👍 or 👎 to provide feedback.
| this.Surfaceable = surfaceable; | ||
| } |
There was a problem hiding this comment.
📝 Info: AIRuntimeMessage constructor breaking change — all existing callers must add AIMessageCode parameter
The 3-parameter constructor AIRuntimeMessage(severity, origin, message) was removed (see AIRuntimeMessage.cs diff). All call sites now use the 4-parameter version with an explicit AIMessageCode. This is visible in the diff where every new AIRuntimeMessage(...) call in validation, returns, and policy code was updated to include the code parameter. The AddRuntimeMessage(severity, origin, text) convenience method on AIReturn (AIReturn.cs:397) still exists and passes AIMessageCode.Unknown internally. Any external consumers or plugins that used the removed 3-parameter constructor will fail to compile. Since this is an infrastructure library, this is a deliberate API cleanup rather than a bug.
(Refers to lines 102-109)
Was this helpful? React with 👍 or 👎 to provide feedback.
| this.ProcessBatchResults<GH_Boolean>( | ||
| "Result", | ||
| sentinel, | ||
| results, | ||
| (customId, resultBody) => | ||
| { | ||
| // Decode and parse ONCE, cache both values | ||
| var (value, usedFallback) = ParseBooleanWithFallback(resultBody, provider.Decode); | ||
| this._batchParseCache[customId] = (value, usedFallback); | ||
| return value; | ||
| }, | ||
| messages); | ||
|
|
||
| // Process Used Fallback output using cached values (no re-parse!) | ||
| // Use the same sentinel as Result since both trees have identical structure | ||
| this.ProcessBatchResults<GH_Boolean>( | ||
| "Used Fallback", | ||
| sentinel, | ||
| results, | ||
| (customId, resultBody) => | ||
| { | ||
| // Retrieve from cache - no parsing needed | ||
| if (this._batchParseCache.TryGetValue(customId, out var cached)) | ||
| { | ||
| return new GH_Boolean(cached.usedFallback); | ||
| } | ||
| return null; | ||
| }, | ||
| messages); | ||
|
|
||
| // Clear cache | ||
| this._batchParseCache = null; |
There was a problem hiding this comment.
🚩 ProcessBatchResults called twice for AIText2Boolean/AIList2Boolean with same sentinel tree
In AIText2BooleanComponent.OnBatchCompleted (lines 107-139), ProcessBatchResults is called twice on the same sentinel tree: once for 'Result' and once for 'Used Fallback'. Each call to ProcessBatchResults calls FinishResults internally, which calls SetPersistentOutput and SetMetricsOutput. The second call to ProcessBatchResults will overwrite the metrics persisted by the first call and call FinishResults again for the 'Used Fallback' output. This works because FinishResults is additive for SetPersistentOutput (it doesn't clear previous outputs), but the metrics are emitted twice. The _batchParseCache pattern avoids double-parsing, which is good, but the double FinishResults call is architecturally unusual compared to other components that use a single FinishResults with additionalOutputs.
Was this helpful? React with 👍 or 👎 to provide feedback.
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SmartHopper.Providers.OpenAI", "src\SmartHopper.Providers.OpenAI\SmartHopper.Providers.OpenAI.csproj", "{C7B03DDF-7342-9BBA-E63A-8B911F79F8F4}" | ||
| EndProject | ||
| Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "SmartHopper.Providers.Gemini", "src\SmartHopper.Providers.Gemini\SmartHopper.Providers.Gemini.csproj", "{087FFA5E-1049-459D-9C68-1C0B8E7F9EBC}" |
There was a problem hiding this comment.
📝 Info: Solution file swaps OpenAI project GUID and reuses old one for Gemini
In SmartHopper.sln, the OpenAI project GUID changed from {087FFA5E-1049-459D-9C68-1C0B8E7F9EBC} to {C7B03DDF-7342-9BBA-E63A-8B911F79F8F4}, and the old OpenAI GUID {087FFA5E-...} is now used for the new Gemini project. While solution-level project GUIDs are just internal references and this is not a functional bug, it's unusual to swap GUIDs between projects rather than generating a fresh one for the new project. This could cause confusion in version control and build tooling.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if (Uri.TryCreate(imageUrl, UriKind.Absolute, out var uri)) | ||
| { | ||
| this.SetResult(uri, imageData, revisedPrompt); | ||
| return; | ||
| } | ||
| } | ||
|
|
||
| // Handle case where only imageData is provided (no URL) | ||
| if (imageData != null) | ||
| { | ||
| this.ImageData = imageData; | ||
| } | ||
|
|
||
| if (revisedPrompt != null) | ||
| { | ||
| this.RevisedPrompt = revisedPrompt; | ||
| } | ||
| } |
There was a problem hiding this comment.
📝 Info: AIInteractionImage.SetResult silently drops invalid URL string with imageData=null
In AIInteractionImage.SetResult(string, string, string) at line 173-181, when imageUrl is a non-null string that fails Uri.TryCreate (e.g., a relative path or malformed URL), and imageData is also null, the method silently completes without setting any image data and without throwing. The precondition check at line 168 only throws when BOTH are null. A call like SetResult("not-a-url") would pass the null check (imageUrl is non-null), fail the Uri parse, skip the imageData branch (it's null), and return with no image data set. This edge case existed before the PR but the added return statement at line 179 and the new fallback logic at lines 183-192 were added to address image-data-only cases.
(Refers to lines 166-193)
Was this helpful? React with 👍 or 👎 to provide feedback.
| else if (interaction is AIInteractionImage imageInteraction) | ||
| { | ||
| if (!string.IsNullOrWhiteSpace(imageInteraction.ImageData)) | ||
| { | ||
| parts.Add(new JObject | ||
| { | ||
| { | ||
| "inline_data", new JObject | ||
| { | ||
| { "mime_type", imageInteraction.MimeType ?? "image/png" }, | ||
| { "data", imageInteraction.ImageData }, | ||
| } | ||
| }, | ||
| }); | ||
| } | ||
| else if (imageInteraction.ImageUrl != null) | ||
| { | ||
| // Fetch image from URL and convert to base64 inline data | ||
| var (base64Data, mimeType) = this.FetchImageFromUrl(imageInteraction.ImageUrl); | ||
| if (!string.IsNullOrWhiteSpace(base64Data)) | ||
| { | ||
| parts.Add(new JObject | ||
| { | ||
| { | ||
| "inline_data", new JObject | ||
| { | ||
| { "mime_type", mimeType }, | ||
| { "data", base64Data }, | ||
| } | ||
| }, | ||
| }); | ||
| } | ||
| else | ||
| { | ||
| // Fallback: add URL as text if fetch fails | ||
| parts.Add(new JObject { { "text", imageInteraction.ImageUrl.ToString() } }); | ||
| Debug.WriteLine($"[GeminiProvider] Warning: Failed to fetch image from URL, sending URL as text: {imageInteraction.ImageUrl}"); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| if (parts.Count > 0) | ||
| { | ||
| contents.Add(new JObject | ||
| { | ||
| { "role", role }, | ||
| { "parts", parts }, | ||
| }); | ||
| } |
There was a problem hiding this comment.
📝 Info: GeminiProvider.Encode duplicates image-handling logic between single-interaction and full-request paths
The Encode(IAIInteraction) method (GeminiProvider.cs:170) and Encode(AIRequestCall) method (GeminiProvider.cs:306) both contain nearly identical image encoding logic (inline_data for base64, URL fetch fallback). The single-interaction encoder calls EncodeToJToken which has the logic, but the full-request encoder re-implements it inline instead of delegating. This creates a maintenance risk where a fix to one path might not be applied to the other. Consider extracting the image part encoding to a shared private method.
Was this helpful? React with 👍 or 👎 to provide feedback.
| // NOTE: reasoning_effort is incompatible with function tools on gpt-5.4, so omit it when tools are present | ||
| if (OSeriesModelRegex().IsMatch(request.Model) || Gpt5ModelRegex().IsMatch(request.Model)) | ||
| { | ||
| requestBody["reasoning_effort"] = reasoningEffort; | ||
| // Only add reasoning_effort if no tools are present (gpt-5.4 incompatibility) | ||
| if (!hasTools) |
There was a problem hiding this comment.
📝 Info: OpenAI reasoning_effort omitted when tools are present on gpt-5 models
At OpenAIProvider.cs:583-587, reasoning_effort is explicitly omitted when tools are present (if (!hasTools) { requestBody["reasoning_effort"] = reasoningEffort; }). The comment says this is due to a "gpt-5.4 incompatibility". This is a deliberate design decision but changes observable behavior: users requesting reasoning with function calling on o-series/gpt-5 models will silently get no reasoning effort parameter. Consider surfacing a diagnostic message when this suppression occurs.
Was this helpful? React with 👍 or 👎 to provide feedback.
feat: SmartHopper 2.0.0 - Google Gemini Provider, Batch API, Vision Input, and Breaking Changes
Description
This PR introduces SmartHopper 2.0.0, a major release with significant new features, architectural improvements, and breaking changes. This is a large-scale refactor spanning 208 files with >22,000 insertions and 2,000 deletions.
Key Features Added
🚀 Google Gemini Provider (Full Integration)
x-goog-api-keyauthentication in centralizedCallApimethod⚡ Mixed-Type Data Tree Support
GH_Boolean,GH_String, etc.)GHStructureConverterutility for converting typed structures toIGH_GooDataTreeProcessorwithIGH_Gootype gate for branch groupingProcessingResult<T>classes for returning output trees and messagesAIText2BooleanComponentandAIList2BooleanComponentto mixed-type pipelines📊 Batch API Support (OpenAI, Anthropic, MistralAI, Gemini)
IAIBatchProviderinterface implementation across three providersBatchTierparameter inAIRequestParameters(replacesservice_tierextra)🖼️ Vision Input Support
img2textAI toolAIInteractionImagewith vision input methods andMimeTypepropertyGH_ExtractedImageGoo type for Grasshopper integrationAIImgToTextComponentfor standalone image descriptionAIFile2MdComponentwith image modes:embed,describe,caption📄 File-to-Markdown Conversion
file2mdAI tool supporting 12 formats: PDF, DOCX, XLSX, PPTX, HTML, CSV, JSON, XML, TXT, EML, EPUB, RTFFile2MdComponent(non-AI) andAIFile2MdComponent(AI-powered)🌐 Web-to-Markdown Conversion
web2mdAI tool (replacesweb_generic_page_read)⚙️ AI Settings Components
AISettingsComponent: Universal settings component for cross-provider parametersAIExtraSettingsComponent: Dynamic inputs based on provider-specific extrasAIRequestParametersimmutable record with fluent builderGH_AIRequestParametersGrasshopper wrapper with backward-compatible string casting🛠️ JSON Tools & Components
text2jsonAI tool for structured JSON generation from promptsAIText2JsonComponentfor AI-powered JSON creationJsonSchemaComponent,JsonObjectComponent,JsonArrayComponent,JsonArray2TextListComponent,JsonObject2TextComponent,JsonGetValueComponent,JsonMergeComponentJsonSchemaPropComponent,JsonSchemaPropObjectComponent,JsonSchemaPropArrayComponentBreaking Changes
text_generatetext2texttext_evaluatetext2booleanlist_generatetext2textlistlist_evaluatetextlist2booleanimg_generatetext2imgimg_to_textimg2textweb_to_mdweb2mdweb_generic_page_readweb2md)AITextGenerateAIText2TextComponentAITextEvaluateAIText2BooleanComponentAITextListGenerateAIText2TextListComponentAIListEvaluateAIList2BooleanComponentAIImgGenerateComponentAIText2ImgComponentAIImgToTextComponentAIImg2TextComponentWebPageReadComponentWeb2MdComponent)Testing Done
Comprehensive testing plan created in docs/Reviews/260402-PR-Testing-Plan.md covering 202 test cases across 11 feature areas.
🔴 P0 - Breaking Changes (15 tests)
.ghfiles with old components - verify they load without errorsAITextGenerate→AIText2TextComponentmigration worksAITextEvaluate→AIText2BooleanComponentmigration worksAITextListGenerate→AIText2TextListComponentmigration worksAIListEvaluate→AIList2BooleanComponentmigration worksAIImgGenerateComponent→AIText2ImgComponentmigration worksAIImgToTextComponent→AIImg2TextComponentmigration worksWebPageReadComponentremoval doesn't crash file loadservice_tier=batchin existing files is silently ignoredAISettingsComponentworks correctlyBatchinput instead ofservice_tierextra🔴 P0 - Google Gemini Provider (19 tests)
AIText2TextComponentAIText2BooleanComponentAIText2JsonComponent🔴 P0 - Mixed-Type Data Trees (17 tests)
GH_Structure<GH_String>toGH_Structure<IGH_Goo>GH_Structure<GH_Boolean>toGH_Structure<IGH_Goo>GH_Structure<GH_Integer>toGH_Structure<IGH_Goo>GH_Structure<GH_Number>toGH_Structure<IGH_Goo>groupIdenticalBrancheswithIGH_Gootype gateRunAsync<T>overload for heterogeneous outputExtractTypedTree<U>helper methodAIText2BooleanComponent- mixed-type input tree withGH_BooleanfallbackAIList2BooleanComponent- mixed-type input tree withGH_BooleanfallbackProcessingResult<IGH_Goo>.OutputsaccessExtractTypedTree<GH_String>()from heterogeneous resultsRunProcessingAsync<GH_String>with tree broadcastingItemGraftpath management consistencyComponentProcessingOptionsproperty behavior🔴 P0 - Batch API (26 tests)
/v1/files/v1/batches/v1/files/{output_file_id}/content/v1/batches/{id}/cancelrequest_counts.completedupdates progressPOST /v1/messages/batchesprocessing_statusonGET /v1/messages/batches/{id}results_urlPOST /v1/messages/batches/{id}/cancelrequest_counts.succeededupdates progressPOST /v1/batch/jobsGET /v1/batch/jobs/{id}/v1/files/{output_file}/contentPOST /v1/batch/jobs/{id}/cancelsucceeded_requestsupdates progressAITextGeneratebatch completion withOnBatchCompletedoverrideReconstructOutputTree<T>replaces sentinels correctlyCustomIdsserialization inWrite()/Read()AIInteractionErrordetection in providerDecode()methodsPreparing X/X...during data collection🟡 P1 - Vision Input (25 tests)
MimeTypeproperty is correctly setCreateVisionInput()methodCreateVisionInputFromBase64()methodAddImageInput()fluent methodAddImageInputFromBase64()fluent methoddata:{mime};base64,{data})imagecontent blocks with base64image_urlcontent blocksimageUrlparameterimageBase64+mimeTypepromptparameterAICapability.Image2Textrequirement enforcementAIImgToTextComponent- file path inputAIImgToTextComponent- URL inputAIImgToTextComponent- base64 inputAIImgToTextComponent-GH_ExtractedImageinput (regression test for MistralAI fix)AIFile2MdComponent- image modeembedAIFile2MdComponent- image modedescribeAIFile2MdComponent- image modecaptionScriptVariable()returnsBitmapGH_StringBitmap🟡 P1 - File-to-Markdown (32 tests)
RecursiveXYCutImagePartsSlidePart.ImagePartsTryGetPngFile2MdComponent- basic conversionFile2MdComponent-Imagesoutput withGH_ExtractedImageAIFile2MdComponent- AI-powered conversionAIFile2MdComponent- image modes (embed,describe,caption)AIFile2MdComponent- batch context persistence across save/reloaddescribeImagesparameterimagesarray always returned in resultimageModeparameter (embed,describe,caption)🟡 P1 - Web-to-Markdown (12 tests)
Web2MdComponent- basic URL conversionweb_generic_page_readremoval - useweb2mdinstead🟡 P1 - AI Settings Components (23 tests)
AIRequestParametersfrom inputsSettings (S)wire connects to AI componentsAIRequestParametersBuilderfluent methodsWithBatchTier()/ClearBatchTier()GH_AIRequestParametersEncode()🟡 P1 - JSON Tools (24 tests)
promptparameter (required)instructionsparameter (optional)jsonSchemaparameter (required)AICapability.TextInput | JsonOutputenforcementSmartHopper > JSONcategoryItemGraft + GroupIdenticalBranchesprocessing topologyJsonSchemaComponent- build schema from propertiesJsonSchemaComponent- nested properties via dot-notationJsonObjectComponent- create JSON from Key+Value listsJsonArrayComponent- create JSON array from itemsJsonArray2TextListComponent- parse JSON array to GH text listJsonObject2TextComponent- serialize JSON to stringJsonGetValueComponent- extract nested value by dot-notationJsonMergeComponent- merge multiple JSON objectsJsonSchemaPropComponent- scalar property definitionJsonSchemaPropObjectComponent- object property with sub-propertiesJsonSchemaPropArrayComponent- array property with items type🟢 P2 - Provider Model Updates (11 tests)
🟢 P2 - UI/UX (8 tests)
Processing batch (0/XX)...(YY/XX)...Preparing X/X...shown during data collectionMigration Guide
SmartHopper 2.0.0-alpha will automatically install from Rhino Package Manager once released.
For Breaking Changes
.ghfiles - Components should migrate automatically on load. You might need to remove and replace some components if you expirience issues.WebPageReadComponent→Web2MdComponentFor New Features
AISettingsComponentwithBatch=trueon compatible components and providersAIImgToTextComponentor use inAIFile2MdComponentAIText2JsonComponentwith schema inputsChecklist
2.0.0-dev.260402[Unreleased]Related Issues