Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions proto/tim-api/tim/api/thread/v1alpha1/thread_service.proto
Original file line number Diff line number Diff line change
Expand Up @@ -255,12 +255,16 @@ message StreamThreadEventsResponse {
ContentDeltaEvent content_delta = 2;
// A content block has completed
ContentStopEvent content_stop = 3;
// A tool call made by the LLM (input may be null if >512KB)
tim.api.tool.v1alpha1.ToolCall tool_call = 4;
// A tool call is about to be executed (tool name known, input may be incomplete)
ToolCallStartEvent tool_call_start = 4;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps call this ToolCallCreated since this is not that execution is started or that the call has even be made. Could be a hair confusing.

// A tool call made by the LLM with full input (input may be null if >512KB)
tim.api.tool.v1alpha1.ToolCall tool_call = 5;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this full ToolCall is here in order for local exec runner to be able to execute actions? It feels a bit out of place where everything else in this is a Event type created specifically for this? Similar to how we don't record full ToolCall Results through the stream, could be awkward in some instances as well?

Maybe we should have some sort of user side event that would only include full parameters on things we expect user side? With the above note we could have a lifecycle:

  • ToolCallCreated
  • ToolCallStarted - when the full tool call is known, has a "summary" for user facing information, and perhaps only full parameters when a user or local exec runner would be expected to execute?
  • ToolCallComplete

Maybe is worth a follow up ticket? Curious your thoughts here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this generally make sense. There is a tool call api endpoint that a client can hit to get the full tool call. Right now we don't even pass the full params if they are over a certain size to prevent the events from being too "fat". I am wondering if it also might make sense to just say that, if you are streaming events and see that you need to run a tool, you need to request the tool parameters from the API. Basically what you outlined, except ToolCallStarted would only ever have a summary. Any client listening that believes it should run the tool would need to fetch the full parameters from the API.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ToolCallStarted having a summary would be nice in that currently a client needs to understand the input structure for every single tool just to display something about what's happening to the end user. This would greatly simplify that and a client would only need to be concerned with the input to a tool if they are executing it

// A tool call has completed execution with results
ToolCallCompleteEvent tool_call_complete = 6;
// The thread's LLM processing state has changed (IDLE or PROCESSING)
ThreadStateChangeEvent thread_state_change = 5;
ThreadStateChangeEvent thread_state_change = 7;
// The LLM stream encountered an error
StreamErrorEvent stream_error = 6;
StreamErrorEvent stream_error = 8;
}
}

Expand Down
18 changes: 18 additions & 0 deletions proto/tim-api/tim/api/thread/v1alpha1/thread_types.proto
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,24 @@ message ContentStopEvent {
string id = 1 [(google.api.field_info).format = UUID4];
}

// ToolCallStartEvent signals that a tool call is about to be executed
message ToolCallStartEvent {
// Tool call ID (UUID)
string id = 1 [(google.api.field_info).format = UUID4];
// Name of the tool being called
string tool_name = 2;
}

// ToolCallCompleteEvent signals that a tool call has completed execution
message ToolCallCompleteEvent {
// Tool call ID (UUID)
string id = 1 [(google.api.field_info).format = UUID4];
// Whether the tool call was successful
bool success = 2;
// Brief summary of what the tool did
string summary = 3;
}

// ThreadStateChangeEvent notifies when thread LLM state changes
message ThreadStateChangeEvent {
// IDLE or PROCESSING
Expand Down
4 changes: 4 additions & 0 deletions proto/tim-api/tim/api/tool/v1alpha1/tool_types.proto
Original file line number Diff line number Diff line change
Expand Up @@ -35,4 +35,8 @@ message ToolResult {
// Whether this tool result should stop the LLM iteration loop
// Set to true by tools like query_complete, code_complete to signal conversation completion
bool stop_iteration = 4;

// A brief summary of what the tool did (e.g., "Updated todo to in_progress", "Read 234 lines")
// Used for client notifications without exposing full result
string summary = 5;
}
1 change: 1 addition & 0 deletions shared/llm/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,7 @@ func GenerateToolUseID() string {
type ToolResultContent struct {
ToolUseID string `json:"tool_use_id"`
Result string `json:"result"`
Summary string `json:"summary,omitempty"`
IsError bool `json:"is_error,omitempty"`
StopIteration bool `json:"stop_iteration,omitempty"` // Signal to stop LLM iteration loop (e.g., query_complete, code_complete)
}
Expand Down
16 changes: 10 additions & 6 deletions shared/tools/add_todo.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,18 +60,18 @@ func (t *AddTodoTool) ValidateRawInput(input json.RawMessage) error {
return err
}

func (t *AddTodoTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (bool, string, error) {
func (t *AddTodoTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (*ToolExecutionResult, error) {
parsed, err := ValidateAndParse(input, t.validate)
if err != nil {
return false, "", err
return nil, err
}

if t.deps.Backend == nil {
return false, "", fmt.Errorf("add_todo requires backend (tool cannot run in this context)")
return nil, fmt.Errorf("add_todo requires backend (tool cannot run in this context)")
}

if t.deps.ThreadPath == "" {
return false, "", fmt.Errorf("thread path required for add_todo (not available in current context)")
return nil, fmt.Errorf("thread path required for add_todo (not available in current context)")
}

// Default priority to medium
Expand All @@ -86,10 +86,14 @@ func (t *AddTodoTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (bo

todoPath, err := t.deps.Backend.CreateTodo(ctx, t.deps.ThreadPath, parsed.Title, parsed.Description, priority)
if err != nil {
return false, "", fmt.Errorf("failed to create todo: %w", err)
return nil, fmt.Errorf("failed to create todo: %w", err)
}

t.deps.Logger.Infow("todo created", "todo_path", todoPath)

return false, fmt.Sprintf("Todo added: %s", parsed.Title), nil
return &ToolExecutionResult{
StopIteration: false,
Result: fmt.Sprintf("Todo added: %s", parsed.Title),
Summary: fmt.Sprintf("Added todo: %s", parsed.Title),
}, nil
}
2 changes: 1 addition & 1 deletion shared/tools/clarify.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,6 @@ func (t *ClarifyTool) ValidateRawInput(input json.RawMessage) error {
return err
}

func (t *ClarifyTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (bool, string, error) {
func (t *ClarifyTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (*ToolExecutionResult, error) {
panic("TODO: clarify tool requires client-side execution - implement in CLI")
}
16 changes: 10 additions & 6 deletions shared/tools/complete_todo.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,18 +56,18 @@ func (t *CompleteTodoTool) ValidateRawInput(input json.RawMessage) error {
return err
}

func (t *CompleteTodoTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (bool, string, error) {
func (t *CompleteTodoTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (*ToolExecutionResult, error) {
parsed, err := ValidateAndParse(input, t.validate)
if err != nil {
return false, "", err
return nil, err
}

if t.deps.Backend == nil {
return false, "", fmt.Errorf("complete_todo requires backend (tool cannot run in this context)")
return nil, fmt.Errorf("complete_todo requires backend (tool cannot run in this context)")
}

if t.deps.ThreadPath == "" {
return false, "", fmt.Errorf("thread path required for complete_todo (not available in current context)")
return nil, fmt.Errorf("thread path required for complete_todo (not available in current context)")
}

// Build todo path from thread path + todo UID
Expand All @@ -78,10 +78,14 @@ func (t *CompleteTodoTool) ExecuteRaw(ctx context.Context, input json.RawMessage
// Update progress to completed
err = t.deps.Backend.UpdateTodo(ctx, todoPath, "completed", "")
if err != nil {
return false, "", fmt.Errorf("failed to complete todo: %w", err)
return nil, fmt.Errorf("failed to complete todo: %w", err)
}

t.deps.Logger.Infow("todo completed", "todo_uid", parsed.TodoUID, "thread_path", t.deps.ThreadPath)

return false, "Todo marked as completed", nil
return &ToolExecutionResult{
StopIteration: false,
Result: "Todo marked as completed",
Summary: "Completed todo",
}, nil
}
22 changes: 13 additions & 9 deletions shared/tools/delegate_work.go
Original file line number Diff line number Diff line change
Expand Up @@ -96,15 +96,15 @@ func (t *DelegateWorkTool) ValidateRawInput(input json.RawMessage) error {
return err
}

func (t *DelegateWorkTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (bool, string, error) {
func (t *DelegateWorkTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (*ToolExecutionResult, error) {
parsed, err := ValidateAndParse(input, t.validate)
if err != nil {
return false, "", err
return nil, err
}

// Check if backend is available
if t.deps.Backend == nil {
return false, "", fmt.Errorf("delegate_work requires backend (tool cannot run in this context)")
return nil, fmt.Errorf("delegate_work requires backend (tool cannot run in this context)")
}

t.deps.Logger.Infow("delegating work to sub-agent",
Expand All @@ -113,19 +113,19 @@ func (t *DelegateWorkTool) ExecuteRaw(ctx context.Context, input json.RawMessage

// Check if user path is available
if t.deps.UserPath == "" {
return false, "", fmt.Errorf("user path required for delegate_work (not available in current context)")
return nil, fmt.Errorf("user path required for delegate_work (not available in current context)")
}

// Check if current thread path is available (to set as parent for sub-agent)
if t.deps.ThreadPath == "" {
return false, "", fmt.Errorf("thread path required for delegate_work (not available in current context)")
return nil, fmt.Errorf("thread path required for delegate_work (not available in current context)")
}

// Create a new thread for the sub-agent with current thread as parent
threadPath, err := t.deps.Backend.CreateThread(ctx, t.deps.UserPath, t.deps.ThreadPath, parsed.Persona, parsed.Description)
if err != nil {
t.deps.Logger.Errorw("failed to create sub-agent thread", "error", err)
return false, "", fmt.Errorf("failed to create sub-agent thread: %w", err)
return nil, fmt.Errorf("failed to create sub-agent thread: %w", err)
}

t.deps.Logger.Infow("created sub-agent thread", "thread_path", threadPath)
Expand All @@ -134,22 +134,26 @@ func (t *DelegateWorkTool) ExecuteRaw(ctx context.Context, input json.RawMessage
err = t.deps.Backend.SubmitUserMessage(ctx, threadPath, parsed.Prompt)
if err != nil {
t.deps.Logger.Errorw("failed to send prompt to sub-agent", "error", err)
return false, "", fmt.Errorf("failed to send prompt to sub-agent: %w", err)
return nil, fmt.Errorf("failed to send prompt to sub-agent: %w", err)
}

t.deps.Logger.Infow("sent prompt to sub-agent", "thread_path", threadPath)

// Poll until the thread is idle (sub-agent finished)
result, err := t.waitForCompletion(ctx, threadPath)
if err != nil {
return false, "", err
return nil, err
}

t.deps.Logger.Infow("sub-agent completed",
"thread_path", threadPath,
"result_length", len(result))

return false, result, nil
return &ToolExecutionResult{
StopIteration: false,
Result: result,
Summary: "Delegated work to sub-agent",
}, nil
}

// waitForCompletion polls the thread until it's idle and returns the final result
Expand Down
16 changes: 10 additions & 6 deletions shared/tools/exec_command.go
Original file line number Diff line number Diff line change
Expand Up @@ -65,14 +65,14 @@ func (t *ExecCommandTool) ValidateRawInput(input json.RawMessage) error {
return err
}

func (t *ExecCommandTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (bool, string, error) {
func (t *ExecCommandTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (*ToolExecutionResult, error) {
parsed, err := ValidateAndParse(input, t.validate)
if err != nil {
return false, "", err
return nil, err
}

if t.deps.Ops == nil {
return false, "", fmt.Errorf("exec_command requires ops provider (tool cannot run in this context)")
return nil, fmt.Errorf("exec_command requires ops provider (tool cannot run in this context)")
}

processID := uuid.New().String()
Expand All @@ -93,7 +93,7 @@ func (t *ExecCommandTool) ExecuteRaw(ctx context.Context, input json.RawMessage)
// Run command
runResult, err := t.deps.Ops.ExecRun(ctx, "", processID, rootDir, parsed.Command, waitForCompletion, timeout, nil)
if err != nil {
return false, "", err
return nil, err
}

// Determine result based on actual process state
Expand All @@ -109,10 +109,14 @@ func (t *ExecCommandTool) ExecuteRaw(ctx context.Context, input json.RawMessage)
}

if err != nil {
return false, "", err
return nil, err
}

return false, result, nil
return &ToolExecutionResult{
StopIteration: false,
Result: result,
Summary: fmt.Sprintf("Executed command: %s", parsed.Command),
}, nil
}

func (t *ExecCommandTool) processStarted(runResult *ops.OpsExecRunResult) (string, error) {
Expand Down
18 changes: 11 additions & 7 deletions shared/tools/execute_connected_app_action.go
Original file line number Diff line number Diff line change
Expand Up @@ -72,18 +72,18 @@ func (t *ExecuteConnectedAppActionTool) ValidateRawInput(input json.RawMessage)
return err
}

func (t *ExecuteConnectedAppActionTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (bool, string, error) {
func (t *ExecuteConnectedAppActionTool) ExecuteRaw(ctx context.Context, input json.RawMessage) (*ToolExecutionResult, error) {
parsed, err := ValidateAndParse(input, t.validate)
if err != nil {
return false, "", err
return nil, err
}

if t.deps.ConnectedAppClient == nil {
return false, "", fmt.Errorf("execute_connected_app_action requires a connected app client (tool cannot run in this context)")
return nil, fmt.Errorf("execute_connected_app_action requires a connected app client (tool cannot run in this context)")
}

if t.deps.UserPath == "" {
return false, "", fmt.Errorf("execute_connected_app_action requires a user path (not available in current context)")
return nil, fmt.Errorf("execute_connected_app_action requires a user path (not available in current context)")
}

// Initialize configured_props if nil
Expand All @@ -93,7 +93,7 @@ func (t *ExecuteConnectedAppActionTool) ExecuteRaw(ctx context.Context, input js

userID := extractUserIDFromPath(t.deps.UserPath)
if userID == "" {
return false, "", fmt.Errorf("failed to extract user ID from user path")
return nil, fmt.Errorf("failed to extract user ID from user path")
}

t.deps.Logger.Infow("executing connected app action",
Expand All @@ -112,15 +112,19 @@ func (t *ExecuteConnectedAppActionTool) ExecuteRaw(ctx context.Context, input js
parsed.ConfiguredProps,
)
if err != nil {
return false, "", fmt.Errorf("failed to execute action %s: %w", parsed.ActionKey, err)
return nil, fmt.Errorf("failed to execute action %s: %w", parsed.ActionKey, err)
}

t.deps.Logger.Infow("successfully executed connected app action",
"user_id", userID,
"action_key", parsed.ActionKey)

// Format the response
return false, formatActionResult(parsed.ActionKey, result), nil
return &ToolExecutionResult{
StopIteration: false,
Result: formatActionResult(parsed.ActionKey, result),
Summary: fmt.Sprintf("Executed action: %s", parsed.ActionKey),
}, nil
}

func formatActionResult(actionKey string, result *ConnectedAppActionResponse) string {
Expand Down
14 changes: 9 additions & 5 deletions shared/tools/executor.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,13 +73,14 @@ func (e *Executor) Execute(ctx context.Context, toolUse *llm.ToolUseContent) (*l
return &llm.ToolResultContent{
ToolUseID: toolUse.ID,
Result: fmt.Sprintf("Failed to load tool: %v", err),
Summary: fmt.Sprintf("Failed to load tool: %v", err),
IsError: true,
StopIteration: false,
}, nil
}

// Execute the tool
stopIteration, result, err := tool.ExecuteRaw(ctx, toolUse.Input)
execResult, err := tool.ExecuteRaw(ctx, toolUse.Input)
if err != nil {
e.logger.Errorw("tool execution failed",
"tool_name", toolUse.Name,
Expand All @@ -88,6 +89,7 @@ func (e *Executor) Execute(ctx context.Context, toolUse *llm.ToolUseContent) (*l
return &llm.ToolResultContent{
ToolUseID: toolUse.ID,
Result: fmt.Sprintf("Tool execution error: %v", err),
Summary: fmt.Sprintf("Error: %v", err),
IsError: true,
StopIteration: false, // Don't stop on errors
}, nil
Expand All @@ -96,14 +98,16 @@ func (e *Executor) Execute(ctx context.Context, toolUse *llm.ToolUseContent) (*l
e.logger.Infow("tool executed successfully",
"tool_name", toolUse.Name,
"tool_use_id", toolUse.ID,
"stop_iteration", stopIteration,
"result_length", len(result))
"stop_iteration", execResult.StopIteration,
"result_length", len(execResult.Result),
"summary", execResult.Summary)

return &llm.ToolResultContent{
ToolUseID: toolUse.ID,
Result: result,
Result: execResult.Result,
Summary: execResult.Summary,
IsError: false,
StopIteration: stopIteration, // Preserve the stop signal
StopIteration: execResult.StopIteration,
}, nil
}

Expand Down
Loading
Loading