Skip to content

feat(cloudflare): diagnose failed checks, and react when they fail #22

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
K-Mistele merged 1 commit into from
Oct 10, 2026
Merged

feat(cloudflare): diagnose failed checks, and react when they fail #22

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 35 additions & 8 deletions examples/alchemy-cloudflare/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,16 +155,42 @@ The AgentSession runs a [Fold](https://github.com/humanlayer/fold) agent on Open

**Its tools.**

| Kind | Tools |
| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Files and shell | `read`, `write`, `edit`, `apply_patch`, `bash` |
| Git, with the App's token | `git_fetch`, `git_pull`, `git_merge`, `git_push` (its own branch only, never forced) |
| GitHub | `github_discussion`, `github_comments`, `github_post_comment` (with `reply_to` for review threads), and, on pull requests, the diff, checks, reviews, and line comments |
| Progress | `update_plan`, a checklist comment the agent keeps up to date |
| Web | `web_search`, `web_fetch`, and the repository's skills |
| Kind | Tools |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Files and shell | `read`, `write`, `edit`, `apply_patch`, `bash` |
| Git, with the App's token | `git_fetch`, `git_pull`, `git_merge`, `git_push` (its own branch only, never forced) |
| GitHub | `github_discussion`, `github_comments`, `github_post_comment` (with `reply_to` for review threads), and, on pull requests, the diff, reviews, and line comments |
| Checks | `github_pull_request_checks`, and `github_check_failure` for a failed check's output, annotations, failed steps, and log (the whole log is saved in the workspace) |
| Progress | `update_plan`, a checklist comment the agent keeps up to date |
| Web | `web_search`, `web_fetch`, and the repository's skills |

Its final answer is posted for it. It uses `github_post_comment` for anything else, such as answering each line comment in its own thread.

## When checks fail

Once the agent has been mentioned on a pull request, the app follows it. When a check fails on the pull request's latest commit, the agent looks into it without being asked.

```mermaid
sequenceDiagram
autonumber
participant GH as GitHub
participant MB as DeliveryMailbox
participant AS as AgentSession

GH->>MB: check_run completed: failure
MB->>GH: is it on the latest commit?
Note over MB: older commits, passing,<br/>and skipped checks are ignored
MB->>AS: "these checks failed" + delivery
AS->>GH: github_check_failure: output,<br/>annotations, failed steps, log
alt the fix is clear
AS->>GH: commit and push to the branch
end
AS->>MB: complete(cause and fix)
MB->>GH: post it on the pull request
```

Checks that fail together arrive as one request. A check that fails after the agent pushes a fix starts another turn, so a fix the agent can't get right keeps coming back to it.

## Labels

New issues and pull requests are labeled by [Workers AI's Clef](https://developers.cloudflare.com/workers-ai/) model, which scores each of GitHub's default labels. A label scoring 0.7 or more is added. This needs no mention and no agent.
Expand All @@ -174,14 +200,15 @@ New issues and pull requests are labeled by [Workers AI's Clef](https://develope
```text
src/
├── Worker.ts Ingress Worker: webhook routes and the delivery API
├── GithubBot.ts GitHub callbacks: labels, mention access, the request, handoff
├── GithubBot.ts GitHub callbacks: labels, mentions, failed checks, handoff
├── DeliveryMailboxDO.ts the mailbox Durable Object
├── AgentSessionDO.ts the agent: model, tools, system prompt, session start and resume
├── DeliveryTurn.ts one delivery's turn: send the prompt, report back, recover after restarts
├── DiscussionContext.ts what's new in the discussion
├── GitHubTools.ts, GitTools.ts the agent's GitHub and git tools
├── DeliveryPlanTool.ts update_plan
├── BashTool.ts bash on the shell or the container
├── CheckTools.ts a pull request's checks, and why one failed
├── Workspace.ts the AgentSession's view of its Computer
├── AutoLabel.ts Workers AI labels
└── computer/ the Computer Durable Object, its Worker, and the container image
Expand Down
8 changes: 7 additions & 1 deletion examples/alchemy-cloudflare/src/AgentSessionDO.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ import { Config, Context, Effect, FileSystem, Layer, Match, Option, Path, Predic
import { HttpClient } from 'effect/http'

import { bashTool } from './BashTool'
import { checkTools } from './CheckTools'
import { WORKSPACE_ROOT } from './computer/Contract'
import { DeliveryApi } from './DeliveryApi'
import { updatePlanTool } from './DeliveryPlanTool'
Expand Down Expand Up @@ -85,6 +86,10 @@ export const workBranchFor = (discussion: GitHubDiscussion) =>
}),
)

/** For a pull request, the tools for its checks. */
const pullRequestCheckTools = ({ githubDiscussion }: AgentSessionMetadata) =>
Predicate.isTagged(githubDiscussion, 'GitHubPullRequest') ? checkTools(githubDiscussion) : []

/** For an issue with a branch, the tool that opens its pull request. */
const issuePullRequestTools = ({ githubDiscussion, workBranch, defaultBranch }: AgentSessionMetadata) =>
Predicate.isTagged(githubDiscussion, 'GitHubIssue') && Predicate.isNotNull(workBranch)
Expand All @@ -107,7 +112,7 @@ const branchPolicy = ({ githubDiscussion, branch, defaultBranch, workBranch }: A
GitHubIssue: () =>
`When the work is ready for review, push it, then open a pull request with github_create_pull_request; if one is already open from ${workBranch}, it returns that one. Put the pull request's link in your answer.`,
GitHubPullRequest: () =>
`${workBranch} is this pull request's branch, so pushing updates the pull request.`,
`${workBranch} is this pull request's branch, so pushing updates the pull request. Use github_pull_request_checks and github_check_failure to see why a check failed.`,
}),
),
`To bring in the latest ${defaultBranch}, git_fetch it, then git_merge origin/${defaultBranch}.`,
Expand Down Expand Up @@ -218,6 +223,7 @@ const AgentConversationLive = Layer.effect(
...githubTools(metadata.githubDiscussion),
...gitTools({ repository: metadata.githubDiscussion.ref, workBranch: metadata.workBranch }),
...issuePullRequestTools(metadata),
...pullRequestCheckTools(metadata),
],
})

Expand Down
32 changes: 2 additions & 30 deletions examples/alchemy-cloudflare/src/BashTool.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,18 @@
*/
import { OutputStore } from '@humanlayer/fold-agent'
import {
CurrentToolCall,
defaultMaxBytes,
defaultMaxLines,
defineTool,
formatSize,
ToolResultFailure,
ToolResultText,
truncateTail,
type FoldTool,
} from '@humanlayer/fold-core'
import { Effect, Schema } from 'effect'

import { type CommandOutput, WORKSPACE_ROOT } from './computer/Contract'
import { visibleOutput } from './ToolOutput'
import { Workspace } from './Workspace'

const DEFAULT_TIMEOUT_MS = 120_000
Expand Down Expand Up @@ -71,33 +70,6 @@ const DESCRIPTION =
const combinedOutput = ({ stdout, stderr }: CommandOutput) =>
stdout.length > 0 && stderr.length > 0 ? `${stdout.replace(/\n?$/, '\n')}${stderr}` : stdout + stderr

/**
* The output as the model sees it: its tail, and when that is not all of it, where the whole output is
* saved. Saving is best-effort; the notice says when it failed.
*/
const visibleOutput = (text: string) =>
Effect.gen(function* () {
const truncation = truncateTail(text)
if (!truncation.truncated) return text
const outputStore = yield* OutputStore
const { toolCallId } = yield* CurrentToolCall
const saved = yield* outputStore.append(toolCallId, text).pipe(
Effect.tap((ref) =>
Effect.logInfo('bash.output_saved').pipe(
Effect.annotateLogs({ path: ref.path, lines: truncation.totalLines, bytes: text.length }),
),
),
Effect.map((ref) => `Full output: ${ref.path}`),
Effect.catch((error) =>
Effect.logWarning('bash.output_save failed', error).pipe(
Effect.as('The full output could not be saved'),
),
),
)
const start = truncation.totalLines - truncation.outputLines + 1
return `${truncation.content}\n\n[Showing lines ${start}-${truncation.totalLines} of ${truncation.totalLines}. ${saved}]`
})

/** The bash tool, over the session's {@link Workspace}, saving long output to the {@link OutputStore}. */
export const bashTool: FoldTool<Workspace | OutputStore> = defineTool({
name: 'bash',
Expand Down Expand Up @@ -127,7 +99,7 @@ export const bashTool: FoldTool<Workspace | OutputStore> = defineTool({
timeoutMs,
})
.pipe(Effect.mapError((error) => `The shell could not run the command: ${error.message}`))
const text = (yield* visibleOutput(combinedOutput(output))).replace(/\n+$/, '')
const text = (yield* visibleOutput(combinedOutput(output), 'bash')).replace(/\n+$/, '')

if (output.status === 'cancelled') {
return yield* Effect.fail(`${text}\n\nCommand timed out after ${timeoutMs} milliseconds`)
Expand Down
129 changes: 129 additions & 0 deletions examples/alchemy-cloudflare/src/CheckTools.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
/**
* The agent's tools for a pull request's checks: what ran on its latest commit, and why one failed. A failed
* check's full job log is saved in the workspace, and the agent sees its end.
*/
import {
GitHubCheckRun,
GitHubCheckRunRef,
GitHubId,
type GitHubApi,
type GitHubCheckAnnotation,
type GitHubCheckRunInfo,
type GitHubPullRequest,
} from '@humanlayer/channels-github'
import type { OutputStore } from '@humanlayer/fold-agent'
import { defineTool, ToolResultFailure, ToolResultText, type FoldTool } from '@humanlayer/fold-core'
import { Array as Arr, Effect, Predicate, Schema } from 'effect'

import { visibleOutput } from './ToolOutput'
import { NoParameters } from './ToolParameters'

/** Longest check summary or output text shown in full. */
const MAX_TEXT_LENGTH = 2_000
/** Most annotations listed for one check. */
const MAX_ANNOTATIONS = 50

const clip = (text: string | null) =>
Predicate.isNull(text) || text.length <= MAX_TEXT_LENGTH ? text : `${text.slice(0, MAX_TEXT_LENGTH)}…`

const failure = (error: { readonly message: string }) =>
ToolResultFailure.make({ text: `GitHub request failed: ${error.message}` })

/** A check as the agent sees it in a list. */
const checkSummary = (check: GitHubCheckRunInfo) => ({
id: check.ref.id,
name: check.name,
status: check.status,
conclusion: check.conclusion,
title: check.outputTitle,
summary: clip(check.outputSummary),
annotations: check.annotationCount,
url: check.detailsUrl ?? check.url,
})

/** The checks that ran on the pull request's latest commit, each with its ID for `github_check_failure`. */
export const listPullRequestChecks = Effect.fn('agent_session.list_checks')(function* (pullRequest: GitHubPullRequest) {
const { headSha } = yield* pullRequest.fetchInfo()
const runs = yield* pullRequest.listCheckRunsForRef(headSha)
const checks = yield* Effect.forEach(runs, (run) => run.fetchInfo(), { concurrency: 8 })
const current = checks.filter((check) => check.headSha === headSha).map(checkSummary)
return ToolResultText.make({ text: JSON.stringify({ headSha, checks: current }, null, 2) })
})

const annotationLine = (annotation: GitHubCheckAnnotation) => {
const lines =
annotation.startLine === annotation.endLine
? `${annotation.startLine}`
: `${annotation.startLine}-${annotation.endLine}`
const text = [annotation.title, annotation.message].filter(Predicate.isNotNull).join(': ')
return `- ${annotation.level ?? 'note'} at ${annotation.path}:${lines}: ${text}`
}

/**
* Why a check failed: its result and output, its annotations, and, for a GitHub Actions job, its failed
* steps and the end of its log, with the whole log saved in the workspace.
*/
export const describeCheckFailure = Effect.fn('agent_session.describe_check_failure')(function* (
pullRequest: GitHubPullRequest,
checkRunId: number,
) {
const { installationId, repositoryId, owner, repository } = pullRequest.ref
const checkRun = GitHubCheckRun.make({
ref: GitHubCheckRunRef.make({ installationId, repositoryId, owner, repository, id: GitHubId.make(checkRunId) }),
})
const info = yield* checkRun.fetchInfo()
const annotations = info.annotationCount > 0 ? yield* checkRun.listAnnotations() : []
const job = yield* checkRun.resolveActionsJob()

const sections: Array<string> = [
`Check "${info.name}" (check run ${info.ref.id}) on ${info.headSha.slice(0, 7)}: ${info.conclusion ?? info.status}. ${info.detailsUrl ?? info.url ?? ''}`.trim(),
]
const output = [info.outputTitle, clip(info.outputSummary), clip(info.outputText)].filter(Predicate.isNotNull)
if (Arr.isArrayNonEmpty(output)) sections.push(`Output:\n${output.join('\n\n')}`)
if (Arr.isReadonlyArrayNonEmpty(annotations)) {
const listed = annotations.slice(0, MAX_ANNOTATIONS).map(annotationLine)
const more = annotations.length > MAX_ANNOTATIONS ? [`- …and ${annotations.length - MAX_ANNOTATIONS} more`] : []
sections.push(`Annotations:\n${[...listed, ...more].join('\n')}`)
}
if (Predicate.isNull(job)) {
sections.push('This check is not a GitHub Actions job, so there is no job log.')
return ToolResultText.make({ text: sections.join('\n\n') })
}
const jobInfo = yield* job.fetchInfo()
const failedSteps = jobInfo.steps.filter(
(step) =>
Predicate.isNotNull(step.conclusion) && step.conclusion !== 'success' && step.conclusion !== 'skipped',
)
sections.push(
Arr.isReadonlyArrayNonEmpty(failedSteps)
? `Failed steps in job "${jobInfo.name}":\n${failedSteps.map((step) => `- ${step.number}. ${step.name}: ${step.conclusion}`).join('\n')}`
: `Job "${jobInfo.name}" has no failed steps.`,
)
const log = yield* job.downloadLog()
sections.push(`Job log:\n${yield* visibleOutput(log, 'github_check_failure')}`)
return ToolResultText.make({ text: sections.join('\n\n') })
})

/** The check tools for a pull request session. */
export const checkTools = (pullRequest: GitHubPullRequest): ReadonlyArray<FoldTool<GitHubApi | OutputStore>> => [
defineTool({
name: 'github_pull_request_checks',
description:
"List the checks on the pull request's latest commit: each one's ID, name, status, result, summary, annotation count, and link.",
parameters: NoParameters,
success: ToolResultText,
failure: ToolResultFailure,
handler: () => listPullRequestChecks(pullRequest).pipe(Effect.mapError(failure)),
}),
defineTool({
name: 'github_check_failure',
description:
"Find out why a check failed: its output, its annotations, and, for a GitHub Actions job, the failed steps and the end of the job's log. The whole log is saved in the workspace.",
parameters: Schema.Struct({
check_run_id: GitHubId.annotate({ description: 'The check run ID, from github_pull_request_checks' }),
}),
success: ToolResultText,
failure: ToolResultFailure,
handler: ({ check_run_id }) => describeCheckFailure(pullRequest, check_run_id).pipe(Effect.mapError(failure)),
}),
]
12 changes: 0 additions & 12 deletions examples/alchemy-cloudflare/src/GitHubTools.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import {
GitHubCheckRuns,
GitHubContent,
GitHubId,
GitHubIssueInfo,
Expand Down Expand Up @@ -169,17 +168,6 @@ export const githubTools = (discussion: GitHubDiscussion): ReadonlyArray<FoldToo
Effect.mapError(failure),
),
}),
defineTool({
name: 'github_pull_request_checks',
description: 'List check runs for the current GitHub pull request.',
parameters: NoParameters,
success: ToolResultText,
failure: ToolResultFailure,
handler: () =>
pullRequest
.listCheckRuns()
.pipe(Effect.flatMap(jsonResult(GitHubCheckRuns)), Effect.mapError(failure)),
}),
defineTool({
name: 'github_pull_request_reviews',
description:
Expand Down
Loading
Loading