Skip to content

Retry a cut-off answer, and wait out a rate limit on its own - #11

Merged
jeffwall-curlewlabs merged 2 commits into
mainfrom
fix/retry-a-cut-off-answer
Oct 14, 2026
Merged

jeffwall-curlewlabs merged 2 commits into
mainfrom
fix/retry-a-cut-off-answer

Conversation

@jeffwall-curlewlabs

Copy link
Copy Markdown
Contributor

gh_api retries a busy or unreachable API, but not an answer that arrived empty or cut off. gh decodes an answer before it reports a status whenever it filters one with --jq, reads a GraphQL answer, or reads a JSON error body (gh 2.101.0's processResponse; go-gh's jq.EvaluateFormatted). A short body then fails with Go's decoder error and nothing else on stderr:

unexpected end of JSON input

That counted as a refusal, so it failed on the first attempt. A clock host logged it once on the mark read: the tick sent nothing and exited 1, and the same read succeeded a few minutes later. Since gh prints no status here, the log can't say whether the answer was an error with an empty body or a success that was cut short.

Change

  • A status-less failure whose message is exactly unexpected end of JSON input is now transient. The match is exact so that other status-less failures, such as a usage error, stay refusals.
  • The busy ladder's waits go from 5, 20, 60 to 1, 2, 4, 8, 10, 10, 10. Most of these failures are one lost answer that the next attempt gets, and the total wait stays under a minute.
  • A rate limit (a 429, or a 403 that says "secondary rate limit") has its own path now: one retry, after 60s. The old ladder's last step was 60s because that is GitHub's documented minimum wait after a secondary rate limit. The short waits would spend every retry with no chance of clearing the limit, while sending the requests GitHub warns can get an integration banned.
  • gh_api takes the runner as a parameter, as it already took sleep, so the retry path is tested without a network.
  • The live job's cap goes from 20 to 25 minutes. The worst case for a failing call is now about 9.75 minutes: every busy attempt times out at 60s, plus the rate-limit wait. The cap still has to hold that twice, for the check and for its cleanup.

Retrying a request that may already have applied isn't new, and the callers cover it:

  • A mark claim reads back whose blob the mark holds.
  • A ledger create or delete takes its 422 as done.
  • A second dispatch of a slot is one more delivery, which the concurrency group and the slot action run once.

Tests

  • The classifier treats the cut-off line as transient and as not a rate limit. The rate-limit lines are rate limits and not transient, and every refusal is neither.
  • gh_api against a stand-in gh: a cut-off answer is retried after the first busy wait; a busy API is retried once per ladder step and then raises; a rate limit waits RATE_LIMIT_DELAY_SECONDS, is retried once, and a second one raises.
  • I checked that the tests can fail. With the cut-off match removed, two tests fail. With rate limits routed through the busy ladder, the rate-limit test fails.

I ran these on a Mac: the unit tests on Python 3.14, ruff format --check, ruff check, mypy and actionlint. I didn't run the live check locally; CI runs it.

The version goes to 0.2.3.

gh_api retried a busy or unreachable API, but not an answer that arrived
empty or cut off. gh decodes an answer before it reports a status whenever
it filters one with --jq, reads a GraphQL answer, or reads a JSON error
body, so such an answer fails with Go's decoder error and nothing else:

  unexpected end of JSON input

That counted as a refusal and failed on the first attempt. On a clock host
it happened to the mark read, so the tick sent nothing and exited 1; the
same read succeeded minutes later. It is now retried like a request that got
no answer, and its callers already cover a request that applied before its
answer was lost: a mark claim reads back whose blob the mark holds, a ledger
create or delete takes its 422 as done, and a second dispatch of a slot is
one more delivery, which the concurrency group and slot action run once.

The busy ladder also gets shorter waits, 1, 2, 4, 8, 10, 10 and 10 seconds,
since most of these failures are one lost answer that the next attempt gets.
Its old last step was 60s because that is GitHub's documented minimum wait
after a secondary rate limit, which the new ladder no longer reaches, so a
rate limit now has its own path: one retry, after 60s. Retrying it on the
short waits would spend the attempts with no chance of clearing the limit,
while sending the requests GitHub warns can get an integration banned.

A call can now spend longer before it fails, so the live job's cap moves
from 20 to 25 minutes to keep holding it twice, for the check and for its
cleanup.

The version goes to 0.2.3.
gh reports a failed request's status in one of two forms. With a message
from the error body it prints `gh: <message> (HTTP <status>)`; with none,
as for a 5xx whose body is empty or not JSON, it falls back to the status
alone (gh 2.101.0's processResponse):

  gh: HTTP 500

classify only read the first form, so the second came back status-less,
counted as a refusal, and failed on the first attempt. The live check on
this branch hit it on a ref delete during its cleanup. Both forms now yield
the status, so a bare 5xx gets the busy ladder, and a bare 404 or 422
reaches the callers that act on those statuses as an HttpError. A status
without a message has an empty one, and HttpError's text leaves the empty
message out.
@jeffwall-curlewlabs

Copy link
Copy Markdown
Contributor Author

The first run's live check failed in its cleanup. A ref delete got this from GitHub:

gh: HTTP 500

It failed on the first attempt (the whole step took 39s, shorter than the busy ladder's waits). That's an older gap of the same kind as this PR's fix. When an error body holds no message gh can read, such as a 5xx with an empty or non-JSON body, gh 2.101.0 prints the status alone (processResponse falls back to HTTP %d). classify only read the gh: <message> (HTTP <status>) form, so this line came back status-less and counted as a refusal.

cce9f59 reads both forms:

  • A bare 5xx now gets the busy ladder.
  • A bare 404 or 422 now reaches the callers that act on those statuses as an HttpError, instead of a plain GitHubError.
  • HttpError's text leaves out an empty message.

The gh: HTTP 500 line is now a case in the classifier test. Against the previous regex it reads as not transient.

Still not handled, and not seen anywhere: with --jq, a 5xx whose body is HTML would fail in gh's JSON decoder with invalid character '<' ... before gh prints any status. I left that out because nothing has logged it.

@jeffwall-curlewlabs
jeffwall-curlewlabs merged commit 6655172 into main Oct 14, 2026
10 checks passed
@jeffwall-curlewlabs
jeffwall-curlewlabs deleted the fix/retry-a-cut-off-answer branch October 14, 2026 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant