Repository navigation
Settle failed jobs on chain (abandonJob), skip stale session requests, remove-models CLI - #30
Merged
Merged
Conversation
… bits An envelope v4 job with output "public" now runs on a copy of the inference client pinned to the envelope's numCtx and numPredict, temperature 0, the deterministic seed, thinking off and the envelope's JSON-schema format; operator MODEL_OPTIONS do not apply. Stage 6 skips encryption and commits the canonical public response R (github.com/lightchain/pkg/publicresp): the terminal frame carries R, the blob anchors R, completeJob commits keccak256(R). The checkpoint, relay signature and settlement paths are unchanged because they only ever handled opaque bytes. A public job is refused without retry, so the fee is refunded, when the worker has PUBLIC_OUTPUT=false, when the job is a search job, when the envelope's params exceed OLLAMA_NUM_CTX or OLLAMA_NUM_PREDICT, or when the envelope is malformed. A generation that is not valid JSON under a format is committed as it came with valid false, never regenerated. Capabilities: "public-output" is declared unless PUBLIC_OUTPUT=false; "exact-profile" when EXACT_PROFILE=true (operator assertion of the pinned deterministic profile). Sessions that require them never land on a worker that would encrypt. Tests drive HandleTask against a real Ollama client on an httptest server and check the request body and the committed R field by field. Plan: lightchain-agents docs/plans/drafts/04-worker-disputer.md §3, §4, §7 item 2.
…ls CLI Three changes from the 2026-10-19 in-flight-cap incident (a-servers/ incident-2026-10-19-worker-inflight-cap.md): - A job that fails off-chain is now handed back with JobRegistry.abandonJob (best effort, in the background, behind ABANDON_FAILED_JOBS=true): the consumer's fee is refunded and the job leaves this worker's in-flight count at once instead of after deadline + resolutionTimeout. Before this a worker failing a few jobs an hour filled the maxConcurrentJobs cap and the chain refused its session claims, so its models timed out for users while the box looked healthy. Off by default until the abandonJob contract upgrade is live. - The session watcher drops a pending request older than SORTITION_MAX_REQUEST_AGE_BLOCKS (default 150): its consumer gave up after the 60 s claim timeout and never provides keys, so a late claim only spent gas (worker-4 claimed 35 abandoned requests in one burst at 00:40 UTC). - `lightchain-worker remove-models` (SUPPORTED_MODELS names the models to remove) calls WorkerRegistry.removeSupportedModel, because sortition routes by the registry and dropping a model from the unit env alone leaves its requests landing on this worker.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
From the 2026-10-19 finding (a-servers/incident-2026-10-19-worker-inflight-cap.md): g1 worker-4/5 sat at 100/100 in-flight jobs on chain because a job the worker fails off-chain is never settled, so
SessionManager.eligibleNowrefused their claims and every deepseek-r1:32b request timed out.Settle failed jobs.
reportFailurenow also callsJobRegistry.abandonJob(best effort, background, 90 s tx timeout) so the consumer is refunded and the in-flight slot frees at once. BehindABANDON_FAILED_JOBS=true, off by default: the function only exists from lightchain-contracts PR #18 on, and the binding comes from lightchain-shared-pkg PR #9. Implemented through an optionaljobAbandonerinterface so no test double changes.Skip stale session requests.
SORTITION_MAX_REQUEST_AGE_BLOCKS(default 150) drops a pending request that old without a claim; the consumer-api gave up after its 60 s claim timeout, so claiming it only spent gas. Two tests.remove-modelsCLI.SUPPORTED_MODELS=<names> lightchain-worker remove-modelscallsWorkerRegistry.removeSupportedModel; needed because sortition routes by the registry, not the unit env. Mirrorsadd-models.Checks:
GOOS=linux go build ./...andgo vet ./...clean against shared-pkg PR #9; test binaries for cli, pipeline, service and sortition compile; sortition, config, chain and registration tests pass on the dev box (release uses flock and only runs on Linux). CI needs shared-pkg PR #9 merged (or its ref) to build.