Skip to content

ci: standardize custom-code budget reporting - #1136

Open
apcha-oai wants to merge 1 commit into
mainfrom
castiron/promotions/pr-209
Open

apcha-oai wants to merge 1 commit into
mainfrom
castiron/promotions/pr-209

Conversation

@apcha-oai

Copy link
Copy Markdown
Contributor

Standardizes custom-code reporting, budget evaluation, publication tooling, and offline tests. PR evaluations use a captured main checkout and source-run head; publication retains pending results and rejects stale publications. Also clarifies stream-event visitor fallback comments.

The existing API reference, runtime behavior, public API, and repository-owned budget policy remain unchanged.

Validation: SDK formatting and all 59 offline custom-code tests passed locally, including the compiler hash contract.

Castiron-Internal-PR: openai/openai-java-internal#209
Castiron-Source-SHA: 23bc30f0693bf258a1b6ff2b68c402c7e7e09dae
Castiron-Public-Base-SHA: ef6d7ec
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Castiron custom code

✅ No new custom-code files detected.

90 mixed files remain; 0 existing customizations changed; 3 customizations removed; 7 generated baselines changed.

Compared ef6d7ec0aff7 → 6afee1091d77. Generated baselines verified.

File Result Current custom patch
openai-java-core/src/main/kotlin/com/openai/models/beta/responses/BetaResponseStreamEvent.kt Generated baseline changed +2 / −0
openai-java-core/src/main/kotlin/com/openai/models/beta/responses/BetaResponsesServerEvent.kt Generated baseline changed +2 / −0
openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseStreamEvent.kt Generated baseline changed +2 / −0
openai-java-core/src/main/kotlin/com/openai/models/responses/ResponsesServerEvent.kt Generated baseline changed +2 / −0
scripts/castiron/CUSTOM_CODE.md Now generated; already existed None
scripts/castiron/README.md Generated baseline changed None
scripts/castiron/custom_code_budget.py Now generated; already existed None
scripts/castiron/custom_code_report.py Generated baseline changed None
scripts/castiron/test_custom_code_budget.py Now generated; already existed None
scripts/castiron/test_custom_code_report.py Generated baseline changed None
86 existing customizations unchanged
  • openai-java-core/src/main/kotlin/com/openai/models/audio/AudioResponseFormat.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/agents/vaults/credentials/CredentialAuth.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/agents/vaults/credentials/CredentialAuthCreateParam.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/agents/vaults/credentials/CredentialAuthRotateParam.kt
  • openai-java-core/src/main/kotlin/com/openai/models/beta/responses/BetaResponse.kt
  • openai-java-core/src/main/kotlin/com/openai/models/chat/completions/ChatCompletionCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/chat/completions/ChatCompletionMessageFunctionToolCall.kt
  • openai-java-core/src/main/kotlin/com/openai/models/chat/completions/ChatCompletionToolMessageParam.kt
  • openai-java-core/src/main/kotlin/com/openai/models/embeddings/Embedding.kt
  • openai-java-core/src/main/kotlin/com/openai/models/embeddings/EmbeddingCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/Response.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseCreateParams.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseFunctionToolCall.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseFunctionWebSearch.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseInputItem.kt
  • openai-java-core/src/main/kotlin/com/openai/models/responses/ResponseTextConfig.kt
  • openai-java-core/src/main/kotlin/com/openai/models/videos/Video.kt
  • openai-java-core/src/main/kotlin/com/openai/models/webhooks/UnwrapWebhookEvent.kt
  • openai-java-core/src/main/kotlin/com/openai/models/webhooks/WebhookEndpointWithSecret.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/BetaServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/BetaServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/ImageServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/ResponseServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/ResponseServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/SkillServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/VideoServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/WebhookServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/WebhookServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/audio/TranscriptionServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/audio/TranslationServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/beta/agents/SessionServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/beta/agents/SessionServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/chat/ChatCompletionServiceAsync.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/finetuning/checkpoints/PermissionServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/async/skills/VersionServiceAsyncImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/BetaService.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/BetaServiceImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/ResponseService.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/ResponseServiceImpl.kt
  • openai-java-core/src/main/kotlin/com/openai/services/blocking/WebhookService.kt

46 more in the full report.

A changed generated baseline means this report cannot reliably identify which handwritten lines changed.

Inspect the custom-code diff

Download the exact patch produced by this run (requires repository access):

gh run download 37029182029 --repo openai/openai-java \
  --name castiron-custom-code-37029182029-1 --dir /tmp/castiron-custom-code-37029182029-1
git apply --stat /tmp/castiron-custom-code-37029182029-1/custom-code.patch
cat /tmp/castiron-custom-code-37029182029-1/custom-code.patch

Or reproduce it from an SDK checkout containing the vendored reporter:

git fetch --no-tags origin ef6d7ec0aff7eb6baa52090d839469cef85823ad 6afee1091d77c40873cf44abf4e982ba0908fc5e
python3 scripts/castiron/custom_code_report.py report \
  --base ef6d7ec0aff7eb6baa52090d839469cef85823ad \
  --head 6afee1091d77c40873cf44abf4e982ba0908fc5e --fetch --require-head-hash --public \
  --out /tmp/castiron-custom-code-6afee1091d77
cat /tmp/castiron-custom-code-6afee1091d77/custom-code.patch

This is the current full custom patch for mixed files, not an attribution of only the handwritten lines changed by this PR.

Full report and patch

@apcha-oai
apcha-oai marked this pull request as ready for review October 2, 2026 15:44
@apcha-oai
apcha-oai requested a review from a team as a code owner October 2, 2026 15:44
@apcha-oai
apcha-oai enabled auto-merge October 2, 2026 15:44
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-02T15:48:47.105391Z 6afee10 Draft marked ready
🔒 Security Review ✅ Completed 2026-10-02T15:49:41.893317Z 6afee10 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@openai-sdks

openai-sdks Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

OkTest Summary

✅ 236/236 SDK tests passed in 20.396s for Java SDK PR #1136.

Test results — 42 files
Test Result Time
tests/chat-completions-complex-body.test.ts ✅ Passed 953ms
tests/chat-completions-create.test.ts ✅ Passed 678ms
tests/chat-completions-stream.test.ts ✅ Passed 861ms
tests/files-content-binary.test.ts ✅ Passed 303ms
tests/files-create-multipart.test.ts ✅ Passed 507ms
tests/files-list-pagination.test.ts ✅ Passed 429ms
tests/initialize-config.test.ts ✅ Passed 196ms
tests/instance-isolation.test.ts ✅ Passed 171ms
tests/models-list.test.ts ✅ Passed 346ms
tests/responses-background-lifecycle.test.ts ✅ Passed 850ms
tests/responses-body-method-errors.test.ts ✅ Passed 609ms
tests/responses-cancel-timeout.test.ts ✅ Passed 206ms
tests/responses-cancel.test.ts ✅ Passed 247ms
tests/responses-compact-retries.test.ts ✅ Passed 410ms
tests/responses-compact.test.ts ✅ Passed 218ms
tests/responses-create-advanced-stream.test.ts ✅ Passed 677ms
tests/responses-create-advanced.test.ts ✅ Passed 1.867s
tests/responses-create-disconnect.test.ts ✅ Passed 1.083s
tests/responses-create-errors.test.ts ✅ Passed 397ms
tests/responses-create-malformed-api-responses.test.ts ✅ Passed 194ms
tests/responses-create-retries.test.ts ✅ Passed 315ms
tests/responses-create-stream-failures.test.ts ✅ Passed 271ms
tests/responses-create-stream-timeout.test.ts ✅ Passed 302ms
tests/responses-create-stream-wire.test.ts ✅ Passed 9.05s
tests/responses-create-stream.test.ts ✅ Passed 513ms
tests/responses-create-terminal-states.test.ts ✅ Passed 436ms
tests/responses-create-timeout.test.ts ✅ Passed 235ms
tests/responses-create.test.ts ✅ Passed 251ms
tests/responses-delete.test.ts ✅ Passed 447ms
tests/responses-input-items-errors.test.ts ✅ Passed 282ms
tests/responses-input-items-list.test.ts ✅ Passed 277ms
tests/responses-input-items-options.test.ts ✅ Passed 265ms
tests/responses-input-tokens-count-timeout.test.ts ✅ Passed 235ms
tests/responses-input-tokens-count.test.ts ✅ Passed 412ms
tests/responses-malformed-inputs.test.ts ✅ Passed 7.329s
tests/responses-not-found-errors.test.ts ✅ Passed 486ms
tests/responses-parse.test.ts ✅ Passed 636ms
tests/responses-retrieve-retries.test.ts ✅ Passed 282ms
tests/responses-retrieve.test.ts ✅ Passed 485ms
tests/responses-stored-method-errors.test.ts ✅ Passed 943ms
tests/retry-behavior.test.ts ✅ Passed 3.519s
tests/sdk-error-shape.test.ts ✅ Passed 388ms

View OkTest run #37029121300

SDK merge (46b64f77a903) · head (6afee1091d77) · base (ef6d7ec0aff7) · OkTest (6f7cbf36976b)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6afee1091d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

concurrency:
group: castiron-custom-code-comment-${{ github.event.workflow_run.head_repository.id }}-${{ github.event.workflow_run.head_branch }}
cancel-in-progress: false
queue: max # Preserve newer pending evaluations if older runs arrive out of order.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove the unsupported concurrency queue key

GitHub's workflow syntax only supports group and cancel-in-progress in the concurrency mapping; queue is not valid there. This makes the trusted workflow_run workflow invalid, preventing all custom-code status and comment jobs from running. Remove this key or implement ordering with supported workflow logic, and update the authoritative generator template rather than only its generated output.

AGENTS.md reference: AGENTS.md:L5-L8

Useful? React with 👍 / 👎.

Comment on lines +252 to +254
for (const pull of pulls) {
const {data: current} = await github.rest.pulls.get({...context.repo, pull_number: pull.number});
if (current.state !== 'open' || current.head.sha !== run.head_sha) continue;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Restrict failure comments to the source PR

When workflow_run.pull_requests is empty—common for fork runs—and a commit is associated with multiple PRs, this fallback loops over every associated PR and verifies only that it is open at the same SHA. A failed computation can therefore create or overwrite the Castiron comment on another PR sharing that commit, including one targeting a non-main branch because the producer has an unrestricted pull_request trigger. Restore the main-target/repository checks and require exactly one matching PR before writing, with a regression case covering multiple associations.

AGENTS.md reference: AGENTS.md:L41-L45

Useful? React with 👍 / 👎.

@markstuart-oai markstuart-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 6afee1091d77 against its base and the shared Python/Castiron implementation. The generated files match Python, but this promotion introduces 2 publication/discovery regressions relative to this SDK's previous guards, detailed inline. Please fix the canonical implementation and regenerate all SDKs together. The Kotlin edits are KDoc-only.

Source-only review; no repository workloads run. All reported hosted checks in the inspected exact-head snapshot were successful or skipped. I checked the moved test coverage and file sizes; there is no new 1,000-line crossing. The intentional captured-main behavior is not itself a finding. queue: max is supported by current GitHub concurrency documentation, so the existing unsupported-key warning is not a blocker.

});
for (const pull of pulls) {
const {data: current} = await github.rest.pulls.get({...context.repo, pull_number: pull.number});
if (current.state !== 'open' || current.head.sha !== run.head_sha) continue;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep failure reports behind the same live-run/PR validation

This replaces the previous live-run and unique-target checks with the event payload and an open/same-SHA test. A delayed attempt-1 publisher can reach trusted_report while attempt 2 is in progress: the live-status check raises, so compute fails and this fallback runs. With no newer comment marker yet, it can overwrite the report with an obsolete failure. The loop can also write to multiple same-SHA PRs, including a different base/source branch that the previous resolver rejected.

Please fix this in the shared template and regenerate: resolve the current source run/attempt and exactly one matching source-repository/branch PR targeting main once, and reuse that decision for success and failure publication. Recheck freshness before writing. Cover a newer retry still in progress and an unrelated same-SHA PR; the current failure fixtures omit these identity fields.

"GET", f"{root}/commits/{require_sha(run['head_sha'])}/pulls?per_page=100"
)
if associated:
return cast(list[dict[str, Any]], associated)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Fall back after validating commit-associated PRs

Previously an empty run association used the paginated, scoped open-branch lookup directly. Now any nonempty commit-association response returns before checking whether its PRs are still open/current and target main. If that response contains only a closed or stale PR, trusted_report filters it out and exits without ever discovering the valid current fork PR. The commit lookup also reads only its first 100 results. This can leave the current PR without its report and required budget statuses.

Please keep candidate discovery and current-target validation together in the canonical resolver: paginate, validate, and use the scoped open-branch lookup when no valid candidate remains (or keep the previous direct lookup). Preserve rejection of multiple valid targets and add a nonempty stale-association/valid-branch regression. Regenerate the SDK copies together.

@jbeckwith-oai jbeckwith-oai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on 6afee1091d77 for two independently confirmed regressions already described inline:

Please fix the canonical Castiron resolver/publisher and regenerate together. I have not duplicated the inline threads. Shared output matches the merged templates; the Kotlin changes are documentation-only. Hosted head checks passed or were skipped. Source-only review; no local contributor code executed.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants