Skip to content

fix(api)!: represent multi-graders as named grader maps - #729

Open
markstuart-oai wants to merge 1 commit into
mainfrom
castiron/promotions/pr-129
Open

markstuart-oai wants to merge 1 commit into
mainfrom
castiron/promotions/pr-129

Conversation

@markstuart-oai

@markstuart-oai markstuart-oai commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Summary

Changes multi-grader inputs to keyed collections of graders.

Changes

  • Use dictionary keys as variables in multi-grader formulas.

Migration and validation

Pass a named grader hash, for example graders: {quality: grader}, and refer to the same key in calculate_output: "quality". The single-value union is now MultiGrader::Grader (previously MultiGrader::Graders); its RBS alias likewise changes from graders to grader.

Pre-publication validation of this updated candidate passed the complete Ruby 3.3, 3.4 and 4.0 suites, Bedrock tests, lint, RBI/RBS checks and package build. A local full-import consumer passed 26 assertions, including all five grader variants, typed round trips and three captured SDK requests; existing session-reset controls passed 16 assertions. No live API calls or package releases were made.

The committed regression tests cover both string and symbol dictionary keys, all five typed values, round trips, exact Run/Validate request envelopes, and invalid flat/array controls. They pass 3 tests / 27 assertions on this candidate and fail against the unchanged public baseline. Formatting, focused lint, Sorbet and RBS checks pass locally.

Public CI passed Ruby 3.3/3.4/4.0, Bedrock, lint, RBI/RBS validation and packaging checks, including the new named-map regression tests.

Castiron-Internal-PR: openai/openai-ruby-internal#129
Castiron-Source-SHA: 8ebce350cab9bb13fc02e62385fd567346d44d85
Castiron-Public-Base-SHA: 3343547
@github-actions

github-actions Bot commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Castiron custom code

✅ No new custom-code files detected.

70 mixed files remain; 0 existing customizations changed.

Compared 33435479c2d934732dbf341e. Generated baselines verified.

70 existing customizations unchanged
  • lib/openai.rb
  • lib/openai/client.rb
  • lib/openai/models/audio/transcription_create_response.rb
  • lib/openai/models/beta/agents/vault_status_filter.rb
  • lib/openai/models/chat/chat_completion_message.rb
  • lib/openai/models/chat/chat_completion_message_function_tool_call.rb
  • lib/openai/models/chat/completion_create_params.rb
  • lib/openai/models/image_edit_completed_event.rb
  • lib/openai/models/image_edit_partial_image_event.rb
  • lib/openai/models/image_gen_completed_event.rb
  • lib/openai/models/image_gen_partial_image_event.rb
  • lib/openai/models/images_response.rb
  • lib/openai/models/response_format_json_schema.rb
  • lib/openai/models/responses/function_tool.rb
  • lib/openai/models/responses/response.rb
  • lib/openai/models/responses/response_create_params.rb
  • lib/openai/models/responses/response_format_text_config.rb
  • lib/openai/models/responses/response_format_text_json_schema_config.rb
  • lib/openai/models/responses/response_function_tool_call.rb
  • lib/openai/models/responses/response_function_web_search.rb
  • lib/openai/models/responses/response_output_text.rb
  • lib/openai/models/responses/tool.rb
  • lib/openai/models/webhooks/webhook_endpoint_with_secret.rb
  • lib/openai/resources/beta/agents/sessions.rb
  • lib/openai/resources/beta/threads.rb
  • lib/openai/resources/chat/completions.rb
  • lib/openai/resources/containers/files.rb
  • lib/openai/resources/files.rb
  • lib/openai/resources/responses.rb
  • lib/openai/resources/vector_stores/file_batches.rb
  • lib/openai/resources/vector_stores/files.rb
  • lib/openai/resources/webhooks.rb
  • rbi/openai/client.rbi
  • rbi/openai/models/beta/agents/vault_status_filter.rbi
  • rbi/openai/models/chat/chat_completion_message.rbi
  • rbi/openai/models/chat/chat_completion_message_function_tool_call.rbi
  • rbi/openai/models/chat/completion_create_params.rbi
  • rbi/openai/models/image_edit_completed_event.rbi
  • rbi/openai/models/image_edit_partial_image_event.rbi
  • rbi/openai/models/image_gen_completed_event.rbi

30 more in the full report.

A changed generated baseline means this report cannot reliably identify which handwritten lines changed.

Inspect the custom-code diff

Download the exact patch produced by this run (requires repository access):

gh run download 35527220421 --repo openai/openai-ruby \
  --name castiron-custom-code-35527220421-1 --dir /tmp/castiron-custom-code-35527220421-1
git apply --stat /tmp/castiron-custom-code-35527220421-1/custom-code.patch
cat /tmp/castiron-custom-code-35527220421-1/custom-code.patch

Or reproduce it from an SDK checkout containing the vendored reporter:

git fetch --no-tags origin 33435479c2d98603b93c810183d314bc55cd1ee7 34732dbf341e7029c4e0ba3d8468400ac2e33261
python3 scripts/castiron/custom_code_report.py report \
  --base 33435479c2d98603b93c810183d314bc55cd1ee7 \
  --head 34732dbf341e7029c4e0ba3d8468400ac2e33261 --fetch --require-head-hash --public \
  --out /tmp/castiron-custom-code-34732dbf341e
cat /tmp/castiron-custom-code-34732dbf341e/custom-code.patch

This is the current full custom patch for mixed files, not an attribution of only the handwritten lines changed by this PR.

Full report and patch

@markstuart-oai
markstuart-oai marked this pull request as ready for review September 20, 2026 17:51
@markstuart-oai
markstuart-oai requested a review from a team as a code owner September 20, 2026 17:51
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 20, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-20T17:52:58.082445Z 34732db Draft marked ready
🔒 Security Review Completed 2026-09-20T17:53:41.724991Z 34732db Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@openai-sdks

openai-sdks Bot commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

OkTest Summary

236/236 SDK tests passed in 7.675s for Ruby SDK PR #729.

Test results — 42 files
Test Result Time
tests/chat-completions-complex-body.test.ts ✅ Passed 115ms
tests/chat-completions-create.test.ts ✅ Passed 105ms
tests/chat-completions-stream.test.ts ✅ Passed 103ms
tests/files-content-binary.test.ts ✅ Passed 125ms
tests/files-create-multipart.test.ts ✅ Passed 196ms
tests/files-list-pagination.test.ts ✅ Passed 191ms
tests/initialize-config.test.ts ✅ Passed 77ms
tests/instance-isolation.test.ts ✅ Passed 99ms
tests/models-list.test.ts ✅ Passed 186ms
tests/responses-background-lifecycle.test.ts ✅ Passed 87ms
tests/responses-body-method-errors.test.ts ✅ Passed 257ms
tests/responses-cancel-timeout.test.ts ✅ Passed 177ms
tests/responses-cancel.test.ts ✅ Passed 191ms
tests/responses-compact-retries.test.ts ✅ Passed 169ms
tests/responses-compact.test.ts ✅ Passed 165ms
tests/responses-create-advanced-stream.test.ts ✅ Passed 84ms
tests/responses-create-advanced.test.ts ✅ Passed 111ms
tests/responses-create-disconnect.test.ts ✅ Passed 205ms
tests/responses-create-errors.test.ts ✅ Passed 330ms
tests/responses-create-malformed-api-responses.test.ts ✅ Passed 106ms
tests/responses-create-retries.test.ts ✅ Passed 450ms
tests/responses-create-stream-failures.test.ts ✅ Passed 101ms
tests/responses-create-stream-timeout.test.ts ✅ Passed 190ms
tests/responses-create-stream-wire.test.ts ✅ Passed 1.633s
tests/responses-create-stream.test.ts ✅ Passed 95ms
tests/responses-create-terminal-states.test.ts ✅ Passed 162ms
tests/responses-create-timeout.test.ts ✅ Passed 176ms
tests/responses-create.test.ts ✅ Passed 133ms
tests/responses-delete.test.ts ✅ Passed 162ms
tests/responses-input-items-errors.test.ts ✅ Passed 128ms
tests/responses-input-items-list.test.ts ✅ Passed 126ms
tests/responses-input-items-options.test.ts ✅ Passed 105ms
tests/responses-input-tokens-count-timeout.test.ts ✅ Passed 206ms
tests/responses-input-tokens-count.test.ts ✅ Passed 294ms
tests/responses-malformed-inputs.test.ts ✅ Passed 1.828s
tests/responses-not-found-errors.test.ts ✅ Passed 204ms
tests/responses-parse.test.ts ✅ Passed 145ms
tests/responses-retrieve-retries.test.ts ✅ Passed 110ms
tests/responses-retrieve.test.ts ✅ Passed 159ms
tests/responses-stored-method-errors.test.ts ✅ Passed 538ms
tests/retry-behavior.test.ts ✅ Passed 3.183s
tests/sdk-error-shape.test.ts ✅ Passed 284ms

View OkTest run #35527186397

SDK merge (ef3fb559a60a) · head (34732dbf341e) · base (33435479c2d9) · OkTest (a0be4375e02d)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants