Skip to content

Metadata tagging & validation silently mislabel all newly-processed questions (gpt-5-mini reasoning starves the token budget) #249

Description

@drozado
  • After the “rewrite LLM forecaster” refactor (6566923, 2026-06-26), metadata jobs call gpt-5-mini with low max_output_tokens and no reasoning.effort. Since the model defaults to medium reasoning, every request exhausts the budget and fails with status=incomplete, reason=max_output_tokens. After retries, tag_questions / validate_questions fall back to category="Other" and valid_question=True, silently mislabeling new questions.

  • A Kalshi tagging run made 120 requests; 0 succeeded, then hit the 30-minute timeout.

  • Questions processed through this path since 2026-06-26 may be contaminated.

  • Fix: Set reasoning={"effort": "minimal"} and raise the tagging/validation output budgets.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions