Skip to content

TaskEnvironment protocol + SQLite database env (#11, #9) - #52

Merged
askmy-stack merged 1 commit into
mainfrom
feat/parallax-11-9-task-env-database
Aug 5, 2026
Merged

TaskEnvironment protocol + SQLite database env (#11, #9)#52
askmy-stack merged 1 commit into
mainfrom
feat/parallax-11-9-task-env-database

Conversation

@askmy-stack

Copy link
Copy Markdown
Owner

Summary

  • Add shared TaskEnvironment protocol (reset / step / validate_success) under agentfailbench/environments/base.py
  • Wire the episode runner to type environments as TaskEnvironment; CustomerApiEnv documents protocol compliance
  • Implement deterministic in-memory SQLite investigation environment (list_tables, describe_table, execute_sql read-only, submit_answer)
  • Extend TaskSpec with optional expected_answer for investigation validators
  • Document the interface + env map in docs/benchmark-specification.md and README repo map
  • Unit tests cover protocol satisfaction and the normal (non-failure) DB success path

Closes #11
Closes #9

Test plan

  • make ci (ruff, mypy, pytest) — 50 passed
  • tests/unit/test_task_environment.py covers clean episode + write rejection + reset
  • Spot-check README / benchmark-spec render on GitHub

Made with Cursor

Formalize reset/step/validate_success across environments and ship a
deterministic in-memory database task with success validators (#11, #9).

Co-authored-by: Cursor <cursoragent@cursor.com>
Repository owner deleted a comment from cursor Bot Aug 5, 2026
@askmy-stack
askmy-stack merged commit 87aed7d into main Aug 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[M1] Formalize shared task interface across environments [M1] Implement deterministic database task environment

1 participant