feat: atif converter - #205
Conversation
6e2760c to
3332ae8
Compare
3332ae8 to
1a4400c
Compare
Seven docs were reflowed by a formatter run over their code blocks. Stripping whitespace leaves only added trailing commas, so none of the prose or examples changed meaning. Out of scope for this PR.
1a4400c to
83e99b3
Compare
b083698 to
ed29f97
Compare
develra
left a comment
develra
left a comment
There was a problem hiding this comment.
Plan/ atif_mapping.md look good to me! I didn't spend too much time on the logic as it's unlikely I would catch a mistake.
|
|
||
| A field the converter cannot map is skipped, logged, and recorded in the file. Conversion failing must never cost someone a finished run. | ||
|
|
||
| Every skip appends to `extra.kbench.conversion_warnings`: |
There was a problem hiding this comment.
yea this seems like a good approach
|
|
||
| How a `*.run.json` becomes the trajectory and result files harbor tooling reads — the general rules first, then the shapes that do not map one to one. | ||
|
|
||
| The converter is `src/kaggle_benchmarks/kaggle/atif.py`. This document describes what it does today, not what it might do. |
There was a problem hiding this comment.
Not suggesting we do it in this PR but I wonder if a nice way to validate might be to also write a ATIF -> run.json mapper that inverts this and makes sure they match... probably more effort than it's worth - just a thought.
| |----------|------|-------| | ||
| | `pyRunId` | `session_id`, `trajectory_id`, `trial_name` | Falls back to `id`, then `"unknown"` | | ||
| | `taskVersion.name` | `task_name` | Required | | ||
| | `modelVersion.slug` | `agent.model_name`, `agent_info.model_info.name` | | |
There was a problem hiding this comment.
does harbor support having different models for the judge?
There was a problem hiding this comment.
Good catch. Yes atif supports it and this is actually Agent.model_name, not Agent.name. And we already filled it. But here we should change the message [delegated] Other trajectories from this run: judge, judge.. Done.
| "path": "https://www.kaggle.com/static/images/site-logo.png"}}] | ||
| ``` | ||
|
|
||
| Everything else becomes a text marker plus a warning: |
There was a problem hiding this comment.
I could see longer term it being nice to have some better solution here where we like co-locate images with the ATIF and point to those, but I don't think it's worth designing for now.
There was a problem hiding this comment.
Yes Harbor is already doing this, e.g its antigravity adapter writes inline images to a relative images/ folder and points ImageSource.path there.
Maybe leave it for now because our writers generates 2 files instead of arbitrarily many. And the raw data in extra.kbench_media are not lost.
2 participants
Add a converter for run.json -> atif, so it is compatible with ATIF analysis tools. It's on by default, and users can override it by
WRITE_ATIF=falseenvar.