feat: HuggingFace dataset integration#47
Merged
Conversation
Makes AppSprout/rle-benchmarks the public home for benchmark data: - hf_logger: build_dataset_card() renders the README as a live leaderboard from leaderboard.json (billed costs unmarked, estimates marked with ~, baseline-beaters called out); push_card/push_spread methods; text-only allow patterns; token + repo wired from RLEConfig (HF_TOKEN / HF_DATASET_REPO in .env). - scripts/push_hf.py: standalone push of an existing spread dir (card + history + baseline + runs/spread-<date>/) with --dry-run. - run_benchmark --push-hf: N<4 hard error relaxed to a warning (N=1 content spreads are valid dataset entries when labeled); baseline push now points at the static results/baseline. - mypy: per-module override for the untyped wandb/huggingface_hub call surfaces (tracking extra installed locally; CI skips it, so a type: ignore would be flagged unused there). 519 tests pass (+6 card-builder tests), mypy strict clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #45.
What
Wires the (previously never-functional) HF push end to end, targeting the real
AppSprout/rle-benchmarksdataset:build_dataset_card()— pure function rendering the README as a live leaderboard fromleaderboard.json: Hub frontmatter (mit/tags), ranked table, billed costs unmarked vs~-marked estimates (subscription models), baseline-beaters called out, N=1 framing, layout docs.HFLogger— token + repo fromRLEConfig(HF_TOKEN/HF_DATASET_REPOin .env; pydantic-settings doesn't export to os.environ so explicit passing is required), newpush_card/push_spreadmethods, text-only allow patterns (json/jsonl/csv/md/ts) so capture leftovers never upload.scripts/push_hf.py— push an existing run dir without re-running the benchmark;--dry-runlists exactly what would upload (verified: 235 files / 9.6MB spread + 16-file baseline + history).run_benchmark --push-hf— the--runs 4+hard error is now a labeled warning (N=1 content spreads are valid dataset entries; the card carries the framing), and the baseline push points at the staticresults/baselineinstead of the staleresults/baselines.disallow_untyped_calls=falsefor the two tracking modules: with the tracking extra installed locally, strict mode sees wandb/hf_hub's untyped APIs; CI doesn't install the extra, so inline ignores would be flagged unused there.Verification
519 tests pass (+6 card-builder tests), ruff clean, mypy strict clean, dry-run output verified. The inaugural real push (v0.3.0 spread + baseline + history) happens post-merge.
🤖 Generated with Claude Code