Skip to content

feat(experimentalist): add Harbor examples - #1050

Open
ngoncharenko wants to merge 4 commits into
mainfrom
ngoncharenko/aalgo-312-examples
Open

feat(experimentalist): add Harbor examples#1050
ngoncharenko wants to merge 4 commits into
mainfrom
ngoncharenko/aalgo-312-examples

Conversation

@ngoncharenko

@ngoncharenko ngoncharenko commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

It's a follow up for this thread: #955 (comment)

  • add a deterministic hello-harbor-agent example with local train and validation tasks, exact-output verifiers, traces, and an optimizer profile
  • document the Experimentalist architecture and the harbor_native / harbor_evaluator A/B workflow
  • add focused baseline, evaluator A/B, cache, and Tau3 smoke-profile coverage

Why this is separate

This keeps the documentation, examples, and smoke tests independently reviewable and reversible from the evaluator implementation, as requested in this review comment.

The new hello example is also a fast smoke-test path:

  • the baseline test needs no Docker, model API key, or network access
  • the live A/B test needs Docker but no model API key
  • the live test exercises both evaluator paths plus cache reuse and force_rerun

Stack

This PR targets #955 because the docs, configuration, and A/B tests intentionally reference the harbor_evaluator implementation introduced there. Once #955 lands, this becomes an examples-only follow-up to main.

Verification

  • uv run --frozen pytest plugins/nemo-experimentalist/tests/experimentalist/test_evaluator_harbor_ab_e2e.py plugins/nemo-experimentalist/tests/experimentalist/test_hello_example_baseline.py plugins/nemo-experimentalist/tests/test_e2e_helper_defaults.py -q — 17 passed
  • uv run --frozen ruff check plugins/nemo-experimentalist/examples plugins/nemo-experimentalist/tests/experimentalist/test_evaluator_harbor_ab_e2e.py plugins/nemo-experimentalist/tests/experimentalist/test_hello_example_baseline.py plugins/nemo-experimentalist/tests/test_e2e_helper_defaults.py
  • uv run --frozen ruff format --check plugins/nemo-experimentalist/examples plugins/nemo-experimentalist/tests/experimentalist/test_evaluator_harbor_ab_e2e.py plugins/nemo-experimentalist/tests/experimentalist/test_hello_example_baseline.py plugins/nemo-experimentalist/tests/test_e2e_helper_defaults.py
  • git diff --check

Refs AALGO-312

Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
Signed-off-by: Nick Goncharenko <ngoncharenko@nvidia.com>
@ngoncharenko
ngoncharenko marked this pull request as ready for review August 3, 2026 21:27
@ngoncharenko
ngoncharenko requested review from a team as code owners August 3, 2026 21:27
@ngoncharenko
ngoncharenko force-pushed the ngoncharenko/aalgo-312-examples branch from 64a10b5 to 709a3ce Compare August 3, 2026 21:27
@ngoncharenko ngoncharenko changed the title feat(experimentalist): add fast Harbor smoke example and docs feat(experimentalist): add fast Harbor smoke example Aug 3, 2026
@ngoncharenko ngoncharenko changed the title feat(experimentalist): add fast Harbor smoke example feat(experimentalist): add Harbor examples Aug 3, 2026
@ngoncharenko
ngoncharenko force-pushed the ngoncharenko/aalgo-312-wire-eval-into-optimizer branch from a4c72b8 to 093d1fb Compare August 4, 2026 18:45
@ngoncharenko
ngoncharenko force-pushed the ngoncharenko/aalgo-312-wire-eval-into-optimizer branch 2 times, most recently from 756b7f2 to 1406607 Compare August 14, 2026 23:59
Base automatically changed from ngoncharenko/aalgo-312-wire-eval-into-optimizer to main August 16, 2026 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant