Skip to content

Port SWEfficiency benchmark to Harbor #725

Description

@neubig

Context

benchmarks/swefficiency does not appear to have a corresponding Harbor registry dataset/adapter yet. It has resource-sensitive Docker behavior and performance-oriented verification, so it should be ported deliberately with parity checks.

Proposed scope

  • Create a Harbor adapter/dataset for SWEfficiency tasks.
  • Preserve CPU/memory/resource assumptions and performance scoring semantics.
  • Validate Oracle/reference behavior where available.
  • Compare current OpenHands harness results against Harbor runs.

Acceptance criteria

  • SWEfficiency tasks can be run through Harbor with appropriate resource controls.
  • Performance/verifier parity is documented.
  • The OpenHands wrapper can delegate SWEfficiency execution to Harbor.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions