Skip to content

refactor: reuse app metadata extraction flow in evals - #100

Open
juliehinge wants to merge 1 commit into
inveniosoftware:mainfrom
juliehinge:orcha-endpoints-pr
Open

refactor: reuse app metadata extraction flow in evals#100
juliehinge wants to merge 1 commit into
inveniosoftware:mainfrom
juliehinge:orcha-endpoints-pr

Conversation

@juliehinge

Copy link
Copy Markdown
Collaborator

❤️ Thank you for your contribution!

Description

Refactored the evals to use Orcha endpoints and download the dataset from GitHub via dataset_sync.py.

  • Removed duplicated extractor modules and model setup.
  • Normalized the schema in evaluators.py to match the schema predicted by the Orcha app. This required updating some tests in test_evaluators.py.

Fixed an issue in dataset.py related to how the dataset is created/fetched from Langfuse.
Pinned the Langfuse version due to a change in the newer Langfuse API affecting the parameters required when creating dataset items.

  • Therefore the current Langfuse version is intentionally limited because newer versions require an additional parameter when creating dataset items. In the future, we can upgrade to a newer version and update the dataset item creation accordingly.

Checklist

Ticks in all boxes and 🟢 on all GitHub actions status checks are required to merge:

Frontend

Reminder

By using GitHub, you have already agreed to the GitHub’s Terms of Service including that:

  1. You license your contribution under the same terms as the current repository’s license.
  2. You agree that you have the right to license your contribution under the current repository’s license.

Comment thread evals/run.py Outdated
Comment thread evals/run.py Outdated
Comment thread evals/evaluators.py Outdated
@juliehinge
juliehinge force-pushed the orcha-endpoints-pr branch 2 times, most recently from 3767676 to 00e5d53 Compare September 1, 2026 12:25
@juliehinge
juliehinge requested a review from slint September 8, 2026 13:10
@juliehinge
juliehinge marked this pull request as ready for review September 8, 2026 13:10

@OliverGeneser OliverGeneser left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants