Skip to content

Conversation

@rlundeen2
Copy link
Contributor

@rlundeen2 rlundeen2 commented Dec 29, 2025

This PR introduces DatasetConfiguration class that is passed to scenarios in initialize_async to address several pain points.

Big available default datasets

It was tough to change the default datasets - some were too big and some too small. As an example, Foundry used harm_bench, which is too big for the default. So it would randomly select 4 by default. But how could users run against all 100? There was no way to configure this. On the flip side garak.encoding_scenario had a large dataset that took a very long time to run, but there was no way to make it small by default and still have the entire dataset available.

This change allows scenarios to be configured with a default (e.g. name=harm_bench, max=4). But users can easily configure both the dataset name or max differently.

It also helps with our end to end tests, which were taking a million years because encoding scenario had so many datasets (this pr reduces to only 3 by default)

Allows dataset params from the front end

This also allows users to specify the dataset names they want to use from pyrit_scan and pyrit_shell. Previously, users had to use the dataset defaults.

--dataset-names DATASET_NAMES [DATASET_NAMES ...]
Dataset names to load for the scenario (overrides scenario defaults)
--max-dataset-size MAX_DATASET_SIZE
Maximum number of seed groups to use (randomly samples if dataset is larger)

Deprecates Incompatible Parameters

objective, seed_prompts are now deprecated as initialization parameters for scenarios

Tests

  • Unit tests added!
  • End to end tests passed 2c281be

Copy link
Contributor

@nina-msft nina-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

feature looks good overall, i'd love to see the documentation flushed out a bit with my comments and an ack on my if/else logic if you think it doesn't need to be changed :-)

@rlundeen2 rlundeen2 merged commit 698d848 into Azure:main Jan 1, 2026
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants