Skip to content

FEAT: Complete Garak package hallucination ecosystem coverage #2531

Description

Is your feature request related to a problem? Please describe.

PyRIT's existing PackageHallucination scenario does not expose Dart, Perl, or Raku. The existing scorer and registered garak package datasets already support these ecosystems, so users cannot select capabilities that PyRIT already contains. This is part of #511.

Describe the solution you'd like

Extend the existing PackageHallucination scenario. Do not add a second scenario.

Required changes:

  • Add Dart, Perl, and Raku to the scenario technique enum and language specification map.
  • Reuse the registered garak_dart_packages, garak_perl_packages, and garak_raku_packages datasets.
  • Reuse PackageHallucinationScorer, which already supports these three ecosystems. Do not add a loader or scorer.
  • Keep Rust as the bounded default unless maintainers choose another existing default.
  • Load a package registry only when its technique is selected. Do not add large registries to implicit default loading.
  • Increment the scenario version as required by the scenario conventions.
  • Add focused tests for selection, dataset/scorer wiring, and the unchanged default. Update the synchronized Garak scanner documentation.

Follow doc/code/framework.md, the scenario instructions, the dataset instructions, and the applicable test and documentation instructions. Keep dataset selection in the scenario configuration and scoring in the scorer.

Describe alternatives you've considered, if relevant

  • A new scenario, new registry loaders, or a new scorer would duplicate existing PyRIT components.
  • Loading all registries by default would make a small parity change expensive and would change current behavior.

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions