Repository navigation
Conversation
Ports the scanner-mapping work from the GitLab-only branch reformat_input_and_output_csv_and_fetchscanners, which was never carried across to GitHub during the v2.0 refactor and would have been lost when the GitLab project is deleted. Scan responses identify scanners by opaque ID only. This resolves those IDs to human-readable names via the paged /scanners endpoint and records which scanners flagged each prompt in a new "scanners" field. Differences from the original implementation: - The lookup is lazy and cached rather than run at import time, so importing the module no longer performs network I/O. The original would have fired five requests on import and broken CI, which imports this module while mocking only requests.post. - Requests carry an explicit timeout, and a failing source is skipped rather than aborting the whole lookup. - Unknown scanner IDs fall back to the raw ID instead of a placeholder, so the information is preserved rather than discarded. - The field flows through all four output formats and the false positive and false negative exports. - Adds --no-scanner-names to skip the lookup entirely. The CSV input handling from the original commit was already superseded by tools/enhanced_dataset_reader.py and was not ported. Co-Authored-By: Allan Healy <allan.healy@calypsoai.com> Co-Authored-By: Warp <agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While decoupling this project from GitLab I found a commit that only ever existed there. Allan Healy wrote it on 6 May 2025 on a branch called
reformat_input_and_output_csv_and_fetchscanners, and it never made the jump to GitHub during the v2.0 refactor. Deleting the GitLab project would have taken it with it.What it does
Scan responses tell you a prompt was flagged, but not which scanner flagged it. The API returns opaque scanner IDs, and the readable names live behind a separate
/scannersendpoint that's partitioned by source. This resolves those IDs to names and records them in a newscannersfield on every result.Why it's a rewrite rather than a cherry-pick
The original predates the pocketflow refactor so it wouldn't apply cleanly, and it had a few problems worth fixing on the way across.
prompt_evaluatorwould have fired five HTTP requests. That breaks CI, which imports this module intests/test_integration.pywhile mocking onlyrequests.post. The lookup is now lazy and cached, so it runs once per evaluation run and never at import.Scope
The field flows through all four output formats plus the false positive and false negative exports. CSV and TSV get a
scannerscolumn with names joined by semicolons. JSONL and Parquet keep the native list. Records written before this change still serialise, they just get an empty cell.There's also a new
--no-scanner-namesflag for skipping the extra requests when you only care about pass or fail.The other half of Allan's commit handled CSV input with separate prompt and expected columns. That's already covered by
tools/enhanced_dataset_reader.py, so I left it out.Testing
36 new tests in
tests/test_scanner_resolution.py, registered in the core suite so CI picks them up. They cover map caching and force refresh, partial and total source failure, the raw ID fallback, malformed payloads, and serialisation across every output format.The full runner passes 5 of 5 suites at 77 tests. The flake8 critical gate is clean, and bandit reports zero findings on
prompt_evaluator.py, matching main.Original work by Allan Healy, preserved via
Co-Authored-Byon the commit.Warp conversation
Co-Authored-By: Warp agent@warp.dev