IDS²Eval

hashtagRR.github.io/IDS2Eval

A config-driven toolkit that audits network intrusion detection (NIDS) datasets before you trust a result on them. It runs 28 checks for duplicated rows, label conflicts, one-column shortcuts and split problems, tests whether a suspected problem actually changes the accuracy, and records every verdict in a citable scorecard.

IDS² reads both ways: Intrusion Detection Systems and Intrusion Data Sets.

Quickstart

python3 -m venv venv && venv/bin/pip install -e ".[plots]"
venv/bin/ids2eval-dashboard                      # web UI with a pre-filled config editor
venv/bin/ids2eval run --config my_config.yaml    # or the CLI, with a config like the one below

# my_config.yaml: any NIDS dataset in CSV or Parquet form
dataset:
  name: my-dataset
  raw_files: ["data.csv"]
schema:
  label_column: Label

Windows and troubleshooting: INSTALL.md.

Documentation

Guide	What it covers
Usage	the CLI, the dashboard, output files, caching, reproducing a run, and the Python API
Config fields	every config field, with its default and an example
Configuration notes	the reasoning behind non-obvious fields, and larger worked examples
Checks	what each of the 28 checks does, the research behind it, and the scorecard's verdict rule
Example scorecards	the real audit runs behind the paper, on seven widely used NIDS benchmarks
Paper results	configurations, scorecards and analysis outputs behind the accompanying paper
Known issues	how to add a curated known issue or extractor fingerprint
Install guide	platform setup, extras, tests and lint, including Windows
Project site	the project's overview page

Versions

• v0.2.1 (10.5281/zenodo.23301210): a check with no input to test reports not_applicable instead of ok, an audit in which no check produced a result is not_evaluated instead of passed, scorecards list the checks a config left disabled (schema 1.4), and known-issue warnings name the release they document; no statistic or threshold changed. Also adds the validation scripts and results of the paper's revision (host-disjoint splits, workflow ablation, interval coverage, class-adjusted reuse, a comparison with Flood et al.'s released heuristics).
• v0.2.0 (10.5281/zenodo.23217301): the code, configurations and results behind the accompanying paper on auditing NIDS benchmarks (in preparation).

IDS²Eval grew out of the dataset-auditing code of research on machine-learning-based network intrusion detection, generalized into a dataset-independent toolkit.

Citing

Cite the version you used. The concept DOI 10.5281/zenodo.23217300 always resolves to the latest version.

Relapanawa, R. (2026). IDS2Eval 0.2.1 (Version v0.2.1) [Computer software]. Zenodo. https://doi.org/10.5281/zenodo.23301210

The results, configurations and analysis scripts behind the accompanying paper are archived separately:

Relapanawa, R. (2026). IDS2Eval paper: research artifacts (results, configurations and analysis scripts) [Dataset]. Zenodo. https://doi.org/10.5281/zenodo.23301258

License

AGPL-3.0. See LICENSE.