Skip to content

fix(pyats): disable EnvironmentDebugPlugin to prevent credential exposure#697

Draft
oboehmer wants to merge 5 commits intomainfrom
fix/689-570-disable-env-debug-plugin
Draft

fix(pyats): disable EnvironmentDebugPlugin to prevent credential exposure#697
oboehmer wants to merge 5 commits intomainfrom
fix/689-570-disable-env-debug-plugin

Conversation

@oboehmer
Copy link
Collaborator

@oboehmer oboehmer commented Mar 22, 2026

Description

Security fix: PyATS EnvironmentDebugPlugin was writing environment variables (including passwords) to env.txt files within archive artifacts. This change disables the plugin and refactors config file management for cleaner architecture.

Closes

Related Issue(s)

  • N/A

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Refactoring / Technical debt (internal improvements with no user-facing changes)
  • Documentation update
  • Chore (build process, CI, tooling, dependencies)
  • Other (please describe):

Test Framework Affected

  • PyATS
  • Robot Framework
  • Both
  • N/A (not test-framework specific)

Network as Code (NaC) Architecture Affected

  • ACI (APIC)
  • NDO (Nexus Dashboard Orchestrator)
  • NDFC / VXLAN-EVPN (Nexus Dashboard Fabric Controller)
  • Catalyst SD-WAN (SDWAN Manager / vManage)
  • Catalyst Center (DNA Center)
  • ISE (Identity Services Engine)
  • FMC (Firepower Management Center)
  • Meraki (Cloud-managed)
  • NX-OS (Nexus Direct-to-Device)
  • IOS-XE (Direct-to-Device)
  • IOS-XR (Direct-to-Device)
  • Hyperfabric
  • All architectures
  • N/A (architecture-agnostic)

Platform Tested

  • macOS (version tested: macOS with Python 3.12)
  • Linux (distro/version tested: )

Key Changes

Security (#689)

  • Add EnvironmentDebugPlugin: enabled: False to plugin configuration
  • Add E2E regression test scanning ALL output artifacts for credential sentinels
  • Test covers both zip contents and all non-zip files

Architecture Refactoring (#570)

  • Move config file creation from execute_job() into SubprocessRunner.__init__()
  • Add cleanup() method to SubprocessRunner for config file removal
  • Orchestrator calls cleanup() conditionally based on keep_artifacts policy
  • Config file attributes only set after successful file creation (safer error handling)
  • Extract config filenames as constants in constants.py
  • Write config files to output_dir with well-known names instead of temp files

Dead Code Removal

  • Remove unused _build_reporter_config() and _generate_plugin_config() from orchestrator
  • Remove unnecessary tempfile.TemporaryDirectory wrapper in _run_tests_async
  • Remove unused plugin_config_path parameter

Testing Done

  • Unit tests added/updated (25 subprocess_runner unit tests)
  • Integration tests performed (16 subprocess_runner integration tests)
  • Manual testing performed:
    • PyATS tests executed successfully
    • Robot Framework tests executed successfully
    • D2D/SSH tests executed successfully (if applicable)
    • HTML reports generated correctly
  • All existing tests pass (pytest / pre-commit run -a)

Test Commands Used

# Unit tests (25 subprocess_runner tests, 217 pyats_core tests total)
pytest tests/unit/pyats_core/execution/test_subprocess_runner.py -v
pytest tests/unit/pyats_core/ tests/pyats_core/ -v

# E2E credential exposure tests
pytest tests/e2e/test_e2e_scenarios.py -v -k "credential" -n auto --dist loadscope

# Pre-commit hooks
pre-commit run --all-files

Checklist

  • Code follows project style guidelines (pre-commit run -a passes)
  • Self-review of code completed
  • Code is commented where necessary (especially complex logic)
  • Documentation updated (if applicable)
  • No new warnings introduced
  • Changes work on both macOS and Linux
  • CHANGELOG.md updated (if applicable)

Screenshots (if applicable)

N/A

Additional Notes

Design decisions:

  • SubprocessRunner now owns both creation AND cleanup of its config files (cleaner SRP)
  • Orchestrator retains policy control (decides when to call cleanup based on debug flags)
  • Config file attributes (_plugin_config_file, _pyats_config_file) are Path | None and only set after successful creation — this ensures cleanup() is safe even if initialization partially failed

@oboehmer oboehmer force-pushed the fix/689-570-disable-env-debug-plugin branch from 6a826c2 to 92ca3a8 Compare March 22, 2026 17:36
…andling (#689, #570)

Security fix: PyATS EnvironmentDebugPlugin was writing environment variables
(including passwords) to env.txt files within archive artifacts. This change
disables the plugin and refactors config file management for cleaner architecture.

**Security (#689)**
- Add `EnvironmentDebugPlugin: enabled: False` to plugin configuration
- Add E2E regression test scanning ALL output artifacts for credential sentinels
- Test covers both zip contents and all non-zip files

**Architecture Refactoring (#570)**
- Move config file creation from `execute_job()` into `SubprocessRunner.__init__()`
- Add `cleanup()` method to SubprocessRunner for config file removal
- Orchestrator calls `cleanup()` conditionally based on keep_artifacts policy
- Config file attributes only set after successful file creation (safer error handling)
- Extract config filenames as constants in `constants.py`
- Write config files to output_dir with well-known names instead of temp files

**Dead Code Removal**
- Remove unused `_build_reporter_config()` and `_generate_plugin_config()` from orchestrator
- Remove unnecessary `tempfile.TemporaryDirectory` wrapper in `_run_tests_async`
- Remove unused `plugin_config_path` parameter

Closes #689, #570

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@oboehmer oboehmer force-pushed the fix/689-570-disable-env-debug-plugin branch from 92ca3a8 to 9214b78 Compare March 22, 2026 17:42
… credential check

- Remove 3 moot unit tests that checked config strings (fragile, not behavioral)
- Remove duplicate integration test for config creation failure
- Move credential sentinel test imports to module level
- Run credential check for all scenarios, not just PyATS
- Improve error message for PyATS config creation failure
…chestrator

Verify that when SubprocessRunner fails to create config files (e.g., disk full),
PyATSOrchestrator returns PyATSResults with from_error() for discovered test types.
…ix cleanup leak

- Delete ConfigFileCreationError; _create_config_files now raises
  RuntimeError, consistent with the pyats-not-found failure in __init__
- Catch RuntimeError in orchestrator so both init failures populate
  api/d2d results correctly instead of falling through to the broad
  except-Exception handler
- Call subprocess_runner.cleanup() in the report-generation failure
  branch so config files are removed regardless of outcome
- Assert EnvironmentDebugPlugin disabled in plugin config integration
  test using regex; convert all config-content checks to re.search
- Remove leftover try/finally unlink() blocks in TestConfigFileContent
  (files live in tmp_path, cleaned up by pytest)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant