feat(pyats): add tag-based filtering for pyATS tests

[oboehmer]

Description

This PR adds tag-based filtering support for PyATS tests using the --include and --exclude CLI options, bringing PyATS test selection in line with Robot Framework's tag filtering capabilities. PyATS tests are filtered based on their groups class attribute using Robot Framework's TagPatterns API for consistent pattern matching across both frameworks.

Closes

• Fixes [feat] support filtering pyats tests using tags #436

Related Issue(s)

• N/A

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Refactoring / Technical debt (internal improvements with no user-facing changes)
• Documentation update
• Chore (build process, CI, tooling, dependencies)
• Other (please describe):

Test Framework Affected

• PyATS
• Robot Framework
• Both
• N/A (not test-framework specific)

Network as Code (NaC) Architecture Affected

• ACI (APIC)
• NDO (Nexus Dashboard Orchestrator)
• NDFC / VXLAN-EVPN (Nexus Dashboard Fabric Controller)
• Catalyst SD-WAN (SDWAN Manager / vManage)
• Catalyst Center (DNA Center)
• ISE (Identity Services Engine)
• FMC (Firepower Management Center)
• Meraki (Cloud-managed)
• NX-OS (Nexus Direct-to-Device)
• IOS-XE (Direct-to-Device)
• IOS-XR (Direct-to-Device)
• Hyperfabric
• All architectures
• N/A (architecture-agnostic)

Platform Tested

nac-test supports macOS and Linux only

• macOS (version tested: )
• Linux (distro/version tested: )

Key Changes

Core Tag Filtering Implementation

• New TagMatcher class (tag_matcher.py): Wraps Robot Framework's TagPatterns API to provide consistent tag matching behavior for both Robot and PyATS tests

  • Supports all Robot Framework tag pattern syntax: simple tags, wildcards (bgp*), boolean operators (healthANDbgp, bgpORospf, bgpNOTnrfu)
  • Case-insensitive matching with underscore normalization
  • Format helper methods for consistent error messaging

• Refactored test discovery architecture (test_discovery.py, test_type_resolver.py):

  • Renamed TestTypeResolver → TestMetadataResolver to reflect expanded responsibilities
  • AST-based extraction of groups attribute from PyATS test classes without importing
  • Pre-filtering during discovery phase eliminates subprocess overhead
  • Combined discovery and categorization into single pass returning PyatsDiscoveryResult
  • Removed caching (not needed with single-pass architecture)

• New dataclasses (types.py):

  • TestFileMetadata: Structured representation of test file metadata (path, type, groups)
  • PyatsDiscoveryResult: Discovery results with pre-computed test_type_by_path mapping for O(1) lookups

• Orchestrator integration (orchestrator.py):

  • Uses PyatsDiscoveryResult for efficient type lookups during result splitting (replaces re-instantiation of resolver)
  • Improved error messaging: "No pyATS tests matching tag filter (exclude: 'bgp OR ospf')" when all tests filtered out
  • Distinguishes between "no test files" vs "no tests matching filter"

• Documentation (README.md, PRD_AND_ARCHITECTURE.md):

  • Updated --include/--exclude CLI option descriptions
  • Added PyATS groups attribute usage examples
  • Tag pattern syntax reference

Minor Cleanups

• Remove unused ALL_SCENARIOS and get_scenario_by_name from e2e config
• Remove unnecessary __future__ import from tag_matcher.py

Testing

New Tests for Tag Filtering

• E2E tests (tests/e2e/):

  • 4 new scenarios covering include/exclude combinations
  • Fixtures with both Robot and PyATS tests tagged for filtering validation
  • Tests verify correct exit codes and test counts for various filter patterns

• Unit tests (tests/unit/pyats_core/discovery/):

  • test_tag_matcher.py: Tag pattern matching, boolean logic, wildcards, edge cases
  • test_metadata_resolver.py: AST-based test type detection, directory fallback, error handling
  • test_groups_extraction.py: Groups attribute extraction for tag filtering
  • Updated orchestrator tests to use new PyatsDiscoveryResult structure

Test Suite Refactoring (Independent of Feature)

This PR includes test refactoring that improves the PyATS discovery test suite independently of the tag filtering feature:

Consolidation: Integration → Unit Tests

• Moved 42 non-integration tests from test_discovery_integration.py to new tests/unit/pyats_core/discovery/test_test_discovery.py
• Retained only TestDiscoveryPerformance as the true integration test (validates timing with 100+ files)
• Result: Integration tests reduced from 43 → 1

Parametrization

• Consolidated 39 individual test methods in test_tag_matcher.py into 15 parametrized tests (86 total cases)
• Applied pytest parametrization per project testing guidelines

Code Quality

• Added proper type annotations for all test parameters
• Fixed mypy errors for generic types (list → list[str])
• Added MockerFixture type hints for pytest-mock fixtures

Note: This refactoring improves test maintainability and follows project testing guidelines but is not strictly required for the tag filtering feature.

Testing Done

• Unit tests added/updated
  • Unit tests in tests/unit/pyats_core/discovery/:
    • test_tag_matcher.py: Tag pattern matching with Robot Framework semantics
    • test_metadata_resolver.py: AST-based type detection and fallback logic
    • test_groups_extraction.py: Groups attribute extraction for filtering
  • Updated orchestrator tests to use new PyatsDiscoveryResult structure
• E2E tests added
  • 4 tag filtering scenarios validating include/exclude combinations
• Manual testing performed:
  • PyATS tests executed successfully
  • Robot Framework tests executed successfully
  • D2D/SSH tests executed successfully (if applicable)
  • HTML reports generated correctly
• All existing tests pass (pytest / pre-commit run -a)

Test Commands Used

# Run all unit and integration tests
uv run pytest -n auto --dist loadscope

# Test tag filtering - include specific tag
nac-test -d ./tests/e2e/fixtures/tag_filtering -t ./tests/e2e/fixtures/tag_filtering/templates -o ./output --include bgp

# Test tag filtering - exclude specific tag
nac-test -d ./tests/e2e/fixtures/tag_filtering -t ./tests/e2e/fixtures/tag_filtering/templates -o ./output --exclude ospf

# Test tag filtering - boolean pattern
nac-test -d ./data -t ./tests -o ./output --include "healthORbgp" --exclude nrfu

# Test no matches (should exit with code 252)
nac-test -d ./tests/e2e/fixtures/tag_filtering -t ./tests/e2e/fixtures/tag_filtering/templates -o ./output --exclude "bgpORospf"

Checklist

• Code follows project style guidelines (pre-commit run -a passes)
• Self-review of code completed
• Code is commented where necessary (especially complex logic)
• Documentation updated (if applicable)
• No new warnings introduced
• Changes work on both macOS and Linux
• CHANGELOG.md updated (if applicable)

Screenshots (if applicable)

N/A

Additional Notes

Implementation Approach

The tag filtering implementation leverages Robot Framework's TagPatterns API to ensure consistent behavior between Robot and PyATS test selection. Key design decisions:

1. PyATS groups attribute: Uses the native pyATS mechanism (simple list of strings on test classes) for tagging

   class VerifyBgpNeighbors(SDWANTestBase):
       groups = ["bgp", "routing"]  # Matches --include bgp or --include routing

2. Discovery-phase filtering: Tests are filtered during the discovery phase via AST parsing, avoiding subprocess overhead and enabling early validation

3. Single-pass architecture: Discovery and categorization combined into one pass that returns a PyatsDiscoveryResult with pre-computed mappings for efficient O(1) lookups

4. Consistent error messaging: Uses Robot Framework's TagPatterns string formatting for user-facing messages ('bgpORospf' → 'bgp OR ospf')

Breaking Changes

None. The changes are backward-compatible:

• PyATS tests without groups attribute remain unaffected and run as before
• Existing --include/--exclude behavior for Robot Framework unchanged
• All existing test repositories continue to work without modification

Migration Notes

• PyATS tests can optionally add a groups class attribute to enable tag-based filtering
• The groups attribute should be a list of strings (native pyATS format)
• Tests without groups are treated as having no tags and are included by default (unless an --include pattern is specified)