feat: embed issue/PR metadata in test_cases.yaml to eliminate issues.yaml

[ChristopherJHart]

Problem Statement

Currently, the tool maintains two separate files for tracking test automation:

1. test_cases.yaml - Contains test case definitions
2. issues.yaml - Contains GitHub issue/PR definitions

This dual-file approach has several drawbacks:

• Performance: Must fetch all issues/PRs and iterate through them for title-based matching
• Reliability: String matching on titles is fragile (special characters, whitespace, title changes break links)
• Complexity: Two files must be kept in sync manually
• Inconsistency: Catalog workflow already uses embedded metadata (catalog_pr_number, etc.) but project workflow doesn't

Proposed Solution

Eliminate issues.yaml entirely by embedding issue/PR metadata directly in test_cases.yaml, using the same pattern already established for catalog workflow.

New Test Case Schema

Add these fields to test case definitions:

test_cases:
  - title: '[IOS-XE] Verify Error Disable Detection Reason Presence'
    purpose: |
      ...
    generated_script_path: iosxe/verify_*.robot
    
    # NEW: Project repository issue/PR metadata
    project_issue_number: 456
    project_issue_url: https://wwwin-github.cisco.com/US-PS-SVS/project/issues/456
    project_pr_number: 789
    project_pr_url: https://wwwin-github.cisco.com/US-PS-SVS/project/pull/789
    project_pr_branch: feature/147-ios-xe-verify-error-disable
    
    # EXISTING: Catalog repository PR metadata (when catalog_destined=true)
    catalog_destined: true
    catalog_pr_number: 123
    catalog_pr_url: https://wwwin-github.cisco.com/Testing-as-Code/tac-catalog/pull/123
    catalog_pr_git_url: https://wwwin-github.cisco.com/Testing-as-Code/tac-catalog
    catalog_pr_branch: feat/ios-xe/add-verify-iosxe-error-disable

Benefits

Performance

• Before: O(n*m) - Fetch all issues (n), iterate all test cases (m), compare titles
• After: O(n) - Direct lookup by issue number: GET /repos/{owner}/{repo}/issues/{number}
• Significant speedup for repos with many issues

Reliability

• Issue/PR numbers are stable integer identifiers
• No string matching edge cases
• Title changes don't break links
• Robust to special characters, whitespace variations

Simplicity

• Single source of truth for test case data
• Same pattern for both project and catalog workflows
• Less code to maintain

User Experience

• One less file to track
• Clear metadata visible directly in test case definitions
• Easier to understand what's been created

Implementation Plan

Phase 1: Schema Design

• Define exact field names for project issue/PR metadata
• Create GitHub issue in tac-quicksilver to track schema additions
• Update test case validation schema

Phase 2: Migration Command

Create new CLI command: github-ops-manager migrate issues-to-metadata

Migration Logic:

for issue in issues_yaml:
    # 1. Find corresponding GitHub issue/PR
    github_issue = await fetch_github_issue_by_title(issue.title)
    github_pr = await fetch_github_pr_for_issue(github_issue)
    
    # 2. Find corresponding test case in test_cases.yaml files
    test_case = find_test_case_by_title(issue.title, test_cases_dir)
    
    # 3. Update test case with metadata
    if test_case:
        test_case["project_issue_number"] = github_issue.number
        test_case["project_issue_url"] = github_issue.html_url
        if github_pr:
            test_case["project_pr_number"] = github_pr.number
            test_case["project_pr_url"] = github_pr.html_url
            test_case["project_pr_branch"] = github_pr.head.ref
        save_test_cases_yaml(...)

# 4. Report orphaned test cases
orphaned_test_cases = find_test_cases_without_issues(test_cases_dir)
print(f"Test cases without issues/PRs: {len(orphaned_test_cases)}")
for tc in orphaned_test_cases:
    print(f"  - {tc['title']}")

Migration Command Output:

Migrating issues.yaml to test_cases.yaml metadata...

✓ Migrated issue #456 → [IOS-XE] Verify Error Disable
  - Updated: workspace/jobfiles/test_cases/completed_test_cases.yaml
  - Added: project_issue_number=456, project_pr_number=789

✓ Migrated issue #457 → [NX-OS] Verify VLAN Configuration
  - Updated: workspace/jobfiles/test_cases/in_progress_test_cases.yaml
  - Added: project_issue_number=457 (no PR yet)

Summary:
  - Issues migrated: 23
  - Test cases updated: 23
  - Test cases without issues: 5

Test cases without issues/PRs:
  - [IOS-XE] Verify BGP Neighbor Status
  - [NX-OS] Verify Port Channel State
  - [IOS-XR] Verify MPLS LDP Sessions
  - [ACI] Verify Endpoint Group Configuration
  - [SD-WAN] Verify Control Connection Status

You can now safely delete issues.yaml and use 'process-issues' command directly.

Phase 3: Core Workflow Refactoring

• Update sync_github_issues() to use direct ID lookups
• Update sync_github_pull_requests() to use direct ID lookups
• Add metadata writeback after creating issues/PRs
• Remove issues.yaml loading/processing code
• Update documentation and examples

Phase 4: Cleanup

• Mark issues.yaml support as deprecated
• Update CLI help text
• Update README with new workflow
• Add migration guide

New Attributes Required

In github-ops-manager (this repo)

These fields will be written by the tool:

# Project repository tracking
project_issue_number: int       # GitHub issue number
project_issue_url: str          # Full URL to issue
project_pr_number: int          # GitHub PR number (optional)
project_pr_url: str             # Full URL to PR (optional)
project_pr_branch: str          # Branch name for PR (optional)

In tac-quicksilver (schema definitions)

Need to add these optional fields to test case schema validation.

Action: Open issue in tac-quicksilver to track schema additions.

Backwards Compatibility

Migration Path

1. Users run migrate issues-to-metadata command
2. Tool reads existing issues.yaml
3. Tool updates test_cases.yaml files with metadata
4. Users review migration output
5. Users delete issues.yaml (manual step)
6. Future runs use embedded metadata

Graceful Degradation

• If project_issue_number not present, create new issue
• If GitHub returns 404 for issue number, create new issue
• Same behavior as current system for missing/deleted resources

Testing Strategy

• Unit tests for migration logic
• Integration tests with real GitHub repos
• Test migration with complex issues.yaml files
• Test orphaned test case detection
• Test error handling (404s, deleted issues, etc.)

Related Issues

• Add catalog workflow CLI flags for targeting catalog repository #22, Implement PR metadata writeback to test_cases.yaml after catalog PR creation #23, Handle robot file copy to catalog directory structure in catalog PRs #24 - Catalog workflow implementation (established metadata pattern)
• tac-quicksilver#TBD - Add project metadata fields to test case schema

Success Criteria

• Can run migration command successfully
• Can create new issues/PRs without issues.yaml
• Can update existing issues/PRs using embedded metadata
• Performance improvement measurable (benchmark before/after)
• Zero data loss during migration
• Clear migration path documented

🤖 Generated with Claude Code