RFC: Enhanced Execution Logging & Statistics

Status: Implemented (with architectural changes) Date: 2025-10-04 Version: 1.0

Note (January 2026): This RFC has been implemented, but the underlying architecture has evolved significantly since the original design. The CheckExecutionEngine class described below has been replaced by StateMachineExecutionEngine (see src/state-machine-execution-engine.ts). Statistics tracking is now handled via:

src/types/execution.ts - Type definitions for CheckExecutionStats and ExecutionStatistics

src/state-machine/dispatch/stats-manager.ts - Runtime statistics collection

src/state-machine/execution/summary.ts - Result aggregation

The data structures remain largely compatible with the RFC specification, with some additions like skippedRuns, providerDurationMs, and additional skip reasons (forEach_empty, assume). The formatting methods (formatStatusColumn, formatDetailsColumn) remain in the engine class for test compatibility.

Summary

This RFC describes the implementation of comprehensive execution statistics tracking and enhanced logging for Visor checks, providing users with clear visibility into forEach iterations, skip reasons, and overall execution metrics.

Motivation

Users running complex Visor workflows with features like forEach, if conditions, and fail_if lacked visibility into:

How many times each check actually ran (forEach iterations)
Why checks were skipped and what conditions failed
Overall execution metrics (total runs vs. configured checks)
Performance bottlenecks (time per check/iteration)
Issue distribution across checks

This made debugging and understanding workflow execution difficult.

Design

Architecture

The solution adds execution statistics tracking throughout the check execution lifecycle:

CheckExecutionEngine
├── executionStats: Map<string, CheckExecutionStats>  // Tracks all stats
├── initializeCheckStats()                             // Initialize stats for each check
├── recordIterationStart/Complete()                    // Track forEach iterations
├── recordSkip()                                       // Track skipped checks
├── recordError()                                      // Track failures
├── buildExecutionStatistics()                         // Build final stats object
└── logExecutionSummary()                              // Display symbol-rich table

Data Structures

CheckExecutionStats

interface CheckExecutionStats {
  checkName: string;
  totalRuns: number;          // How many times executed (1 or forEach iterations)
  successfulRuns: number;
  failedRuns: number;
  skipped: boolean;
  skipReason?: 'if_condition' | 'fail_fast' | 'dependency_failed';
  skipCondition?: string;     // The actual if condition text
  totalDuration: number;      // Total duration in milliseconds
  perIterationDuration?: number[];  // Duration for each iteration
  issuesFound: number;
  issuesBySeverity: {
    critical: number;
    error: number;
    warning: number;
    info: number;
  };
  outputsProduced?: number;   // For forEach checks
  errorMessage?: string;      // If failed
  forEachPreview?: string[];  // Preview of forEach items
}

ExecutionStatistics

interface ExecutionStatistics {
  totalChecksConfigured: number;
  totalExecutions: number;     // Sum of all runs including forEach
  successfulExecutions: number;
  failedExecutions: number;
  skippedChecks: number;
  totalDuration: number;
  checks: CheckExecutionStats[];
}

Tracking Points

Initialization (executeDependencyAwareChecks start)
- Initialize stats for all checks
forEach Execution (when forEach items detected)
- Record forEach preview items
- Track each iteration start/complete
- Log iteration progress: ✔ 3/5 (2.1s)
Normal Execution (single runs)
- Track start/complete
- Record outputs produced
Skip Conditions (when if evaluates to false)
- Record skip reason and condition
- Log: ⏭ Skipped (if: branch == "main")
Errors (on exception)
- Record error message
- Track failed iteration
Completion (executeChecks end)
- Build final statistics
- Display summary table

Output Formats

Console (Table) - Default

┌───────────────────────────────────────┐
│ Execution Complete (45.3s)            │
├───────────────────────────────────────┤
│ Checks: 8 configured → 23 executions  │
│ Status: 20 ✔ │ 2 ✖ │ 1 ⏭             │
│ Issues: 15 total (3 🔴 12 ⚠️)         │
└───────────────────────────────────────┘

Check Details:
┌─────────────────────┬──────────┬──────────┬─────────────────────┐
│ Check               │ Duration │ Status   │ Details             │
├─────────────────────┼──────────┼──────────┼─────────────────────┤
│ list-files          │ 0.5s     │ ✔        │ →5                  │
│ validate-file       │ 12.3s    │ ✔ ×5     │ 12⚠️                │
│ security-scan       │ 8.2s     │ ✔        │ 3🔴                 │
│ transform-data      │ 6.1s     │ ✔ ×3     │ →3                  │
│ notify-slack        │ 2.1s     │ ✖ ×3     │ HTTP 500            │
│ production-only     │ -        │ ⏭ if     │ branch=="main"      │
│ archive-results     │ 1.3s     │ ✔        │                     │
│ final-report        │ 14.8s    │ ✔        │ →1                  │
└─────────────────────┴──────────┴──────────┴─────────────────────┘

Legend: ✔=success │ ✖=failed │ ⏭=skipped │ ×N=iterations │ →N=outputs │ N🔴=critical │ N⚠️=warnings

JSON Output

Full executionStatistics object included in JSON output with all tracked data:

Per-iteration timings
Skip condition evaluations
forEach item previews
Complete error messages

Symbol Reference

Symbol	Meaning	Example
`✔`	Success (1 run)	`✔`
`✔ ×5`	Success (5 forEach iterations)	`✔ ×5`
`✖ ×3`	Failed (3 iterations)	`✖ ×3`
`✔/✖ 2/3`	Partial success (2 of 3 succeeded)	`✔/✖ 2/3`
`⏭ if`	Skipped (if condition)	`⏭ if`
`⏭ ff`	Skipped (fail-fast)	`⏭ ff`
`⏭ dep`	Skipped (dependency failed)	`⏭ dep`
`→N`	Produced N outputs	`→5`
`N🔴`	N critical issues	`3🔴`
`N⚠️`	N warnings	`12⚠️`
`N💡`	N info messages	`5💡`

Implementation

Files Modified

src/check-execution-engine.ts (~300 lines added)
- Added CheckExecutionStats and ExecutionStatistics interfaces
- Added executionStats property to class
- Implemented tracking methods:
  - initializeCheckStats()
  - recordIterationStart/Complete()
  - recordSkip()
  - recordError()
  - recordForEachPreview()
  - buildExecutionStatistics()
- Implemented display methods:
  - logExecutionSummary()
  - formatStatusColumn()
  - formatDetailsColumn()
  - truncate()
- Updated executeDependencyAwareChecks() to track stats
- Updated executeChecks() to return statistics
src/output-formatters.ts (~5 lines added)
- Added executionStatistics? field to AnalysisResult interface

Backward Compatibility

✅ No Breaking Changes

All changes are additive
executionStatistics is optional in AnalysisResult
Existing output formats continue to work
Statistics appear automatically in table output (default)
JSON/SARIF include statistics when available

Examples

Scenario 1: forEach with Multiple Iterations

Config:

steps:
  list-files:
    type: command
    exec: echo '["a.json", "b.json", "c.json"]'
    forEach: true

  validate-file:
    type: command
    exec: jsonlint {{ outputs.list-files }}
    depends_on: [list-files]

Output:

▶ Running check: list-files [1/2]
✔ Check complete: list-files (0.3s) - 3 items

▶ Running check: validate-file [2/2]
  Processing 3 items...
  ✔ 1/3 (1.2s)
  ✔ 2/3 (1.1s)
  ✔ 3/3 (1.3s)
✔ Check complete: validate-file (3.6s) - 3 runs

┌───────────────────────────────────────┐
│ Execution Complete (3.9s)             │
├───────────────────────────────────────┤
│ Checks: 2 configured → 4 executions   │
│ Status: 4 ✔ │ 0 ✖ │ 0 ⏭              │
└───────────────────────────────────────┘

Check Details:
┌─────────────────────┬──────────┬──────────┬─────────────────────┐
│ Check               │ Duration │ Status   │ Details             │
├─────────────────────┼──────────┼──────────┼─────────────────────┤
│ list-files          │ 0.3s     │ ✔        │ →3                  │
│ validate-file       │ 3.6s     │ ✔ ×3     │                     │
└─────────────────────┴──────────┴──────────┴─────────────────────┘

Scenario 2: Conditional Skip

Config:

steps:
  production-deploy:
    type: command
    exec: ./deploy.sh
    if: 'branch == "main"'

Output (on feature branch):

▶ Running check: production-deploy [1/1]
⏭  Skipped (if: branch == "main")

┌───────────────────────────────────────┐
│ Execution Complete (0.0s)             │
├───────────────────────────────────────┤
│ Checks: 1 configured → 0 executions   │
│ Status: 0 ✔ │ 0 ✖ │ 1 ⏭              │
└───────────────────────────────────────┘

Check Details:
┌─────────────────────┬──────────┬──────────┬─────────────────────┐
│ Check               │ Duration │ Status   │ Details             │
├─────────────────────┼──────────┼──────────┼─────────────────────┤
│ production-deploy   │ -        │ ⏭ if     │ branch == "main"    │
└─────────────────────┴──────────┴──────────┴─────────────────────┘

Benefits

🎯 Clarity: Users immediately see forEach iterations and skip reasons
📊 Metrics: Clear stats on total work done vs. configured checks
⚡ Performance: Easy to spot slow checks and iterations
🐛 Debugging: Skip conditions shown inline for quick troubleshooting
📈 Automation: Full stats in JSON for CI/CD analytics
✅ Professional: Clean, symbol-rich output matches modern CLIs

Future Enhancements

Potential future improvements:

Color-coded output (green/red/yellow) when terminal supports it
Configurable verbosity levels for forEach iteration logging
Export statistics to external monitoring systems
Historical statistics comparison across runs
Performance regression detection

References

Implementation: See src/state-machine-execution-engine.ts, src/types/execution.ts, and src/state-machine/dispatch/stats-manager.ts
Tests: tests/unit/execution-statistics.test.ts, tests/unit/execution-statistics-formatting.test.ts, tests/integration/execution-statistics-integration.test.ts

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: Enhanced Execution Logging & Statistics

Summary

Motivation

Design

Architecture

Data Structures

CheckExecutionStats

ExecutionStatistics

Tracking Points

Output Formats

Console (Table) - Default

JSON Output

Symbol Reference

Implementation

Files Modified

Backward Compatibility

Examples

Scenario 1: forEach with Multiple Iterations

Scenario 2: Conditional Skip

Benefits

Future Enhancements

References

FilesExpand file tree

execution-statistics-rfc.md

Latest commit

History

execution-statistics-rfc.md

File metadata and controls

RFC: Enhanced Execution Logging & Statistics

Summary

Motivation

Design

Architecture

Data Structures

CheckExecutionStats

ExecutionStatistics

Tracking Points

Output Formats

Console (Table) - Default

JSON Output

Symbol Reference

Implementation

Files Modified

Backward Compatibility

Examples

Scenario 1: forEach with Multiple Iterations

Scenario 2: Conditional Skip

Benefits

Future Enhancements

References