Skip to content

feat(ci): nightly native smoke tests for published CLI binaries #22

@eskenazit

Description

@eskenazit

Component

CLI (polychro-cli)

Problem & Motivation

The mvn test (JVM) build cannot reproduce native-image runtime failures.
A crash introduced in PR #21 (missing jsv-messages ResourceBundle, MissingResourceException on startup)
was only caught by manually running the published Windows binary — there was no automated gate.

Running polychro lint <file> -s <schema> on the published binary produced:

Exception in thread "main" java.lang.ExceptionInInitializerError
Caused by: java.util.MissingResourceException: Can't find bundle for base name jsv-messages, locale en_US
    at com.networknt.schema.I18nSupport.<clinit>(I18nSupport.java:15)

PR #21 fixes the root cause (adds jsv-messages.*\.properties to resource-config.json) and adds a JVM-level
config guard (NativeImageResourceConfigTest), but there is no end-to-end validation that the compiled binary
actually works before a release is published.

Proposed Solution

Add a nightly GitHub Actions workflow (nightly-native-smoke.yml) that:

  1. Builds the native binary for each OS/arch matrix (linux-amd64, windows-amd64, macos-aarch64) — same matrix as publish-cli-bin.yml.
  2. Runs the binary against a minimal fixture that forces I18nSupport / jsv-messages to load:
    • input: a small YAML file with a JSON Schema violation
    • flag: -s pointing to a minimal schema (e.g. {"type": "object", "required": ["missing"]})
  3. Fails the job if stderr contains MissingResourceException, ExceptionInInitializerError, or any uncaught exception — regardless of the lint exit code.
  4. Also runs polychro --version to verify the VersionProvider loads version.properties correctly.

Trigger: schedule: cron: '0 2 * * *' + workflow_dispatch for manual runs.

This provides an automated safety net for native-image regressions that JVM tests cannot catch,
without blocking the release workflow.

Alternatives Considered

  • Running the smoke test as part of publish-cli-bin.yml after the release step: too late — the binary is already published if the smoke test is added post-release.
  • Running the smoke test before the release step in publish-cli-bin.yml: modifies the release workflow, which is restricted by project policy.
  • A nightly workflow is the right balance: catches regressions on main before the next release, without touching the release pipeline.

Example

# .github/workflows/nightly-native-smoke.yml
name: Nightly native smoke test
on:
  schedule:
    - cron: '0 2 * * *'
  workflow_dispatch:
jobs:
  smoke:
    strategy:
      matrix:
        include:
          - os: ubuntu-latest
            binary: polychro-cli
          - os: windows-latest
            binary: polychro-cli.exe
          - os: macos-latest
            binary: polychro-cli
    runs-on: ${{ matrix.os }}
    steps:
      - uses: actions/checkout@v4
      - uses: graalvm/setup-graalvm@v1
        with:
          java-version: '21'
          distribution: 'graalvm-community'
      - run: mvn -Pnative clean package -pl polychro-cli -am --no-transfer-progress -DskipTests
      - name: Smoke test — version
        run: |
          output=$(./polychro-cli/${{ matrix.binary }} --version 2>&1 || true)
          echo "$output"
          echo "$output" | grep -v "ExceptionInInitializerError" | grep -v "MissingResourceException"
          echo "$output" | grep "polychro"
      - name: Smoke test — lint with JSON Schema (forces jsv-messages load)
        run: |
          echo '{"type":"object","required":["missing"]}' > /tmp/schema.json
          echo 'foo: bar' > /tmp/sample.yaml
          stderr=$(./polychro-cli/${{ matrix.binary }} lint /tmp/sample.yaml -s /tmp/schema.json 2>&1 >/dev/null || true)
          echo "stderr: $stderr"
          if echo "$stderr" | grep -q "MissingResourceException\|ExceptionInInitializerError"; then
            echo "FAIL: native binary crashed with a ResourceBundle error"
            exit 1
          fi

Agent Context (optional)

agent_name: GitHub Copilot
llm: Naftiko - Claude Sonnet 4.6
tool: VS Code Copilot Chat
confidence: high
source_event: PR review handoff (pr-review-21.md)
discovery_method: runtime_observation
files_suspected:
  - polychro-cli/src/main/resources/META-INF/native-image/io.polychro/polychro-cli/resource-config.json
  - .github/workflows/publish-cli-bin.yml

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions