Skip to content

check-tests exits with code 0 on sync/auth failures, causing CI jobs to pass #259

@nonkor

Description

@nonkor

Describe the bug
check-tests reports synchronization/authentication errors in stdout, but still exits with status code 0. Because of that, CI jobs are marked as successful even when sync actually failed.

This was observed with both:

npx check-tests@latest manual "**/*.md"
npx check-tests@latest push --sync --create --no-detached --force

Examples of errors printed by the CLI while the process still exits successfully:

{"message":"Project API Token is invalid"} (403: Forbidden)
Error in updating test ids {"message":"Project API Token is invalid"}

Pull latest changes from Testomat.io before pushing or use --force option (422: Unprocessable Content)
This makes the tool unreliable in CI, because pipelines trust the process exit code.

To Reproduce
Steps to reproduce the behavior:

  1. Create a project with markdown manual tests and a CI job using check-tests.
  2. Run one of the following commands with an invalid token, read-only token, or missing token:
  • npx check-tests@latest push --sync --create --no-detached --force
  • npx check-tests@latest manual "**/*.md"
  1. Observe that the CLI prints an error such as 403: Forbidden, API key not provided, or 422: Unprocessable Content.
  2. Check the process exit code / CI job result.
  3. See that the command exits with code 0, so the CI job is marked as successful.

Expected behavior
If synchronization, authentication, or remote update fails, the CLI should exit with a non-zero status code.

Example output:

🚀 Sending data to testomat.io
Wait for Testomatio to synchronize tests...
✖️ {"message":"Project API Token is invalid"} (403: Forbidden)
Updating test ids in the source code...
✖️ Error in updating test ids {"message":"Project API Token is invalid"}
Cleaning up project directory and file based variables
Job succeeded

Desktop (please complete the following information):

  • OS: macOS
  • Browser: N/A, CLI tool executed in GitLab CI / terminal
  • Version: check-tests v0.17.0

Additional context
push appears to be implemented as an alias/wrapper around the same manual import flow, so both commands seem to share the same exit-code problem.

Current workaround in CI is to parse stdout/stderr and fail the job if output contains markers such as:

  • Error
  • 4xx/5xx

But that is only a workaround. The proper fix should be in check-tests itself:

  • return non-zero exit code on sync/auth/import/update failures
  • document exit code behavior clearly
  • optionally add a strict mode if backward compatibility is needed

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingpriority highusersusers need help with the issue

Type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions