Describe the bug
check-tests reports synchronization/authentication errors in stdout, but still exits with status code 0. Because of that, CI jobs are marked as successful even when sync actually failed.
This was observed with both:
npx check-tests@latest manual "**/*.md"
npx check-tests@latest push --sync --create --no-detached --force
Examples of errors printed by the CLI while the process still exits successfully:
{"message":"Project API Token is invalid"} (403: Forbidden)
Error in updating test ids {"message":"Project API Token is invalid"}
Pull latest changes from Testomat.io before pushing or use --force option (422: Unprocessable Content)
This makes the tool unreliable in CI, because pipelines trust the process exit code.
To Reproduce
Steps to reproduce the behavior:
- Create a project with markdown manual tests and a CI job using
check-tests.
- Run one of the following commands with an invalid token, read-only token, or missing token:
npx check-tests@latest push --sync --create --no-detached --force
npx check-tests@latest manual "**/*.md"
- Observe that the CLI prints an error such as
403: Forbidden, API key not provided, or 422: Unprocessable Content.
- Check the process exit code / CI job result.
- See that the command exits with code
0, so the CI job is marked as successful.
Expected behavior
If synchronization, authentication, or remote update fails, the CLI should exit with a non-zero status code.
Example output:
🚀 Sending data to testomat.io
Wait for Testomatio to synchronize tests...
✖️ {"message":"Project API Token is invalid"} (403: Forbidden)
Updating test ids in the source code...
✖️ Error in updating test ids {"message":"Project API Token is invalid"}
Cleaning up project directory and file based variables
Job succeeded
Desktop (please complete the following information):
- OS: macOS
- Browser: N/A, CLI tool executed in GitLab CI / terminal
- Version: check-tests v0.17.0
Additional context
push appears to be implemented as an alias/wrapper around the same manual import flow, so both commands seem to share the same exit-code problem.
Current workaround in CI is to parse stdout/stderr and fail the job if output contains markers such as:
But that is only a workaround. The proper fix should be in check-tests itself:
- return non-zero exit code on sync/auth/import/update failures
- document exit code behavior clearly
- optionally add a strict mode if backward compatibility is needed
Describe the bug
check-testsreports synchronization/authentication errors in stdout, but still exits with status code 0. Because of that, CI jobs are marked as successful even when sync actually failed.This was observed with both:
Examples of errors printed by the CLI while the process still exits successfully:
{"message":"Project API Token is invalid"} (403: Forbidden)
Error in updating test ids {"message":"Project API Token is invalid"}
Pull latest changes from Testomat.io before pushing or use --force option (422: Unprocessable Content)
This makes the tool unreliable in CI, because pipelines trust the process exit code.
To Reproduce
Steps to reproduce the behavior:
check-tests.npx check-tests@latest push --sync --create --no-detached --forcenpx check-tests@latest manual "**/*.md"403: Forbidden, API key not provided, or422: Unprocessable Content.0, so the CI job is marked as successful.Expected behavior
If synchronization, authentication, or remote update fails, the CLI should exit with a non-zero status code.
Example output:
Desktop (please complete the following information):
Additional context
pushappears to be implemented as an alias/wrapper around the same manual import flow, so both commands seem to share the same exit-code problem.Current workaround in CI is to parse stdout/stderr and fail the job if output contains markers such as:
But that is only a workaround. The proper fix should be in
check-testsitself: