tina4 test always exits 0 even when pytest fails (_test discards pytest's return code)

[callanrichardsmith]

Summary

tina4 test (the CLI _test command) runs pytest but discards its exit code, so the command
always exits 0 — even when tests fail. Any CI or pre-push gate that relies on tina4 test's exit
status (tina4 test || exit 1) silently passes on a red suite, letting broken code through to
production.

Affected code

tina4_python/cli/__init__.py::_test, on both branches:

v3 (default / release branch):

def _test(args):
    """Run the test suite."""
    subprocess.run([sys.executable, "-m", "pytest", "tests/"] + args)

main:

def _test(args):
    """Run the test suite."""
    import subprocess
    cmd = [sys.executable, "-m", "pytest", "tests/"] + args
    subprocess.run(cmd)

Neither inspects result.returncode nor calls sys.exit(...), so the function returns None and the
CLI exits 0 regardless of the pytest result. Contrast _build a few lines below, which does it right:

result = subprocess.run([docker, "build", ...])
if result.returncode != 0:
    sys.exit(result.returncode)

Reproduction

tina4-python 3.13.75 (also reproduced by reading 3.13.78 / current v3 + main), Windows, Python 3.12.2, pytest 9.1.1:

1. Add a failing test under tests/: def test_x(): assert False
2. Run tina4 test — it prints 1 failed, N passed but the exit code is 0.
3. Run python -m pytest tests/ on the same tree — exit code is 1.

$ tina4 test -k some_failing_test ; echo "exit=$?"
...
====================== 1 failed, 336 deselected in 0.33s ======================
exit=0        # <-- should be non-zero

$ python -m pytest tests/ -k some_failing_test ; echo "exit=$?"
...
====================== 1 failed, 336 deselected in 0.33s ======================
exit=1        # correct

Fix (one line)

Propagate pytest's exit code:

def _test(args):
    """Run the test suite."""
    result = subprocess.run([sys.executable, "-m", "pytest", "tests/"] + args)
    sys.exit(result.returncode)

Happy to open a PR against v3 (and main) if that helps.

Impact

Real-world: a project's pre-push hook ran tina4 test || exit 1 as its backend gate and pushed a
red suite (1 failed, 334 passed) to production because tina4 test returned 0. Anyone gating
CI or a git hook on tina4 test's exit code is currently unprotected against failing backend tests.

[andrevanzuydam]

Fixed and released in 3.13.81 across all four frameworks.

Python (the reported bug): _test ran pytest via subprocess.run(...) and discarded the return code, so tina4 test always exited 0 - a tina4 test || exit 1 gate passed on a red suite. It now does sys.exit(result.returncode), so the command exits with pytest own code.

Cross-framework parity check (the fix is a family, not one file):

• PHP had the identical bug - tina4php test ran phpunit through passthru() without capturing the exit code. Fixed the same way (capture + exit). While there we also dropped the --verbose flag PHPUnit 10+ removed: on PHPUnit 11 it exits 2, which - once the code was propagated - would have turned a passing suite into a false failure. Now uses --colors=always.
• Ruby and Node were already correct (both already exit non-zero on failure). They each gained a real lock-in test so it cannot regress.

Every fix is pinned by a REAL test that spawns the actual CLI against a temp project with a failing suite and asserts a non-zero exit (and 0 for a passing one) - no mocks. Each negative test was proven to fail against the pre-fix code.

This release also re-aligns the four frameworks on one version (Python/Ruby/Node skipped 3.13.80, a PHP-only patch). Thanks for the precise report - the _test code excerpt made this quick to confirm.

[andrevanzuydam]

Fixed and released. tina4 test no longer swallows the pytest exit code - a failing run now exits non-zero. Commit 82b8cd2, released 3.13.81, current 3.13.85.