Repository navigation
[Bug]: CI skips 85 tests because unittest discovery cannot collect module-level pytest functions #156
Description
Primary affected area
Other or unsure — continuous integration and test collection.
Affected capability
Not capability-specific.
What happened?
CI discovers tests with python -m unittest discover -s tests, which collects only unittest.TestCase subclasses. Eight test files define module-level pytest functions instead, so unittest discovery collects nothing from them and reports success. 85 tests never run in CI, with no warning that they were skipped.
The counts reconcile exactly:
| Runner | Tests |
|---|---|
python -m unittest discover -s tests -q (what CI runs) |
655 |
python -m pytest |
740 |
| Difference | 85 |
Per file:
| File | pytest collects | unittest collects |
|---|---|---|
tests/test_upload_service.py |
48 | 0 |
tests/test_codex_plugin.py |
10 | 0 |
tests/test_agent_ablation.py |
7 | 0 |
tests/test_artifact_download.py |
7 | 0 |
tests/test_tusd_hooks.py |
6 | 0 |
tests/test_native_ingestion.py |
4 | 0 |
tests/test_upload_page.py |
2 | 0 |
tests/test_tusd_live.py |
1 | 0 |
| Total | 85 | 0 |
The uncovered set is not peripheral. It is most of the media-ingestion path: the upload service, the tus hooks, the upload page, artifact download, and the end-to-end native ingestion tests that assert an imported video becomes searchable. Two of those test_native_ingestion.py tests currently fail on a local Windows checkout with AssertionError: Autonomous indexing did not finish without status polling, which is the kind of regression this gap is designed to hide. I have not yet established whether that failure is environmental, and I am not reporting it as a defect here.
This also means the contributor validation instructions and CI exercise different test sets. docs/CONTRIBUTING.md directs contributors to run the suite with pytest, so a contributor can see a failure that CI will never reproduce, and CI can pass a change that the documented command would have caught. (Separately, the documented command needed a correction, which I sent as #155.)
How can we reproduce it?
uv run --no-sync python -m unittest discover -s tests -p "test_upload_service.py" -q
# Ran 0 tests in 0.000s
# NO TESTS RAN
uv run --no-sync python -m pytest -q --collect-only tests/test_upload_service.py
# 48 tests collectedThe same holds for each of the eight files above. To see the aggregate:
uv run --no-sync python -m unittest discover -s tests -q # Ran 655 tests
uv run --no-sync python -m pytest -q --collect-only # 740 tests collectedEnvironment
- VidXP version: 0.4.0, source checkout at
24b916a - Operating system: Windows 11
- Installation: source,
uv sync --frozenwith the contributor extras - Python: 3.12.14 in the project venv
- CI reference:
.github/workflows/ci.yml, thepython -m unittest discover -s tests -qstep
The collection gap itself is platform-independent: it follows from the discovery mechanism, not from the operating system.
Logs or other details
Two directions, whichever you prefer:
- Run the suite in CI with
python -m pytestinstead. It collects both styles, so all 740 tests run and the 85 stop being invisible. Lowest-effort fix, but it changes the runner CI standardizes on. - Keep
unittestas the runner and add a collection guard — a test that asserts the discovered count matches the number of test functions undertests/, so a file written in the unsupported style fails loudly instead of silently.
I am happy to send either as a pull request, but the choice of runner is a maintainer decision so I would rather ask first than guess. Option 2 is the smaller change and does not disturb the existing CI contract.
Found while fixing two Windows path assertions (#154), one of which lives in a file CI never runs.
Activity
Hasnain2430 commentedon Sep 7, 2026
Following up to close out the one open question in the report: the two
test_native_ingestion.pyfailures I mentioned were environmental, not a defect. They pass once the scene model artifacts are prepared.So there is no ingestion bug here, and I am withdrawing that part of the report. The collection gap itself is unaffected: those 4 tests still never run in CI, along with the other 81.
Two smaller observations from confirming this, in case either is useful:
AssertionError: Autonomous indexing did not finish without status polling, which does not point at the cause.docs/CONTRIBUTING.mdtells contributors that an unprepared fresh checkout is expected and then directs them to run the suite, so this is the first thing some contributors will hit. A precondition check that skips or fails with "scene model artifacts are not prepared" would save the diagnosis. Happy to send that if you want it.vidxp prepare --modalities sceneprompts for confirmation and exits 0 without downloading anything when stdin is not a terminal. In a scripted or CI setting that reads as success, and the missing models only surface later as the misleading assertion above.--yesis the correct flag and works; the exit code on a declined-by-EOF prompt may still be worth a look.I can open either of those as its own issue rather than expanding this one, if you would prefer to keep this thread on the collection gap.
Hasnain2430 commentedon Sep 7, 2026
Correction to my previous comment: the second observation was wrong, and I withdraw it.
I claimed
vidxp prepareexits 0 when it declines at the confirmation prompt with no terminal on stdin. It does not.typer.confirm(..., abort=True)raisestyper.Abort, which Typer turns into exit code 1:The exit 0 I reported came from my own shell pipeline. I had piped the command into
tail, so the status I read wastail's, not the command's. The behavior is correct: the prompt refuses to proceed without confirmation,--yesis the documented way through, and the exit status reflects the refusal. Apologies for the noise.My first observation stands, and it is the smaller of the two: when scene model artifacts are missing,
test_native_ingestion.pyfails withAssertionError: Autonomous indexing did not finish without status polling, which does not name the cause. That is the only thing from that comment worth acting on, and it is minor next to the collection gap in the report above.