test: add real E2E integration suite across the ecosystem (issue #9)#15
Conversation
Adds scripts/e2e_ecosystem.py, exercising the already-compiled simplicio binary end to end in an ephemeral temp repo: clean install/init, valid vs corrupted config loading, project mapping, a full mechanical-edit task (input to final result), error/retry-exhaustion propagation, state persistence/resume across process runs, and compatibility with the published npm/PyPI/Homebrew packages (reusing verify_distribution_consistency.py). Wires it into .github/workflows/e2e.yml (PR + push + workflow_dispatch) so it is reproducible in CI, and documents commands/evidence/known findings in docs/e2e-integration.md. No cargo build/check/test is invoked anywhere. One real drift was found and intentionally left failing as evidence the gate works: version.txt/wrapper packages lag simplicio-update-manifest.json (3.0.2 vs 3.5.2) - documented as a follow-up in docs/e2e-integration.md. Closes #9
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b581c2d1c4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| - "scripts/e2e_ecosystem.py" | ||
| - "scripts/verify_distribution_consistency.py" |
There was a problem hiding this comment.
This workflow is the only new place that runs scripts/verify_distribution_consistency.py, but the PR path filter does not include several files that script reads, including README.md, INSTALL.md, install.sh, install.ps1, .github/workflows/release.yml, VERSION.md, and READMEs/*.md. A PR that reintroduces an invalid /main/ install URL in one of those files will not trigger this gate at all, so the added consistency check can be bypassed unless those inputs are added to the filter or the filter is removed.
Useful? React with 👍 / 👎.
| p = Path(explicit) | ||
| return p if p.exists() else None |
There was a problem hiding this comment.
When --binary is given as a relative path from the repo root, as documented with --binary ./simplicio.exe, this returns the relative Path unchanged, but every subprocess later runs with cwd set to the temporary repo. That makes Python look for the binary inside the temp repo and the suite crashes with FileNotFoundError; resolve the explicit path before storing it.
Useful? React with 👍 / 👎.
| # A corrupted config must not crash the runtime (no panic/traceback) and | ||
| # must still produce a diagnostic the caller can act on. | ||
| crashed = proc_invalid.returncode not in (0, 1) or "panic" in proc_invalid.stderr.lower() | ||
| invalid_ok = not crashed and (proc_invalid.stdout.strip() or proc_invalid.stderr.strip()) |
There was a problem hiding this comment.
When the corrupted-config doctor call emits diagnostics, this expression returns the diagnostic string instead of a boolean, so Report.add() stores a string in Result.passed. With --json, the summary then contains a huge string for passed rather than true/false, breaking consumers that rely on the advertised machine-readable output.
Useful? React with 👍 / 👎.
1 participant
Closes #9
Summary
Adds a real, runnable end-to-end integration test suite for this repo's
role in the Simplicio ecosystem.
wesleysimplicio/simpliciois the publicdistribution repo (compiled runtime binary + npm/pip/brew wrappers) —
the runtime/mapper/dev-cli/prompt/sprint source lives in sibling repos and
is only consumed here as optional PATH adapters (see
simplicio version --json->
components). Per the task constraints, nocargo build/check/testisinvoked anywhere; every check exercises the binary already committed in the
repo root, exactly as a real installer/CI consumer would.
scripts/e2e_ecosystem.py— runs each "cenario minimo" from the issueagainst a disposable ephemeral temp repo (created/torn down per run, never
touching real
~/.simpliciostate):simplicio setup)simplicio doctor)simplicio runtime map)simplicio edit,asserts the file content actually changed on disk)
(
simplicio editwith a bad pattern,simplicio recover)(
.simplicio/directory)simplicio version --jsoncompatibility matrix + reuse of the existing
scripts/verify_distribution_consistency.pypackaging gate).github/workflows/e2e.yml— runs the suite onwindows-latestfor PRsand pushes touching the binaries/wrappers/manifest/scripts, plus
workflow_dispatchfor on-demand runs; uploads stdout as a build artifactso a failing run always ships attached diagnostics.
docs/e2e-integration.md— commands, scenario-to-command mapping, and acaptured evidence run (9/10 scenarios passing).
Known finding (real, pre-existing, not introduced here)
Scenario 7b currently fails:
version.txt(3.0.2) and the npm/PyPI/Homebrewwrapper package versions lag
simplicio-update-manifest.json(3.5.2,published in commit
e24b426). This is exactly the failure mode issue #9asks the pipeline to catch ("Contratos incompativeis fazem o pipeline
falhar"), so it is left failing on purpose rather than hidden. Documented
as a follow-up in
docs/e2e-integration.mdrather than fixed in this PR tokeep the diff focused on the testing infrastructure itself.
Next steps (documented in docs/e2e-integration.md)
a stable cancellation-token flow.
simplicio contracts smoke/simplicio runtime smokein once theirexpected fixture layout is defined for a minimal ephemeral repo.
Test plan
python scripts/e2e_ecosystem.pyrun locally on Windows againstsimplicio-windows-x64.exev1.6.4 — 9/10 scenarios pass (seedocs/e2e-integration.mdfor full captured output and the one known,pre-existing, intentionally-uncovered finding).
cargo build/check/testinvoked.