Conversation
There was a problem hiding this comment.
Pull request overview
Ports the existing FDB “tool” end-to-end tests into the repo’s CI/CD flow by adding a pytest-based harness plus CTest discovery/execution, along with new test fixtures and example GRIB inputs.
Changes:
- Added pytest harness (dynamic parametrization + script runner) to execute many shell-based tool tests.
- Added extensive shell-based e2e coverage for FDB CLI tools across “subtocs” and “no_subtocs” setups.
- Integrated pytest discovery into CTest and added example GRIB data files used by the tests.
Reviewed changes
Copilot reviewed 104 out of 111 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/fdb_e2e/tool_tests/util/run_sh.py | Python helper to execute shell scripts and stream output into logs. |
| tests/fdb_e2e/tool_tests/subtocs/write/simple.sh | Subtocs: e2e coverage for fdb-write basic behavior and file layout counts. |
| tests/fdb_e2e/tool_tests/subtocs/write/intersection_filter.sh | Subtocs: e2e coverage for include/exclude filter intersection behavior. |
| tests/fdb_e2e/tool_tests/subtocs/write/include_filter.sh | Subtocs: e2e coverage for --include-filter behavior. |
| tests/fdb_e2e/tool_tests/subtocs/write/exclude_filter.sh | Subtocs: e2e coverage for --exclude-filter behavior. |
| tests/fdb_e2e/tool_tests/subtocs/wipe/simple.sh | Subtocs: e2e coverage for fdb-wipe basic behavior. |
| tests/fdb_e2e/tool_tests/subtocs/wipe/ranges.sh | Subtocs: e2e coverage for fdb-wipe with range selectors. |
| tests/fdb_e2e/tool_tests/subtocs/wipe/porcelain.sh | Subtocs: e2e coverage for fdb-wipe --porcelain output. |
| tests/fdb_e2e/tool_tests/subtocs/wipe/partial_complete.sh | Subtocs: e2e coverage for partial wipe across differing second-level metadata. |
| tests/fdb_e2e/tool_tests/subtocs/wipe/partial.sh | Subtocs: e2e coverage for partial wipe that deletes all data. |
| tests/fdb_e2e/tool_tests/subtocs/wipe/overspecified.sh | Subtocs: e2e coverage for overspecified wipe requests failing safely. |
| tests/fdb_e2e/tool_tests/subtocs/wipe/minimum_keys.sh | Subtocs: e2e coverage for minimum-key enforcement in wipe. |
| tests/fdb_e2e/tool_tests/subtocs/wipe/extra.sh | Subtocs: e2e coverage for wipe behavior with unexpected extra files. |
| tests/fdb_e2e/tool_tests/subtocs/wipe/duplicate_data.sh | Subtocs: e2e coverage for wiping duplicated writes. |
| tests/fdb_e2e/tool_tests/subtocs/wipe/damaged.sh | Subtocs: e2e coverage for wipe robustness with damaged/missing data files. |
| tests/fdb_e2e/tool_tests/subtocs/root/create.sh | Subtocs: e2e coverage for fdb-root --create and fdb-where behavior. |
| tests/fdb_e2e/tool_tests/subtocs/read/steprange.sh | Subtocs: e2e coverage for fdb-read steprange selection. |
| tests/fdb_e2e/tool_tests/subtocs/read/simple.sh | Subtocs: e2e coverage for fdb-read simple request and GRIB compare. |
| tests/fdb_e2e/tool_tests/subtocs/read/schema_change.sh | Subtocs: e2e coverage for schema changes across days and readback. |
| tests/fdb_e2e/tool_tests/subtocs/read/quantile.sh | Subtocs: e2e coverage for reading quantile GRIB content. |
| tests/fdb_e2e/tool_tests/subtocs/read/extract.sh | Subtocs: e2e coverage for fdb-read --extract. |
| tests/fdb_e2e/tool_tests/subtocs/purge/simple.sh | Subtocs: e2e coverage for fdb-purge deduplication behavior. |
| tests/fdb_e2e/tool_tests/subtocs/purge/porcelain.sh | Subtocs: e2e coverage for fdb-purge --porcelain output. |
| tests/fdb_e2e/tool_tests/subtocs/purge/overspecified.sh | Subtocs: e2e coverage for overspecified purge requests failing safely. |
| tests/fdb_e2e/tool_tests/subtocs/overlay/wipe.sh | Subtocs: e2e coverage for overlay interaction with wipe behavior. |
| tests/fdb_e2e/tool_tests/subtocs/overlay/remove.sh | Subtocs: e2e coverage for overlay removal. |
| tests/fdb_e2e/tool_tests/subtocs/overlay/read.sh | Subtocs: e2e coverage for reading overlaid data via extract. |
| tests/fdb_e2e/tool_tests/subtocs/overlay/permitted_keys.sh | Subtocs: e2e coverage for --variable-keys restrictions. |
| tests/fdb_e2e/tool_tests/subtocs/overlay/list.sh | Subtocs: e2e coverage for fdb-list visibility under overlays. |
| tests/fdb_e2e/tool_tests/subtocs/overlay/hidden.sh | Subtocs: e2e coverage for overlays against hidden/existing DBs. |
| tests/fdb_e2e/tool_tests/subtocs/overlay/force.sh | Subtocs: e2e coverage for forcing overlay against existing DB. |
| tests/fdb_e2e/tool_tests/subtocs/list/ranges.sh | Subtocs: e2e coverage for list range expansion. |
| tests/fdb_e2e/tool_tests/subtocs/list/porcelain.sh | Subtocs: e2e coverage contrasting porcelain vs normal list output. |
| tests/fdb_e2e/tool_tests/subtocs/list/minimum_keys.sh | Subtocs: e2e coverage for minimum-key enforcement in list. |
| tests/fdb_e2e/tool_tests/subtocs/list/masking.sh | Subtocs: e2e coverage for masking vs --full listing. |
| tests/fdb_e2e/tool_tests/subtocs/list/locations.sh | Subtocs: e2e coverage for --location output. |
| tests/fdb_e2e/tool_tests/subtocs/list/all_full.sh | Subtocs: e2e coverage for --all + --full enumeration behavior. |
| tests/fdb_e2e/tool_tests/subtocs/info/version.sh | Subtocs: e2e coverage for fdb-info --version / --all. |
| tests/fdb_e2e/tool_tests/subtocs/hide/wipe.sh | Subtocs: e2e coverage for wipe behavior when data is hidden. |
| tests/fdb_e2e/tool_tests/subtocs/hide/read.sh | Subtocs: e2e coverage for reading with hidden data. |
| tests/fdb_e2e/tool_tests/subtocs/hide/list.sh | Subtocs: e2e coverage for list visibility changes after hide/write. |
| tests/fdb_e2e/tool_tests/subtocs/grib2fdb5/simple.sh | Subtocs: e2e coverage for grib2fdb5 basic import behavior. |
| tests/fdb_e2e/tool_tests/subtocs/grib2fdb5/ignored_options.sh | Subtocs: e2e coverage for grib2fdb5 ignored options behavior. |
| tests/fdb_e2e/tool_tests/subtocs/grib2fdb5/check.sh | Subtocs: e2e coverage for grib2fdb5 verification options. |
| tests/fdb_e2e/tool_tests/no_subtocs/write/simple.sh | No-subtocs: e2e coverage for fdb-write basic behavior and file layout counts. |
| tests/fdb_e2e/tool_tests/no_subtocs/write/intersection_filter.sh | No-subtocs: e2e coverage for include/exclude filter intersection behavior. |
| tests/fdb_e2e/tool_tests/no_subtocs/write/include_filter.sh | No-subtocs: e2e coverage for --include-filter behavior. |
| tests/fdb_e2e/tool_tests/no_subtocs/write/exclude_filter.sh | No-subtocs: e2e coverage for --exclude-filter behavior. |
| tests/fdb_e2e/tool_tests/no_subtocs/wipe/simple.sh | No-subtocs: e2e coverage for fdb-wipe basic behavior. |
| tests/fdb_e2e/tool_tests/no_subtocs/wipe/ranges.sh | No-subtocs: e2e coverage for fdb-wipe with range selectors. |
| tests/fdb_e2e/tool_tests/no_subtocs/wipe/porcelain.sh | No-subtocs: e2e coverage for fdb-wipe --porcelain output. |
| tests/fdb_e2e/tool_tests/no_subtocs/wipe/partial_complete.sh | No-subtocs: e2e coverage for partial wipe across differing second-level metadata. |
| tests/fdb_e2e/tool_tests/no_subtocs/wipe/partial.sh | No-subtocs: e2e coverage for partial wipe that deletes all data. |
| tests/fdb_e2e/tool_tests/no_subtocs/wipe/overspecified.sh | No-subtocs: e2e coverage for overspecified wipe requests failing safely. |
| tests/fdb_e2e/tool_tests/no_subtocs/wipe/minimum_keys.sh | No-subtocs: e2e coverage for minimum-key enforcement in wipe. |
| tests/fdb_e2e/tool_tests/no_subtocs/wipe/extra.sh | No-subtocs: e2e coverage for wipe behavior with unexpected extra files. |
| tests/fdb_e2e/tool_tests/no_subtocs/wipe/duplicate_data.sh | No-subtocs: e2e coverage for wiping duplicated writes. |
| tests/fdb_e2e/tool_tests/no_subtocs/wipe/damaged.sh | No-subtocs: e2e coverage for wipe robustness with damaged/missing data files. |
| tests/fdb_e2e/tool_tests/no_subtocs/root/create.sh | No-subtocs: e2e coverage for fdb-root --create and fdb-where behavior. |
| tests/fdb_e2e/tool_tests/no_subtocs/read/steprange.sh | No-subtocs: e2e coverage for fdb-read steprange selection. |
| tests/fdb_e2e/tool_tests/no_subtocs/read/simple.sh | No-subtocs: e2e coverage for fdb-read simple request and GRIB compare. |
| tests/fdb_e2e/tool_tests/no_subtocs/read/schema_change.sh | No-subtocs: e2e coverage for schema changes across days and readback. |
| tests/fdb_e2e/tool_tests/no_subtocs/read/quantile.sh | No-subtocs: e2e coverage for reading quantile GRIB content. |
| tests/fdb_e2e/tool_tests/no_subtocs/read/extract.sh | No-subtocs: e2e coverage for fdb-read --extract. |
| tests/fdb_e2e/tool_tests/no_subtocs/purge/simple.sh | No-subtocs: e2e coverage for fdb-purge deduplication behavior. |
| tests/fdb_e2e/tool_tests/no_subtocs/purge/porcelain.sh | No-subtocs: e2e coverage for fdb-purge --porcelain output. |
| tests/fdb_e2e/tool_tests/no_subtocs/purge/overspecified.sh | No-subtocs: e2e coverage for overspecified purge requests failing safely. |
| tests/fdb_e2e/tool_tests/no_subtocs/overlay/wipe.sh | No-subtocs: e2e coverage for overlay interaction with wipe behavior. |
| tests/fdb_e2e/tool_tests/no_subtocs/overlay/remove.sh | No-subtocs: e2e coverage for overlay removal. |
| tests/fdb_e2e/tool_tests/no_subtocs/overlay/read.sh | No-subtocs: e2e coverage for reading overlaid data via extract. |
| tests/fdb_e2e/tool_tests/no_subtocs/overlay/permitted_keys.sh | No-subtocs: e2e coverage for --variable-keys restrictions. |
| tests/fdb_e2e/tool_tests/no_subtocs/overlay/list.sh | No-subtocs: e2e coverage for fdb-list visibility under overlays. |
| tests/fdb_e2e/tool_tests/no_subtocs/overlay/hidden.sh | No-subtocs: e2e coverage for overlays against hidden/existing DBs. |
| tests/fdb_e2e/tool_tests/no_subtocs/overlay/force.sh | No-subtocs: e2e coverage for forcing overlay against existing DB. |
| tests/fdb_e2e/tool_tests/no_subtocs/list/ranges.sh | No-subtocs: e2e coverage for list range expansion. |
| tests/fdb_e2e/tool_tests/no_subtocs/list/porcelain.sh | No-subtocs: e2e coverage contrasting porcelain vs normal list output. |
| tests/fdb_e2e/tool_tests/no_subtocs/list/minimum_keys.sh | No-subtocs: e2e coverage for minimum-key enforcement in list. |
| tests/fdb_e2e/tool_tests/no_subtocs/list/masking.sh | No-subtocs: e2e coverage for masking vs --full listing. |
| tests/fdb_e2e/tool_tests/no_subtocs/list/locations.sh | No-subtocs: e2e coverage for --location output. |
| tests/fdb_e2e/tool_tests/no_subtocs/list/all_full.sh | No-subtocs: e2e coverage for --all + --full enumeration behavior. |
| tests/fdb_e2e/tool_tests/no_subtocs/info/version.sh | No-subtocs: e2e coverage for fdb-info --version / --all. |
| tests/fdb_e2e/tool_tests/no_subtocs/hide/wipe.sh | No-subtocs: e2e coverage for wipe behavior when data is hidden. |
| tests/fdb_e2e/tool_tests/no_subtocs/hide/read.sh | No-subtocs: e2e coverage for reading with hidden data. |
| tests/fdb_e2e/tool_tests/no_subtocs/hide/list.sh | No-subtocs: e2e coverage for list visibility changes after hide/write. |
| tests/fdb_e2e/tool_tests/no_subtocs/grib2fdb5/simple.sh | No-subtocs: e2e coverage for grib2fdb5 basic import behavior. |
| tests/fdb_e2e/tool_tests/no_subtocs/grib2fdb5/check.sh | No-subtocs: e2e coverage for grib2fdb5 verification options. |
| tests/fdb_e2e/tool_tests/conftest.py | Pytest collection helpers for enumerating .sh scripts as parametrized test inputs. |
| tests/fdb_e2e/tool_tests/Remarks.md | Notes on test migration status and known failures/todos. |
| tests/fdb_e2e/tool_tests/CMakeLists.txt | CTest integration: pytest discovery + per-test add_test wiring and env setup. |
| tests/fdb_e2e/python_api/util/tools.py | GRIB metadata modification utilities used by tests to generate fixtures. |
| tests/fdb_e2e/python_api/util/dispatcher.py | Helper for running CLI commands and capturing output to a log file. |
| tests/fdb_e2e/python_api/util/init.py | Declares util package for test helpers. |
| tests/fdb_e2e/data/steprange.grib | Added sample GRIB input for steprange test cases. |
| tests/fdb_e2e/data/quantile.grib | Added sample GRIB input for quantile test cases. |
| tests/fdb_e2e/CMakeLists.txt | Adds the new tool_tests subdirectory to the build/test tree. |
| tests/CMakeLists.txt | Wires the new fdb_e2e test suite into the overall tests build. |
| pytest.toml | Attempts to define pytest options and markers for the added tests. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## develop #254 +/- ##
===========================================
+ Coverage 70.72% 70.74% +0.01%
===========================================
Files 372 372
Lines 23309 23309
Branches 2455 2454 -1
===========================================
+ Hits 16486 16490 +4
+ Misses 6823 6819 -4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
tbkr
force-pushed
the
feature/cli_e2e_tests
branch
5 times, most recently
from
ac2ccf9 to
e3a42e5
Compare
tbkr
force-pushed
the
feature/cli_e2e_tests
branch
from
c8cac57 to
22e4c4f
Compare
- Implemented e2e API tests for list - Implementation of simple fdb setup - Refactor into subtocs and no_subtocs - Working parameterized tests - All marsflow tests implemented - Added ctest-pytest discovery - Added example GRIB files
- xfail for the test case linked to FDB-652 - adjusted the expected output for the other test cases
tbkr
force-pushed
the
feature/cli_e2e_tests
branch
from
ec79098 to
3d39ac0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Porting marflow fdb-tool tests to the cicd
Contributor Declaration
By opening this pull request, I affirm the following:
🌈🌦️📖🚧 Documentation FDB 🚧📖🌦️🌈
https://sites.ecmwf.int/docs/fdb/pull-requests/PR-254