diff --git a/CHANGELOG.md b/CHANGELOG.md index 71a50cda..f4107f1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ ## [4.18.17RC] - 2026-06-03 Unreleased in PyPI +- [CHANGED] update the architecture diagram (`doc/img/openTEPES_architecture.svg`) so it matches the code. The old picture used the planned folder names (`io/`, `schema.py`, `solver/`, `solve.py`); it now shows the real flat module names (`openTEPES_InputSchema.py`, `openTEPES_ProblemSolving.py`, and so on), with the five real solver modules. Implemented modules are shaded green and planned ones are left white, with a small legend. Also commit a rendered `doc/img/openTEPES_architecture.png` and point the `README.md` at the PNG instead of the SVG, so the diagram shows on pages that do not display SVG (such as the PyPI project page). Added a short `doc/img/README.md` explaining the SVG is the hand-drawn source (there is no generator script), that the PNG is rendered from it, and how to regenerate the PNG on Windows, macOS and Linux. - [FIXED] a binary investment problem (for example `IndBinNetInvest=1`) crashed under the HiGHS solver with `NoDualsError` on the first solve. `ProblemSolving` attaches the `dual` Suffix before the first solve only for a pure-LP model, because a mixed-integer problem has no duals; it then recovers the duals later by fixing the integer variables and re-solving as an LP. The check that decided "is this a pure LP" also required each variable to already have a value, but before the first solve no variable has a value yet, so a model with binary *investment* variables was wrongly treated as an LP, got the Suffix, and crashed when HiGHS was asked for duals it does not have. The check now looks only at whether a variable is integer/binary and unfixed, not at its value. Unit-commitment models did not hit this because their binary variables are given starting values. No change for any model that already worked. - [ADDED] a test (`test_binary_investment` in `tests/test_run.py`) for a binary (integer) investment decision, which no other test covered — every other case solves the investment variables as a continuous relaxation. It switches on `IndBinNetInvest` for the `9n` case so the single candidate line becomes a {0,1} build-or-not decision; the cost (254.337 MEUR) differs from the continuous result (252.201 MEUR) because the binary decision forces a full line build. The test also guards the fix above. A companion fixture `case_7d_binary` runs the case from a private temporary copy (so its solver log files do not clash with the other 9n tests on Windows, where a log file can stay open after a solve), applies the 7-day truncation, and overrides one or more columns of the Option file. - [CHANGED] split the CI workflow (`.github/workflows/ci.yml`) into two jobs to save time. A `fast` job runs the linter and the tests that do not solve a model, on all three operating systems and all three Python versions (3.11, 3.12, 3.13) — this is where import, packaging and Python-version problems show up. A `solve` job runs the full model test suite (every case, the multi-stage case, and Benders) once per operating system on Python 3.12, since the model results are the same on every Python version. Tests that solve a model are now marked with `@pytest.mark.solve` (registered in `pyproject.toml`); the `fast` job runs `-m "not solve"` and the `solve` job runs `-m solve`. Also added a per-test timeout on the solve job so a stuck solver fails quickly instead of using up the whole job, and turned on dependency caching for `uv`. No test was removed; the same tests still run, just spread across the two jobs. diff --git a/README.md b/README.md index d5f4acfd..20c7af71 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ It has been used by the **Ministry for the Ecological Transition and the Demogra The package is organised in six layers, from input/output (pure pandas, no Pyomo) up to result aggregation. Each module encodes its layer in the file name alongside the other `openTEPES_*.py` modules at the package root — `openTEPES_Input*` for the input-source layer (`openTEPES_InputSchema`, `openTEPES_InputSource`, `openTEPES_InputCSVSource`, `openTEPES_InputDuckDBSource`) and `openTEPES_ProblemSolving*` for the solver layer (`openTEPES_ProblemSolving`, `openTEPES_ProblemSolvingBenders`, `openTEPES_ProblemSolvingDualExtraction`, `openTEPES_ProblemSolvingPersistent`, `openTEPES_ProblemSolvingTuning`) — so concerns are addressable in code and the parallelisation modes (per-case sweep, in-memory overlay, post-build hot-swap) become first-class architectural seams. -![Architecture diagram](/doc/img/openTEPES_architecture.svg) +![Architecture diagram](/doc/img/openTEPES_architecture.png) # How to Cite diff --git a/doc/img/README.md b/doc/img/README.md new file mode 100644 index 00000000..21ae967d --- /dev/null +++ b/doc/img/README.md @@ -0,0 +1,30 @@ +# Architecture diagram + +Two files make up the architecture diagram shown in the project `README.md`: + +- `openTEPES_architecture.svg` — the source. It is hand-drawn SVG (the boxes and + text are plain SVG elements) and is edited directly; there is no script that + generates it. Open it in any text or vector editor, change what you need, save. +- `openTEPES_architecture.png` — a raster copy rendered from the SVG. The project + `README.md` embeds the PNG, because some pages (for example the PyPI project + page) do not display SVG images. + +Modules drawn in **green** are implemented (merged upstream); modules drawn in +**white** are planned and their names are indicative. + +After editing the SVG, regenerate the PNG so the two stay in step. Use a vector +tool such as [Inkscape](https://inkscape.org), which runs on Windows, macOS and +Linux: + +Windows (Command Prompt or PowerShell): + +``` +inkscape openTEPES_architecture.svg --export-type=png --export-filename=openTEPES_architecture.png -w 1710 +``` + +Linux / macOS (Inkscape, or `rsvg-convert` from librsvg): + +``` +inkscape openTEPES_architecture.svg --export-type=png --export-filename=openTEPES_architecture.png -w 1710 +rsvg-convert -w 1710 openTEPES_architecture.svg -o openTEPES_architecture.png +``` diff --git a/doc/img/openTEPES_architecture.png b/doc/img/openTEPES_architecture.png new file mode 100644 index 00000000..8c96f572 Binary files /dev/null and b/doc/img/openTEPES_architecture.png differ diff --git a/doc/img/openTEPES_architecture.svg b/doc/img/openTEPES_architecture.svg index 5ba8c1c6..7d21c3b9 100644 --- a/doc/img/openTEPES_architecture.svg +++ b/doc/img/openTEPES_architecture.svg @@ -20,15 +20,22 @@ .arrow-lbl { font-size: 11px; fill: #444; font-style: italic; } .arrow-lblP { font-size: 11px; fill: #7a4a8a; font-style: italic; font-weight: 700; } .footer { font-size: 11px; fill: #666; } + .legend { font-size: 11px; font-weight: 700; fill: #333; } - Proposed openTEPES architecture (layered, sector-parallel, sweep-aware) + openTEPES layered architecture — implemented and planned Pure-pandas I/O ▸ Pyomo construction ▸ Formulation ▸ Solver ▸ Results — symmetric Elec / Heat / H2 / Hydro · runner orchestrates sweeps + + + implemented (merged) + + planned + @@ -66,28 +73,28 @@ LAYER 2 - io/ - no pyomo — - pure pandas / SQL - DataFrames out + I/O + openTEPES_Input*.py + no pyomo — pure + pandas / SQL - - schema.py + + InputSchema.py TABLE_SPECS single source of truth - - source.py + + InputSource.py InputSource ABC open_source() · shapes - - csv_source.py + + InputCSVSource.py CSVSource historical backend - - duckdb_source.py + + InputDuckDBSource.py DuckDBSource streaming SQL backend @@ -111,10 +118,10 @@ LAYER 3 - model/ - Pyomo Sets, - Params, Variables - (no constraints yet) + Model + openTEPES_InputData.py + (one file today — + planned split) sets.py @@ -151,10 +158,11 @@ LAYER 4 - formulation/ - Pyomo Constraints - + Objective - sector-symmetric + Formulation + openTEPES_Model- + Formulation.py + (one file today — + planned split) CROSS-SECTOR @@ -238,28 +246,37 @@ LAYER 5 - solver/ - solver-agnostic + Solver + openTEPES_Problem- + Solving*.py + + + ProblemSolving.py + orchestrator entry + + + …Tuning.py + solver options - - solve.py - ProblemSolving entry + + …DualExtraction.py + duals via re-solve - - options/ - gurobi · cplex · highs · cbc + + …Persistent.py + appsi persistent - - tuning.py - barrier · scaling · numerics + + …Benders.py + L-shaped decomp - - resolve.py - ★ hot-swap re-solve loop + + resolve.py + ★ hot-swap (Mode C) - + Mode C: Param.store_values() + re-solve @@ -271,11 +288,11 @@ LAYER 6 - results/ - extract → CSV / DB / - plots / aggregates - sector-mirrored - to formulation/ + Results + openTEPES_Output- + Results.py + (one file today — + planned split) CROSS-SECTOR @@ -431,6 +448,6 @@ - Each box ↔ one file (~150–400 lines). Caps every module at ~1 000 LoC. Adding a new sector = new folder in Layer 4 + Layer 6 + entry in schema.py. ★ marks sweep-enabling additions. + Each box ↔ one module (~150–400 lines). Green = implemented (merged upstream); white = planned (names indicative). Adding a sector = new modules in Layer 4 + Layer 6 + an entry in openTEPES_InputSchema.py. ★ marks sweep-enabling additions.