feat: add TelecomBackup benchmark — power-backup scheduling for telecom sites (PowerSystems) - #105
zzy2007zzy2007 wants to merge 4 commits into
Conversation
🤖 AI Code Review (gemini-3-flash-preview)🇬🇧 English Analysis1. Executive Summary
2. AI Content Analysis
3. Engineering & Economic Assessment
4. Quality Assurance
5. Security & Privacy Check
🇨🇳 中文分析1. 摘要
2. AI 成分分析
3. 工程与经济评估
4. 质量保证
5. 安全与隐私检查
|
…om sites (PowerSystems) Time-sequenced on/off scheduling of telecom backup power supplies to maximize outage backup time while keeping LTE coverage >= 80%. Includes 8 fixed instances plus runtime generation (TELECOM_EVAL_GENERATE_SEED), a multi-rest rotation reference heuristic, cross-size determinism probes, and dockerized verification. Co-Authored-By: deepseek-v4-flash <deepseek-ai@claude-code-best.win>
7975f28 to
843389d
Compare
🤖 AI Code Review (gemini-3-flash-preview)🇬🇧 English Analysis1. Executive Summary
2. AI Content Analysis
3. Engineering & Economic Assessment
4. Quality Assurance
5. Security & Privacy Check
🇨🇳 中文分析1. 摘要
2. AI 成分分析
3. 工程与经济评估
4. Quality Assurance
5. 安全与隐私检查
|
y-ji24
left a comment
y-ji24
left a comment
There was a problem hiding this comment.
Review — TelecomBackup
Thank you for the contribution. Overall, I think TelecomBackup is a relatively mature benchmark implementation. The task definition is clear, the simulator / validator / evaluator / sandbox layers are reasonably well separated, and the 34 unit tests covering these components all pass.
I was also able to reproduce the baseline score of 176.25, which matches the documentation. The reference heuristic achieves 271.25 under direct simulation, and the generated-instance mechanism works as expected when --generate-seed 42 is enabled. The threat model is also discussed quite transparently in the README, which I appreciate.
That said, I found two issues that I think should be addressed before merging, since they directly affect the credibility and reproducibility of the evaluation.
1. Blocker — Runtime instance generation is not enforced in the default evaluation path
The README and Task.md describe runtime-generated instances as an important defense against hardcoding. However, this mechanism currently depends on the host environment variable TELECOM_EVAL_GENERATE_SEED.
The problem is that this variable is not enforced by the task configuration, while the default evaluation command in the README does not set it. As a result, the default evaluation path falls back to the 8 publicly available fixed instances.
I verified that this creates a practical hardcoding vulnerability.
I constructed a candidate by offline-optimizing the fixed instances and selecting the corresponding schedule based on the instance seed. The candidate does not contain explicit instance_N identifiers, and it passes the static checks and deterministic probes. It achieves:
valid = 1.0- score = 271.88
This is already slightly higher than the reference score of 271.25.
The issue here is not that the simulator itself is incorrect. Rather, under the default configuration, a candidate can memorize the public instances and obtain a near-reference score without actually solving the general problem.
I therefore recommend making runtime generation mandatory for the formal evaluation path. For example:
- Require
TELECOM_EVAL_GENERATE_SEEDfor official evaluation; - Fail explicitly if the required generation configuration is missing, rather than silently falling back to fixed instances;
- Keep the fixed-instance mode for local development / smoke testing, but clearly distinguish it from the official evaluation mode.
This would make the actual evaluation behavior consistent with the threat model described in the README.
2. Blocker — The documented reference-score reproduction command does not reproduce the reference score
The README / PR description currently provides a command along the lines of:
python verification/evaluate.py verification/ref_solver.py
as a way to reproduce the reference score.
However, running this command returns 0.0.
After checking the evaluation path, the reason appears to be that the reference solver does not contain the required EVOLVE-BLOCK marker and also contains the prohibited ref_solver token, so it is rejected during candidate preflight.
At the same time, result_log.txt correctly indicates that the reported reference score of 271.25 was obtained through direct simulation.
Therefore, the problem is mainly a mismatch between the documentation and the actual evaluation implementation:
The documented command does not reproduce the reference score reported by the benchmark.
I recommend resolving this in one of two ways:
- Provide a dedicated reference-scoring path that bypasses candidate completeness / anti-cheating checks; or
- Update the README to explicitly state that the reference score is obtained through direct simulation and provide the actual reproducible procedure.
The important point is that a maintainer or future contributor should be able to follow the documented instructions and obtain the documented reference score.
3. Non-blocking — Evaluation timeout should be aligned with the task's time budget
The framework-level evaluator currently has a default timeout of 300s, while the task defines a 60s per-instance budget.
Considering that the evaluation also performs deterministic probes and may run multiple instances, a compliant solver that legitimately consumes most of its per-instance budget could potentially exceed the framework-level timeout.
The README mentions adjusting FRONTIER_EVAL_EVALUATOR_TIMEOUT_S, but I would prefer to see the appropriate value explicitly included in the task configuration for the official evaluation path.
This would avoid relying on environment-specific manual configuration.
4. Non-blocking — The 300 / 60 / 10 second time-budget modes are not fully integrated into the framework
The three time-budget modes currently appear to be supported mainly through the local CLI, while the framework evaluation path still uses a hard-coded 60s budget in frontier_eval/evaluator.py, and run_eval.py does not expose the corresponding mode selection.
If these three modes are intended to be official benchmark settings, I recommend connecting them to the standard task configuration. Otherwise, it would be helpful to clarify in the documentation that they are development / local evaluation options rather than formal evaluation modes.
5. Minor — Repository integration and provenance
A few smaller issues would also be worth cleaning up:
- The top-level
TASK_DETAILS.mddoes not appear to have been updated, so the new benchmark is currently missing from the repository-level task index. - It would be useful to add a short provenance / design note explaining the origin of important parameters such as the 80% coverage requirement, -105 dBm threshold, and the power-consumption coefficients. Since this is an original benchmark, external references are not necessarily required, but documenting the rationale would improve long-term maintainability.
- There is a small numerical inconsistency in the documentation. My measurement gives an average improvement of approximately +57.5% for the fixed instances, with a range of +27.9% to +119.0%, while the README reports +54%. The +54% figure appears closer to the reference heuristic's improvement over the baseline. I recommend clarifying these two numbers and their respective definitions.
6. Documentation / collaboration — Please use English for future submissions
One additional request for future contributions: please make the PR description, task introduction, README-facing descriptions, and other repository-facing documentation in English.
This is not a blocker for the technical implementation itself, but I would appreciate keeping future PRs consistent with the language convention of the repository.
Overall
Overall, I think the core implementation is in good shape. The simulator, validator, tests, and task design are all reasonably solid, and I don't think the issues above require a major redesign.
My main concern is that the formal evaluation path currently does not fully enforce the assumptions described by the benchmark's threat model, and the documented reference-score reproduction procedure is not consistent with the actual implementation.
I would therefore recommend:
Request Changes
with priority on:
- Enforcing runtime-generated instances in the official evaluation path;
- Fixing the reference-score reproduction procedure;
- Aligning the evaluation timeout / time-budget configuration.
Once these are addressed, I think the benchmark will be in a much stronger position for long-term use in the repository.
…iew) Addresses y-ji24's Request-Changes review of PR Einsia#105. Blocker 1 -- runtime generation was opt-in, so the default evaluation path silently scored the 8 committed public instances and a candidate could memorise them (the reviewer built a seed-keyed lookup scoring 271.88). evaluate.py now has explicit `official` / `local` modes: official (selected by eval_command.txt) scores generated instances ONLY and requires TELECOM_EVAL_GENERATE_SEED, raising OfficialModeConfigError instead of falling back to the public instances. Blocker 2 -- the documented `evaluate.py verification/ref_solver.py` command returned 0.0 because the reference solver fails candidate preflight (no EVOLVE-BLOCK marker, forbidden token). Added an explicit `--reference` bypass that reproduces the documented 271.25. Also: - per-instance budget self-limits to FRONTIER_EVAL_EVALUATOR_TIMEOUT_S so a compliant solver cannot blow the framework cap; budget tiers are wired through TELECOM_EVAL_TIME_BUDGET instead of a hard-coded 60 s; - fixed the generator headroom figure (+57.5%, was +54%) and disambiguated it from the reference solver's +54% over the baseline; - added a provenance / design-rationale section and fixed the generator docstring's battery range; - translated Task.md to English (repo convention); - registered TelecomBackup in TASK_DETAILS.md and TASK_DETAILS_zh-CN.md. Tests: 34 -> 43 (official mode, --reference bypass, budget self-limiting). Co-Authored-By: deepseek-v4-flash <deepseek-ai@claude-code-best.win>
- evaluate.py: _fit_budget no longer declines to shrink when the framework cap is <= the fixed overhead; it now divides the (halved) cap across the runs, so n_runs * budget is always <= the observed cap. Previously a cap <= 30 s left the per-instance budget at 60 s and the run could exceed the cap it observed (the one falsified claim from verification). - evaluate.py: _select_probes dedupes, so official mode (all-generated pool) no longer runs one instance as a probe twice (n_runs 16 -> 14). - tests: lock that eval_command.txt still selects official mode (a silent drop of the prefix would revert the framework path to public-instance scoring); assert the ref-solver cheat is rejected by the static check (named reason), not merely by a runtime crash; give the --reference test a budget that cannot flake under CPU contention. - docs: correct "five modules" -> "four", the official-run worst-case arithmetic (14 runs = 840 s), and the AB-MCTS delta (+51%, was +52%). Tests: 43 -> 44. Co-Authored-By: deepseek-v4-flash <deepseek-ai@claude-code-best.win>
- evaluate.py: _fit_budget is now monotonic in the observed cap. Previously the branch at cap <= 30 s (available = cap - 30 <= 0) produced a discontinuity: cap 30.0 s gave a 1.07 s budget while cap 30.1 s gave 0.007 s. It now uses `cap - 30` above 2*RESERVE and `cap / 2` at or below it, which is continuous at 60 s, non-decreasing, and keeps n_runs * budget <= cap. - README: "44 tests" (was left at 43 when the guard test was added), and note that the budget bound covers solver time only -- fixed overhead (instance generation, interpreter start-up) is not shrinkable. Tests: 44 passing. Co-Authored-By: deepseek-v4-flash <deepseek-ai@claude-code-best.win>
🤖 AI Code Review (gemini-3-flash-preview)🇬🇧 English Analysis1. Executive Summary
2. AI Content Analysis
3. Engineering & Economic Assessment
4. Quality Assurance
5. Security & Privacy Check
🇨🇳 中文分析1. 摘要
2. AI 成分分析
3. 工程与经济评估
4. 质量保证
5. 安全与隐私检查
|
|
Thanks for the detailed review - all points are addressed. Nothing here required a 1. Blocker - runtime generation is now enforced in the official pathConfirmed: the generation seed was opt-in via a host env var that the task config never
I made official scoring generated-only rather than "fixed + generated" on purpose, so One caveat I want to be explicit about: a fixed, committed seed would not close the hole 2. Blocker - the reference-score reproduction pathConfirmed:
3. Timeout / time-budget configurationThe evaluator now derives a safe per-instance budget from On where that value lives: I checked, and the unified framework has no per-benchmark timeout 4. Repository integration and provenance
5. LanguageUnderstood, and apologies. This description and the task-facing docs are now in English, and Tests: 34 -> 44 (official/local modes, the |
2 participants
Summary
Adds TelecomBackup, an original PowerSystems benchmark: time-sequenced on/off
scheduling of telecom backup power supplies to maximize outage backup time while
keeping good LTE coverage (RSRP > -105 dBm) over >= 80% of the region at every instant.
Contents
baseline/solver.py- always-on template; theEVOLVE-BLOCKregion is the only partan agent may edit.
verification/- pure-stdlib simulator (coverage / power / battery), fixed-seedinstance generator, reference heuristic (multi-rest rotation), a 44-test suite, and the
evaluation entry point.
frontier_eval/- UnifiedTask metadata; Docker support(
verification/docker/Dockerfile), validated under WSL.TASK_DETAILS.md/TASK_DETAILS_zh-CN.mdand in the PowerSystemsdomain READMEs.
Evaluation modes
local(default)official(TELECOM_EVAL_MODE=official)In official mode a generation seed (
TELECOM_EVAL_GENERATE_SEED) is mandatory and thecommitted instances are never scored, so a memorised schedule buys nothing; a missing seed
is a hard error instead of a silent fall-back to the public instances.
Reference scores (deepseek-v4-flash agents; saved programs re-evaluated from the task directory)
verification/ref_solver.py): 271.2 min (+54%)AB-MCTS (15 iterations): 266.9 min
ShinkaEvolve 331.9 +/- 18.9, AB-MCTS 234.4 +/- 24.2
Reproducing the reference score