Frontier-Eng: Large-Scale Engineering Optimization Benchmark for AI Agents

English | 简体中文

Frontier-Eng is a benchmark designed to evaluate the ability of AI Agents to solve open-ended optimization problems in real-world engineering domains.

Unlike existing benchmarks that focus on Computer Science (CS) or purely abstract mathematical problems, Frontier-Eng focuses on engineering challenges with actual economic benefits and physical constraints. It is expected to cover multiple fields such as aerospace, civil engineering, EDA, bioengineering, and more.

🎯 Motivation

Current AI4Research evaluation systems have the following limitations:

1. Limited Evaluation Methods: Most adopt 0/1 binary evaluation or closed-interval rubrics, failing to effectively measure an Agent's ability to perform iterative optimization through interaction in an open world.
2. Domain Limitations: Existing benchmarks are mostly confined to the CS domain (e.g., code generation) or highly abstract real problems into math problems, stripping away real-world complexity and preventing Agents from utilizing rich external knowledge and tools.
3. Metric Bias: Traditional computational metrics focus on model average performance, whereas for engineering optimization problems, we should focus more on the Peak Performance a model can achieve on a single problem through exploration mechanisms.

Frontier-Eng aims to evaluate the ability of Agents to solve problems with practical value across a wide range of engineering disciplines by providing rich context and tool support.

🤝 Contribution Guidelines

We need the power of the community to expand the coverage of the Benchmark. We welcome the submission of new engineering problems via Pull Requests (PR). If you wish to contribute, please follow the standards and processes below:

AI-Assisted Contributions: We welcome contributions created with the assistance of AI tools. If you're using an AI assistant to help with your contribution, we recommend providing the prompt guide from this repository (AGENT.md) to ensure your AI assistant follows our standards and requirements. However, please do not over-rely on AI tools or leave the process entirely to AI. Human review and supervision are essential to ensure quality and correctness.

Sample Requirements

1. Reality Gap: Must be close to reality, considering real-world influencing factors, not purely abstract mathematics.
2. Economic Value: The problem should have clear engineering or economic value upon solution.
3. Verifiability: Must provide an executable verification program (Docker preferred) capable of completing the evaluation within an acceptable time.

Submission Format

Each Task should contain the following file structure:

<Domain_Name>/                       # Level 1 Directory: Domain Name (e.g., Astrodynamics)
├── README.md                        # [Required] Domain Overview (Default entry, EN or CN): Background & sub-task index
├── README_zh-CN.md                  # [Optional] Domain Overview (Chinese version. Used only if README.md is in English)
├── <Task_Name_A>/                   # Level 2 Directory: Specific Task Name (e.g., MannedLunarLanding)
│   ├── README.md                    # [Required] Navigation Doc: File structure, how to run & quick start
│   ├── README_zh-CN.md              # [Optional] Navigation Doc (Chinese version)
│   ├── Task.md                      # [Required] Task Detail Doc: Core doc including background, physical model, I/O definitions
│   ├── Task_zh-CN.md                # [Optional] Task Detail Doc (Chinese version)
│   ├── references/                  # References Directory
│   │   ├── constants.json           # Physical constants, simulation parameters, etc.
│   │   └── manuals.pdf              # Domain knowledge manual, physical equations, or constraints docs
│   ├── verification/                # Verification & Scoring System
│   │   ├── evaluator.py             # [Core] Scoring script entry point
│   │   ├── requirements.txt         # Dependencies required for the scoring environment
│   │   └── docker/                  # Environment containerization configuration
│   │       └── Dockerfile           # Ensures consistency of the evaluation environment
│   └── baseline/                    # [Optional] Baseline Solution / Example Code
│       ├── solution.py              # Reference code implementation
│       └── result_log.txt           # Execution log or scoring result of the reference code
└── <Task_Name_B>/                   # Another task under this domain
    └── ...

The above directory structure serves only as a reference template. Contributors may adjust the file organization based on specific circumstances, provided that all core elements (e.g., background, input/output, evaluation metrics) are included. Additionally, there are no restrictions on the programming language and format of the verification code.

Submission Guidelines

1. Keep test commands as short as possible (ideally single-line commands). Testing is mandatory before submission!

2. python verification/evaluator.py scripts/init.py # Run under benchmark, using verification/evaluator.py as the evaluation entry point. The target of the test, i.e., the target of agent evolution, is scripts/init.py.

3. python -m frontier_eval task=<task_name> algorithm.iterations=0 # Framework compatibility verification. Note: Please specify the task_name registered in the README.

4. Please avoid files containing private information, such as: .env, API keys, IDE configurations (.vscode/), temporary files (*.log, temp/, __pycache__, and personal test scripts). Also, please check that the submitted content does not contain absolute paths to avoid reproducibility issues and privacy leaks.

5. EVOLVE-BLOCK Markers (Required for ShinkaEvolve / ABMCTS): The file evolved by the agent (e.g., scripts/init.py, or language-specific baselines like malloclab-handout/mm.c) must include EVOLVE-BLOCK-START and EVOLVE-BLOCK-END markers to define the only editable region.

   • Keep the marker lines intact, and keep all code outside the markers read-only (CLI/I/O contracts, constraint checks, evaluator glue, etc.).
   • Use the correct comment style for your language:
     • Python: # EVOLVE-BLOCK-START / # EVOLVE-BLOCK-END
     • C/C++/CUDA/Rust/Swift: // EVOLVE-BLOCK-START / // EVOLVE-BLOCK-END

Contribution Process

We adopt the standard GitHub collaboration flow:

1. Fork this Repository: Click the "Fork" button in the top right corner to copy the project to your GitHub account.
2. Create Branch:

• Clone your Fork locally.
• Create a new branch for development, recommended naming format: feat/<Domain>/<TaskName> (e.g., feat/Astrodynamics/MarsLanding).

3. Add/Modify Content:

• Add your engineering problem files following the submission format above.
• Ensure all necessary explanatory documentation and verification code are included.

4. Local Test: Run evaluator.py or build the Docker image to ensure the evaluation logic is correct and runs normally.
5. Submit Pull Request (PR):

• Push changes to your remote Fork.
• Initiate a Pull Request to the main branch of this repository.
• PR Description: Please briefly explain the background, source, and how to run the verification code for the Task.

6. Code Review:

• Agent Review: After submitting the PR, an AI Agent will first conduct an automated preliminary review (including code standards, basic logic verification, etc.) and may propose modifications directly in the PR.
• Maintainer Review: After the Agent review passes, maintainers will conduct a final re-check. Once confirmed correct, your contribution will be merged.

---

💡 If this is your first contribution or you have questions about the directory structure, feel free to submit an Issue for discussion first.

📊 Task Progress & Planning

The table below lists the current coverage of domain tasks in the Benchmark. We welcome not only code contributions but also ideas for challenging new engineering problems from the community.

Domain	Task Name	Status	Contributor	Reviewer	Remarks
Astrodynamics	MannedLunarLanding	Completed	@jdp22	@jdp22	Lunar soft landing trajectory optimization
ParticlePhysics	MuonTomography	Completed	@SeanDF333	@ahydchh	Muon detector placement optimization under flux, budget, and excavation constraints
Kernel Engineering	MLA	Completed	@ahydchh	@ahydchh	GPUMode
	TriMul	Completed	@ahydchh	@ahydchh	GPUMode
	FlashAttention	Completed	@Geniusyingmanji	@ahydchh	Optimize a causal scaled dot-product attention forward kernel for GPU execution
Single Cell Analysis	denoising	Completed	@ahydchh	@ahydchh	Open Problems in Single-Cell Analysis
	perturbation_prediction	Completed	@llltttwww	@llltttwww	NeurIPS 2023 scPerturb
	predict_modality	Completed	@llltttwww	@llltttwww	NeurIPS 2021, RNA→ADT
QuantumComputing	routing qftentangled	Completed	@ahydchh	@ahydchh	Routing-Oriented Optimization
	clifford t synthesis	Completed	@ahydchh	@ahydchh	Clifford+T Synthesis Optimization
	cross target qaoa	Completed	@ahydchh	@ahydchh	Cross-Target Robust Optimization
Cryptographic	AES-128 CTR	Completed	@ahydchh	@ahydchh	Advanced Encryption Standard, 128-bit key, Counter mode
	SHA-256	Completed	@ahydchh	@ahydchh	Secure Hash Algorithm 256-bit
	SHA3-256	Completed	@ahydchh	@ahydchh	Secure Hash Algorithm 3 256-bit
CommunicationEngineering	LDPCErrorFloor	Completed	@WayneJin0918	@ahydchh	LDPC code error floor estimation using importance sampling for trapping sets
	PMDSimulation	Completed	@WayneJin0918	@ahydchh	Polarization Mode Dispersion simulation with importance sampling for rare outage events
	RayleighFadingBER	Completed	@WayneJin0918	@ahydchh	BER analysis under Rayleigh fading with importance sampling for deep fade events
EnergyStorage	BatteryFastChargingProfile	Completed	@kunkun04	@ahydchh	Fast-charge current-profile optimization for a lithium-ion cell under voltage, thermal, and degradation constraints
	BatteryFastChargingSPMe	Completed	@kunkun04	@ahydchh	Staged fast-charge optimization under a reduced SPMe-T-Aging style electrochemical, thermal, plating, and aging model
ReactionOptimisation	snar_multiobjective	Completed	@ahydchh	@ahydchh	Continuous-flow SnAr reaction optimization with a Pareto trade-off between productivity and waste
	mit_case1_mixed	Completed	@ahydchh	@ahydchh	Mixed-variable reaction yield maximization with continuous process settings and a categorical catalyst
	reizman_suzuki_pareto	Completed	@ahydchh	@ahydchh	Reizman Suzuki emulator Pareto optimization over catalyst choice and operating conditions
	dtlz2_pareto	Completed	@ahydchh	@ahydchh	DTLZ2 Pareto-front approximation task integrated through the unified evaluation pipeline
MolecularMechanics	weighted_parameter_coverage	Completed	@ahydchh	@ahydchh	Rare force-field parameter coverage under a molecule budget
	diverse_conformer_portfolio	Completed	@ahydchh	@ahydchh	Low-energy, high-diversity conformer portfolio selection
	torsion_profile_fitting	Completed	@ahydchh	@ahydchh	Force-field torsion-scale fitting against target energy profiles
Optics	adaptive_constrained_dm_control	Completed	@ahydchh	@ahydchh	Constrained deformable mirror control
	adaptive_temporal_smooth_control	Completed	@ahydchh	@ahydchh	Temporal smoothness versus correction quality
	adaptive_energy_aware_control	Completed	@ahydchh	@ahydchh	Energy-aware adaptive optics control
	adaptive_fault_tolerant_fusion	Completed	@ahydchh	@ahydchh	Fault-tolerant multi-WFS fusion
	phase_weighted_multispot_single_plane	Completed	@ahydchh	@ahydchh	Single-plane weighted multispot phase DOE
	phase_fourier_pattern_holography	Completed	@ahydchh	@ahydchh	Fourier pattern holography
	phase_dammann_uniform_orders	Completed	@ahydchh	@ahydchh	Dammann grating uniform diffraction orders
	phase_large_scale_weighted_spot_array	Completed	@ahydchh	@ahydchh	Large-scale weighted spot array synthesis
	fiber_wdm_channel_power_allocation	Completed	@ahydchh	@ahydchh	WDM channel and launch power allocation
	fiber_mcs_power_scheduling	Completed	@ahydchh	@ahydchh	Joint MCS and power scheduling
	fiber_dsp_mode_scheduling	Completed	@ahydchh	@ahydchh	Receiver DSP mode scheduling
	fiber_guardband_spectrum_packing	Completed	@ahydchh	@ahydchh	Spectrum packing with guard-band constraints
	holographic_multifocus_power_ratio	Completed	@ahydchh	@ahydchh	Multi-focus power ratio control
	holographic_multiplane_focusing	Completed	@ahydchh	@ahydchh	Multi-plane holographic focusing
	holographic_multispectral_focusing	Completed	@ahydchh	@ahydchh	Multispectral holographic focusing
	holographic_polarization_multiplexing	Completed	@ahydchh	@ahydchh	Polarization-multiplexed holography
Computer Systems	Malloc Lab	Completed	@ahydchh	@ahydchh	Dynamic memory allocation
EngDesign	CY_03, WJ_01, XY_05, AM_02, AM_03, YJ_02, YJ_03	Completed	@ahydchh	@ahydchh	EngDesign
InventoryOptimization	tree_gsm_safety_stock	Completed	@ahydchh	@ahydchh	Tree-structured multi-echelon safety-stock placement (GSM)
	general_meio	Completed	@ahydchh	@ahydchh	General-topology MEIO with simulation-based objective
	joint_replenishment	Completed	@ahydchh	@ahydchh	Multi-SKU joint replenishment with shared setup cost
	finite_horizon_dp	Completed	@ahydchh	@ahydchh	Finite-horizon stochastic inventory control via time-varying policy
	disruption_eoqd	Completed	@ahydchh	@ahydchh	EOQ lot-sizing optimization under supply disruptions
PyPortfolioOpt	robust_mvo_rebalance	Completed	@ahydchh	@ahydchh	Robust mean-variance rebalancing with sector/factor/turnover constraints
	cvar_stress_control	Completed	@ahydchh	@ahydchh	CVaR stress-controlled portfolio allocation under return and exposure constraints
	discrete_rebalance_mip	Completed	@ahydchh	@ahydchh	Discrete lot-constrained rebalancing with mixed-integer optimization
JobShop	abz	Completed	@ahydchh	@ahydchh	Classical JSSP ABZ family (Adams, Balas, Zawack 1988)
	ft	Completed	@ahydchh	@ahydchh	Classical JSSP FT family (Fisher and Thompson 1963)
	la	Completed	@ahydchh	@ahydchh	Classical JSSP LA family (Lawrence 1984)
	orb	Completed	@ahydchh	@ahydchh	Classical JSSP ORB family (Applegate and Cook 1991)
	swv	Completed	@ahydchh	@ahydchh	Classical JSSP SWV family (Storer, Wu, Vaccari 1992)
	ta	Completed	@ahydchh	@ahydchh	Classical JSSP TA family (Taillard 1993)
	yn	Completed	@ahydchh	@ahydchh	Classical JSSP YN family (Yamada and Nakano 1992)
StructuralOptimization	ISCSO2015	Completed	@yks23	@yks23	45-bar 2D truss size + shape
	ISCSO2023	Completed	@yks23	@yks23	284-member 3D truss sizing
	TopologyOptimization	Completed	@Geniusyingmanji	@ahydchh	MBB beam 2D topology optimization (SIMP), Continuous, volume-constrained, compliance minimization
Robotics	DynamicObstacleAvoidanceNavigation	Completed	@MichaelCaoo	@yks23	Navigate a differential-drive robot from start to goal
	QuadrupedGaitOptimization	Completed	@MichaelCaoo	@yks23	Maximize the forward locomotion speed of a quadruped robot by optimizing 8 gait parameters
	RobotArmCycleTimeOptimization	Completed	@MichaelCaoo	@yks23	Minimize the motion time of a 7-DOF KUKA LBR iiwa arm moving from a start to a goal configuration, collision-free
	PIDTuning	Completed	@Geniusyingmanji	@ahydchh	Tune a cascaded PID controller for a 2D quadrotor across multiple flight scenarios
	UAVInspectionCoverageWithWind	Completed	@MichaelCaoo	@ahydchh	UAV inspection under wind field disturbance
Aerodynamics	CarAerodynamicsSensing	Completed	@LeiDQ, @llltttwww	@llltttwww	Sensor placement on 3D car surface for pressure field reconstruction
WirelessChannelSimulation	HighReliableSimulation	Completed	@tonyhaohan	@yks23, @ahydchh	BER estimation with importance sampling for Hamming(127,120)

💡 Have an idea for a new engineering problem? Even if you cannot provide complete verification code for now, we highly welcome you to share good Task concepts! Please create an Issue detailing the real-world background and engineering value of the problem. After discussion and confirmation, we will add it to the table above to rally community power to solve it together.

🧪 Evaluation Framework

An initial integration between some evaluation algorithms and benchmarks has been implemented. The core implementation is located in ./frontier_eval. For usage instructions, see the Evaluation README. Note: some optional algorithms/benchmarks require extra repos under third_party/ (local clones); the Evaluation README documents how to set them up.

💬 Join the Community

Welcome to our developer community! Whether you want to discuss new engineering problem concepts, find task collaborators, or encounter technical issues during your contribution, you can always communicate with us in the group.

• 🟢 Feishu (Lark): Click here to join our Feishu discussion group

• 🔜 Discord: Click here to join our Discord community