| myst |
|
|---|
This topic documents run configuration files, per-task configuration, command-line flags, scoring formula, and agent registry.
A run configuration defines a single experiment. Start from a file under
example_configs/ and copy it when creating a new experiment.
| Field | Type | Description |
|---|---|---|
agent.template |
string | Agent to run. One of the supported agents. |
tasks |
list of strings | Task selectors relative to tasks/. Use all for every task, a category prefix for a group, or a full path for a single task. |
target_gpu_model |
string | Target GPU model, for example MI300X, MI325X, MI300A, MI355X, or RDNA4; MI300 remains the generic gfx942 fallback. Used to select the Docker image architecture, set PYTORCH_ROCM_ARCH, and name the workspace. |
log_directory |
string | Directory for run logs. |
workspace_directory_prefix |
string | Prefix for the workspace directory. The full name is <prefix>_<gpu>_<agent>. |
GEAK resolves GPU IDs within the process-visible namespace; a masked Docker
worker uses logical device zero. Host allocations remain controlled by the
Docker/Slurm runner. GEAK search settings use the agent mapping in the
GEAK guide. The geak_v4 template
name is a registry alias for this same integration.
Agent-specific defaults live in agents/<agent_name>/agent_config.yaml.
Supported fields in the run config's agent mapping override those defaults;
see agent settings
and the selected integration's directory for supported fields and precedence.
| Field | Type | Default | Description |
|---|---|---|---|
agent.budget_mode |
iterations or time |
iterations |
Cursor, Claude Code, Codex, and DeepSeek Harness. time resumes the agent session with the remaining timeout_seconds instead of asking for max_iterations versions. Any other value, or time with another agent, fails when the run starts. See Iteration and time budgets. |
Each launched task also gets an agent cost record at
<run_dir>/agent_costs/<task_workspace>.json. Unreported dollars and tokens
are null. See Agent cost records
for the fields and per-agent sources.
Example:
agent:
template: cursor
tasks:
- hip2hip/GELU
- triton2triton/vllm/triton_rms_norm
target_gpu_model: MI300
log_directory: logs
workspace_directory_prefix: workspaceevaluation_tools configures optional, isolated kernel-analysis sidecars. The
section is disabled when it is absent, null, or false. A configured mapping
with no enabled tools is also disabled unless the host runner supplies
AKA_EVAL_TOOLS; for a mapping, that host subset replaces its enabled value.
The built-in IDs are triton_fpsan, gpu_asan, rocjitsu,
rocjitsu_waitcheck, rocjitsu_consan, and hip_fpsan.
Sidecar build locks, integrated positive controls, and end-to-end fixtures
currently exist only for gfx950. The ROCm 10 migration has five passing
startup controls; GPU ASan remains unqualified, as recorded in the
runtime compatibility guide. Each applicable candidate still needs a task-specific
adapter and attestation, and enabling an image alone does not imply that a
kernel was analyzed. See Check kernels with evaluation
tools for the support matrix and operational
requirements.
When tools are enabled, the selected scoring image must resolve to the same
immutable local Docker image ID as the scoring manifest pinned in
docker/eval-tools/images.lock.yaml. The runner rejects a different build, launches by the verified ID,
and records both the selected reference and verified ID in plan source evidence.
Worker reports live at repository-root
.eval-tool-artifacts/<worker-label>. The runner mounts that specific host
directory read/write into both sidecars and the scoring container; the latter
submount remains writable when the quality-loop repository root is read-only.
| Field | Type | Default | Description |
|---|---|---|---|
evaluation_tools.enabled |
boolean, string, or list of strings | empty | Tool IDs to plan. true expands to all six built-ins; false disables the feature. A single string is accepted. Hyphens are normalized to underscores. When the host sets AKA_EVAL_TOOLS, its normalized subset is authoritative for both started sidecars and the in-container plan. |
evaluation_tools.policy |
advisory or required |
advisory |
advisory always permits performance but records an unsatisfied policy. required permits performance only when every applicable selected tool is ready, completes, and reports clean. |
evaluation_tools.positive_control |
required, optional, disabled, or boolean |
required |
Requires the applicable synthetic known-bug startup control to pass before runtime capability is ready. optional, disabled, and false normalize to not required; the worker still runs and reports its control. |
evaluation_tools.timeout_s |
exact integer from 1 through 3600 | 3600 |
Default maximum execution time for each selected tool. Booleans, floats, and numeric strings are rejected. |
evaluation_tools.runtime_profile |
string or null |
null |
Fallback runtime-identity assertion for plans whose tool has no runtime_ref. It does not select an image and must exactly match worker health when set. |
evaluation_tools.tools |
mapping | {} |
Per-tool configuration keyed by normalized tool ID. Entries do not enable tools. |
evaluation_tools.tools.<id>.runtime_ref |
string or null |
automatic host image ID | Exact bare local Docker image ID (sha256:...) asserted by the plan and compared with worker health. The image_digest key is an alias. This field does not select an image; omit it when using automatic host injection. |
evaluation_tools.tools.<id>.timeout_s |
exact integer from 1 through 3600 | top-level timeout | Per-tool timeout; it cannot exceed the top-level timeout. |
evaluation_tools.tools.<id>.options |
mapping | {} |
Adapter options, including argv lists and candidate-evidence paths. Reserved framework keys are rejected at run and task level: positive_control_required; GPU ASan runtime/preload/library keys; rocJITsu binary/config keys; Waitcheck CLI/C API keys; ConSan hook keys; and HIP-FpSan include/header keys. |
The exact reserved option keys are positive_control_required for every tool;
asan_runtime_dir, hip_asan_runtime, host_asan_preload,
host_asan_lib_dir, and normal_rocm_lib_dir for gpu_asan;
rocjitsu_binary and config_path for rocjitsu; waitcheck_binary and
waitcheck_capi_wrapper for rocjitsu_waitcheck; consan_hook for
rocjitsu_consan; and include_dir and public_header for hip_fpsan. They
are selected and attested by worker health, not YAML.
An explicit evaluation_tools mapping and each per-tool configuration reject
unknown fields. Unknown enabled tool IDs are also rejected. If both
runtime_ref and image_digest are supplied, they must be identical.
Example run-level section:
evaluation_tools:
enabled:
- gpu_asan
policy: advisory
positive_control: required
timeout_s: 600
tools:
gpu_asan:
options: {}The image is selected separately with a host override. The runner resolves and attests its immutable local image ID automatically:
export AKA_EVAL_TOOL_IMAGE_GPU_ASAN='registry.example/eval-tool-gpu-asan@sha256:<digest>'The in-container main.py entrypoint accepts these flags:
| Flag | Description |
|---|---|
--config_name <file> |
Config file to load (default example_configs/quickstart_claude_mi300.yaml for the generic MI300-series target). Pass a matching config explicitly on another GPU |
--run-suffix <suffix> |
Suffix appended to the run directory name (letters, numbers, ., _, - only). Useful for labeling A/B runs |
--resume-run <run_dir> |
Resume a specific run directory, skipping completed tasks |
--resume-latest |
Resume the most recent run in the workspace |
These flags are passed to the in-container entrypoint through make docker-run
or make docker-parallel-run (CONFIG= sets --config_name; RUN_ARGS=
forwards the rest):
make docker-run CONFIG=config_triton.yaml RUN_ARGS="--run-suffix with_mcp"
make docker-parallel-run CONFIG=config_triton.yaml GPU_IDS=0,1 RUN_ARGS="--run-suffix with_mcp_parallel"The following flags are internal implementation details used by
docker-parallel-run and should not be passed manually in normal use:
| Flag | Description |
|---|---|
--run-name <run_dir> |
Explicit run directory shared by parallel init, workers, and post-processing |
--parallel-init |
Initialize the shared .parallel/ queue |
--parallel-worker |
Claim and execute tasks from the shared queue |
--worker-id <id> |
Worker identifier used in queue descriptors and logs |
--postprocess-only |
Aggregate results once after all workers finish |
The following Make targets are available for running experiments.
| Target | Description |
|---|---|
make docker-run CONFIG=example_configs/quickstart_claude_mi300.yaml |
Run tasks serially in one Docker container |
make docker-parallel-run CONFIG=example_configs/benchmark_cursor_mi355x.yaml GPU_IDS=0,1 |
Run one Docker worker per listed GPU, using a shared dynamic task queue |
make docker-smoke |
Verify Docker, ROCm runtime visibility, Python imports, and GPU access |
make docker-check-agents CONFIG=example_configs/quickstart_claude_mi300.yaml |
Verify the first-class host CLI selected by the config inside Docker (task_validator resolves to its backend; GEAK and its v2 aliases resolve to Claude Code). Override with AGENTS=claude_code,codex; use AGENTS=all for all three. Engine/SDK checks are separate from this CLI check |
make docker-shell |
Open an interactive shell in the experiment runtime |
docker-parallel-run accepts these environment variables:
| Variable | Description |
|---|---|
GPU_IDS |
Comma- or space-separated host GPU IDs. If omitted, the runner uses rocm-smi --showid |
RUN_ARGS |
Additional main.py flags, such as --run-suffix, --resume-run, or --resume-latest |
AKA_LOGICAL_GPU |
Logical GPU index inside a masked worker container. Defaults to 0 and normally should not be changed |
Each parallel worker sets ROCR_VISIBLE_DEVICES to the host GPU ID and sets
HIP_VISIBLE_DEVICES, CUDA_VISIBLE_DEVICES, and GPU_DEVICE_ORDINAL to the
logical GPU index inside the masked container. See
Run tasks in parallel across multiple GPUs for the
full scheduling model.
Task definition, schema, and authoring is the canonical reference for task configuration. It replaces the former task-family-specific field tables on this page and covers:
- The selected unified v2 schema and compact examples.
- Command and result protocols, baseline/reference/candidate roles, and lifecycle.
- Workspace paths, edit boundaries, agent independence, and task-local checks.
- Optional sanitizer commands, common exports, and how to add or modify a task.
- Legacy field mappings and runtime migration requirements.
Implementation status: all retained task configurations and runners use v2 through the shared loader, evaluator, and validator. GPU qualification remains separate from schema migration. Run and final-result schemas remain documented on this page; the task guide describes executable task contracts.
The evaluator infers a profile from candidate.language, declared candidate
paths, and source ownership. Add evaluation_profile only when that
inference is insufficient. Recognized profile overrides, including
submission_paths, are recorded in
resolved_task_profile.explicit_overrides. Unknown profile fields are currently
ignored, so use only the documented keys.
| Field | Type | Description |
|---|---|---|
evaluation_profile.language |
string | Canonical values are triton, hip, flydsl, and unknown. |
evaluation_profile.artifact_kind |
string | source_aot, python_jit, hsaco_precompiled, or unknown. |
evaluation_profile.framework |
string | Framework identity such as standalone, aiter, rocblas, or rccl. |
evaluation_profile.instrumentation_control |
string | compiler_controlled, recompile, none, or unknown. This describes whether the selected candidate can be rebuilt/instrumented. |
evaluation_profile.adapter |
string or null |
Explicit adapter identity, for example triton_aot, flydsl_aot, or hip_fpsan_manual. It is a claim that must still be supported by adapter options/evidence. |
evaluation_profile.source_available |
boolean | Whether source for the selected candidate is available to the evaluator. |
evaluation_profile.submission_paths |
string or list of strings | Additional workspace-relative evidence files. V2 always includes candidate declarations and protected task inputs; this field extends that set and cannot hide candidate edits. Absolute paths and .. are rejected. |
evaluation_profile.fpsan_ported |
boolean | Explicit evidence that the HIP reference and candidate were manually ported to HIP-FpSan value semantics. |
evaluation_profile.rebuilt_from_source |
boolean | Explicit evidence used when a framework/library path is rebuilt from controlled source. It does not replace artifact attestation. |
A task can register adapters for known tools, including tools the current run does not enable. Disabled adapters are validated and remain dormant. The only allowed task-level structure is:
evaluation_tools:
tools:
gpu_asan:
timeout_s: 300
options:
command: [python3, scripts/eval_tools/run_gpu_asan.py]For an enabled tool, timeout_s must be between 1 and the run-level value. Task configuration cannot
enable another tool, change the top-level policy or positive_control, select
another runtime image, increase a timeout, or set any reserved framework option
listed above. Other options are merged over the run-level options.
Commands are argv lists, not shell strings. The built-in adapter keys are:
| Tool | Adapter keys |
|---|---|
triton_fpsan |
comparison_command or command; optional invocation-artifact-contained attestation_path. |
gpu_asan |
command; optional invocation-artifact-contained candidate attestation_path. ASan runtime/preload/library paths come only from verified sidecar health. |
rocjitsu |
HIP uses launcher or command, with optional expected_kernel and invocation-artifact-contained race_report whose filename must be race.log. Triton/FlyDSL requires capsule and the exact triton_aot/flydsl_aot profile adapter; arbitrary launchers and task-configured race-report paths are rejected. The capsule must be workspace-contained, single-dispatch, contain a golden expected output, be manifest-valid, and target gfx950. Binary/config and the trusted replay helper come only from the image/health. Automatic trusted capsule capture from correctness is not implemented. |
rocjitsu_waitcheck |
code_object, expected_kernel, and non-negative integer kernel_entry. The workspace-contained unbundled final ELF is SHA-256-bound to the plan; an image-owned inventory must match its exact gfx950 descriptor before structured C API analysis. |
rocjitsu_consan |
code_object, focused native command, and independent oracle_command. The instrumented command must explicitly name and load the code object. Strict record/replay, SHA-256/FNV identity, complete accounting, and an oracle pass are required. Broad AITER/rocBLAS/RCCL runtimes are rejected. |
hip_fpsan |
comparison_command or command; optional invocation-artifact-contained candidate attestation_path; requires evaluation_profile.fpsan_ported: true. The include path comes only from verified sidecar health. |
Sidecar health attests and injects runtime-internal assets. Candidate/task configuration cannot override or supply ASan preload/library paths, the rocJITsu binary/config path, the Waitcheck CLI/C API wrapper, the ConSan hook, or the HIP-FpSan include path.
Build-attestation JSON must store artifact_path relative to the directory
containing that JSON. The artifact must be beside or below the attestation;
absolute paths, .., and symlink resolutions that escape the directory are
rejected. The parser resolves the relative path in the scoring namespace and
verifies the declared SHA-256.
Each execution uses a fresh
<tool>/<plan-fingerprint>/<invocation-id> artifact directory. Custom
attestation and race-report paths are resolved against that directory; an
absolute path is accepted only when its resolution remains inside it.
platform_support.status: skip excludes a task unconditionally. An active task
with platform_support.required_arch is run only when that value matches the
detected GPU architecture. If platform_support is omitted, the task remains
runnable on every architecture.
Each task produces a task_result.yaml in its workspace:
| Field | Description |
|---|---|
task_name |
Full task-directory path relative to tasks/, including any suite/difficulty levels |
pass_compilation |
Whether the optimized kernel compiled |
compilation_error_message |
Error text if compilation failed, else null |
pass_correctness |
Whether correctness passed |
correctness_error_message |
Error text if correctness failed, else null |
pass_tool_gate |
Whether the selected evaluation-tool policy allows performance to proceed. Defaults to true when tools are disabled. This is independent of pass_correctness. |
tool_policy_satisfied |
Whether every applicable selected tool was ready and completed with a clean finding status. Under advisory, this can be false while pass_tool_gate remains true. |
tool_evaluation |
Versioned complete plan, plan fingerprint, profile, capability, execution, finding, evidence, and decision data. Omitted when no tool is enabled. |
base_execution_time |
Baseline runtime in ms |
best_optimized_execution_time |
Best optimized runtime in ms |
speedup_ratio |
Speedup over baseline |
baseline_benchmark_methods |
Timing methods observed while measuring the baseline |
optimized_benchmark_methods |
Timing methods observed while measuring the optimized kernel |
benchmark_method_consistent |
Whether baseline and optimized timing methods matched |
workload_consistent |
Whether every baseline/optimized case paired and reported the same shape, parameters, and dtype |
workload_mismatches |
Per-case workload differences; mismatches disable performance scoring |
valid_baseline_cases |
Number of baseline test cases with usable timing results |
valid_optimized_cases |
Number of optimized test cases with usable timing results |
speedup_calculation_error_message |
Error text if speedup could not be calculated, else null |
optimization_summary |
Framework-generated note identifying the optimizing agent and centralized evaluator |
score |
Computed score (see below) |
tool_evaluation uses this high-level shape:
| Field | Description |
|---|---|
schema_version |
Evaluation-tool result schema version. |
plan_fingerprint |
SHA-256 over normalized configuration, resolved task profile, plugin versions, captured original/candidate evidence for declared paths, verified scoring-image reference/ID, and the content digest of a configured replay capsule. |
plan |
Complete immutable plan: schema, policy, profile, ordered tool records (runtime reference, plugin version, timeout, and options), fingerprint, and source evidence. |
policy |
advisory or required. |
overall_status |
clean, finding, incomplete, or not_applicable. |
resolved_task_profile |
Inferred profile plus auditable explicit overrides. |
source_evidence |
Captured-original and candidate fingerprints plus manifest metadata, including metadata.scoring_runtime.image_id and .reference when tools run. |
decision |
allowed, policy_satisfied, and machine-readable reason strings. |
tools.<id>.capability |
Separate engine, adapter, runtime, and resolved effective checks. |
tools.<id>.result.execution |
not_run, completed, tool_error, or timeout. |
tools.<id>.result.finding |
not_evaluated, clean, found, or inconclusive. |
tools.<id>.result.findings |
Structured finding records. |
tools.<id>.result.artifacts |
Paths to retained reports/attestations. Raw stdout/stderr is omitted from the YAML summary by default. |
Execution status and finding status are deliberately independent. A sanitizer can terminate the candidate while producing a valid finding, and a process can exit zero without proving that an instrumented kernel ran.
The complete plan.tools records make the selected runtime, plugin version,
timeout, and options reconstructable from the report. Ordinary option path
strings still do not hash referenced adapter, HSACO, or input contents; include
those files in evaluation_profile.submission_paths or add explicit digests. A
configured rocJITsu capsule is handled specially: its JSON SHA-256/size are
added to source evidence and the fingerprint, while capsule validation verifies
the manifest's HSACO and blob hashes. This does not prove the capsule came from
the ordinary correctness run.
The existing score field uses a bounded power-family reward. The default is
the linear curve (p = 1) with a 1.05× speedup cutoff:
s = speedup_ratio
score = max(0, 1 - speedup_threshold / s) ** p
The formula applies only when compilation and correctness pass and s is a
finite positive ratio with benchmark_method_consistent: true. Failures,
missing/invalid ratios, or missing/mismatched timing-method evidence score 0.
Aggregate timing fields are not a fallback for missing speedup evidence.
The evaluator also rejects incomplete case coverage or mismatched workloads.
For a single case, s = baseline_time / candidate_time. Equivalently, let
u = candidate_time / baseline_time and c = 1 / speedup_threshold; then
score = max(0, 1 - u/c) ** p. At the default cutoff, the candidate must take
less than baseline_time / 1.05 (about 95.24% of the baseline time) to score
above zero. Exactly 1.05× still scores 0.
For multi-case tasks, s is the evaluator's arithmetic mean of matched per-case
speedups. Apply the formula once to that mean; do not average per-case
rewards or divide the two aggregate timing fields. This retains the existing
aggregation contract, so a positive task score does not guarantee every case
improved.
Edit these two constants in src/score.py before running or aggregating an
experiment. There is one scoring function, with no additional reward field or
run-configuration section:
| Constant | Default | Effect |
|---|---|---|
SCORE_POWER |
1.0 |
Exponent p: 1.0 linear, 0.5 concave, 2.0 convex |
SCORE_SPEEDUP_THRESHOLD |
1.05 |
Speedup at or below which the score is zero |
Both constants must be finite and positive. With the default p = 1, the score
is the fraction of runtime saved beyond the threshold, and its slope
d score / d s = speedup_threshold / s**2 stays finite at the cutoff. Any
p < 1 has an unbounded slope as s approaches the cutoff from above, so
scores of candidates just above 1.05× swing with timing noise. p = 0.5
raises scores for modest qualifying improvements; p = 2.0 emphasizes large
improvements. All choices preserve the ranking above the cutoff. Changing p does not provide a signal
below the cutoff. Setting the threshold to 1.0 removes the 5% margin but still
gives zero to baseline-equivalent and slower candidates.
Example: For a correct candidate taking 80 microseconds against a
100-microsecond baseline, s = 1.25. The default score is
1 - 1.05 / 1.25 = 0.16. The same measurement scores 0.4 with p = 0.5
(sqrt(0.16)) and 0.0256 with p = 2.
Scores are in [0, 1], with faster valid candidates approaching 1. Compilation
and correctness are prerequisites, with no separate point bonuses. A correct
but slower candidate can score 0; use the explicit correctness and acceptance
fields for those outcomes.
This replaces the previous 20 + 100 + 100 * speedup_ratio scoring scale.
Stored historical scores and runs using different constants are not directly
comparable. Existing reports are not migrated automatically. Running
post-processing again recalculates each task_result.yaml score using the
current constants, so use copies when rescoring historical runs and retain the
code revision and parameter values with the experiment.
For an A/B pair, compare completed run directories with:
python3 src/tools/compare_runs.py <baseline-run-directory> <treatment-run-directory>Agents register themselves into a shared registry with the register_agent
decorator, and the framework loads only the selected agent:
from agents import register_agent
@register_agent("your_agent")
def launch_agent(eval_config, task_config_dir, workspace):
...
return resultThe selectable agent names are defined by the AgentType enum in
src/module_registration.py. See Configure agents and models
for the integration steps.