Human in the loop

[IdirLISN]

@ mention of reviewers

@acletournel
@ObadaS
@wlln
@Didayolo

Upgrade instructions

Django migration:

docker compose exec django ./manage.py migrate

Upgrade compute workers:

https://docs.codabench.org/dev/Organizers/Running_a_benchmark/Compute-Worker-Management---Setup/

Description

Human in the loop feature (HITL), enables organizers to add this option for their competitions

Setting needed on worker side:

HUMAN_IN_THE_LOOP=True

The HITL implementation follows four principles:

1. Backward compatibility
   • Existing deployments require no configuration changes.
2. Safety
   • Configuration mismatches are detected before execution.
3. Isolation
   • HITL is supported only on private Compute Workers.
4. Atomic publication
   • No artifact is published before manual approval.

These principles ensure that sensitive competitions can safely introduce manual validation without impacting the standard Codabench execution workflow.

If this option is enabled, each submission will require a validation on compute worker side.

If no validation occurs, a time out will return a submission failed status
If the compute worker administrator validates the scoring file, the submission returns a scoring to codabench.

Issues this PR resolves

It enables the check of the scoring files and detailed results from compute worker side before sending them to codabench.

A checklist for hand testing

• Create a competition and click on HITL button in competition editor page, see screenshot.
• Submit and check compute worker logs.
• Check the scoring file, path displayed in compute worker logs.
• Choose to validate or not.
• Check Codabench interface and make sure the score is returned or not in regards of the choice made in the last step.

Checklist

• Code review by me
• Hand tested by me
• I'm proud of my work
• Code review by reviewer
• Hand tested by reviewer
• CircleCi tests are passing
• Ready to merge

[Didayolo]

@IdirLISN I find this feature a bit confusing.

Naming and placement

First of all, as shown on your screenshot, we have an option "Auto-run submission". When it is disabled, organizers need to validate submissions before sending them:

I understand that this new feature is a variation of that where the validation happens at the compute worker level. However it is confusing to have a completely different naming and checkbox for a variation of the same feature.

We could, for instance, have an additionnal option when disabling "auto-run submissions" : "Validate submissions from website" VS "Validate submissions from compute worker", or something like this.

Also, the naming "Human in the loop" has different definitions among the community, but is usually used to refer to setup where the evaluation is done by human (not just the pre-run validation).

How does it work?

Question: concretely, how does the organizers receive and validate the submissions? Is it directly inside the compute worker through command line? How does it work if there are 10 workers in the queue? Did you document this?

[IdirLISN]

@Didayolo

This validation procedure is made to secure datasets when they are in compute worker side.
When the dataset provider doesn't want to give access to data and mount them inside of compute worker volume, we need to make sure the scoring file doesn't return anything which is not metrics and enable the dataset owner to verify the scoring content before sending it to the plateform.

The organizer just activate the option in the edit section of the competition and then there is someone from CW side who's going to validate the scoring before sending it.

As i progress through the feature i will elaborate the PR to make it clear.
About the name and the feature design, we should talk about it because i'm not the only one involved.

Thank you for your review :)

[Didayolo]

Review

Critical issues

• HITL can be bypassed by the programs it's meant to gate. The sentinel files are polled inside self.output_dir, which is mounted read-write into the scoring container. A scoring (or ingestion-during-scoring) program can simply create hitl_approved before exiting — polling then instantly self-approves the submission. Sentinels must live in a directory not mounted into any container.
• run_wrapper now sets FINISHED unconditionally after push_output() — including for the prediction run (is_scoring=False), where status was previously left at SCORING. A submission will be marked "Finished" between prediction and scoring. It's also redundant for the non-HITL scoring path, which already sets FINISHED inside start().
• tasks.py: apparent rebase leftover. The second block starting if (submission.phase.competition.queue): re-sets submission.queue, overrides run_args["execution_time_limit"], and saves — duplicating/conflicting with the effective_queue logic just above it (and potentially clobbering a participant-group queue). Looks like old code accidentally reintroduced; it should be removed.
• HITL rejection may fire on validly configured competitions. The submission.queue is None checks in _send_to_compute_worker and _run_submission run before effective_queue = submission.queue or submission.phase.competition.queue is applied. A competition with a private queue configured at competition level but whose submission hasn't had queue copied yet would be wrongly failed. Check the effective queue instead.
• Worker blocked for up to 24h. wait_for_human_validation() blocks the Celery task synchronously (3s poll, 24h max). With default concurrency the worker processes nothing else while waiting. The docs even say "The compute worker is enable to run another submission while waiting" — which is both a typo and, as written, not what the code does. Either document the blocking behavior clearly or rework (e.g., requeue/resume).

Bugs / risks

• HTTP server reachability: it binds 127.0.0.1:8765 inside the worker container. The documented ssh -L 8765:127.0.0.1:8765 operator@<compute-worker> tunnels to the host's loopback, which won't reach the container unless it runs with network_mode: host. Verify against the actual compose setup.
• Hardcoded port 8765, no allow_reuse_address: a crashed run or two concurrent HITL submissions on one host → Address already in use → the submission crashes. Use port 0 (log the chosen port) or make it configurable, and set allow_reuse_address = True.
• send_detailed_results finally block references websocket: if it isn't initialized to None before the try, a failure before assignment raises NameError inside finally. Verify initialization.
• Fragile frontend detection: is_hitl_failure() matches status_details.indexOf('Human in the Loop'), but the message in _send_to_compute_worker says "Human-in-the-Loop (HITL)" — that failure won't match. Use a consistent marker or a dedicated status/flag rather than string-matching a human message.
• Sentinel files pollute results: hitl_approved sits in output_dir and will be included in the pushed output archive.
• validate_hitl_configuration() runs after prepare() — image pulled and data downloaded before rejecting. Cheap to move before prepare().
• Duplicated checks: the HITL/public-worker rejection exists in both _send_to_compute_worker and _run_submission with different messages. Keep one.
• Timeout message: missing space — "...without validation(submission 42)".
• start() unpacking: the nested if self.human_in_the_loop branches for task_results unpacking are brittle; a comment above still says "2 or 3 gathered tasks" which is now wrong. Consider tracking task names/indices instead of positional unpacking.

Style / conventions

• Mixed %-style and f-string logging in the same block; pick one (repo mostly uses f-strings).
• AWAITING_VALIDATION is ordered before SCORING in the worker's list and after it in the Django model — cosmetic, but confusing.
• Unrelated noise: removed # 20 minutes comment, added blank lines in watch_detailed_results, doc whitespace tweak in the setup page.
• Docs have typos ("exemple", "reviwed" in commit, "the check the scoring file", "is enable to run") and an incomplete sentence ("Each competition exposes the following option:" with nothing following before the image).

Test coverage

None added. At minimum: unit tests for the queue-gating logic in tasks.py (HITL + no queue → Failed; HITL + private queue → human_in_the_loop=True in run_args), serializer round-trip of the new field, and worker-side tests for validate_hitl_configuration and approve/reject/timeout paths (sentinel polling is easy to test with tmpdirs and a short timeout).

[Didayolo]

On worker3 in Partages instance:

2026-08-07 14:02:38.818 | ERROR    | celery.app.trace:_log_error:285 - Task compute_worker_run[bbdcf3a8-b9e9-4f7b-8a94-c0feab276157] raised unexpected: FileExistsError(17, 'File exists')
Traceback (most recent call last):

> File "/.venv/lib/python3.13/site-packages/celery/app/trace.py", line 479, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/.venv/lib/python3.13/site-packages/celery/app/trace.py", line 779, in __protected_call__
    return self.run(*args, **kwargs)

  File "//compute_worker.py", line 338, in run_wrapper
    if not run.wait_for_human_validation():

  File "//compute_worker.py", line 1594, in wait_for_human_validation
    host_detailed_results = self._get_host_path(detailed_results)

  File "//compute_worker.py", line 1116, in _get_host_path
    os.makedirs(path, exist_ok=True)

  File "<frozen os>", line 228, in makedirs

FileExistsError: [Errno 17] File exists: '/codabench/uPK-3_sID-313__fl2a45f7/output/detailed_results.html'

[Didayolo]

Human in the loop option enabled both in the instance and the worker:
The submission did run without waiting for validation.

EDIT: OK apparently this is the normal behavior