Skip to content

full_reset() gives no signal whether the environment was reset; skipped reset is indistinguishable from a successful one #406

Closed
#411
Closed
full_reset() gives no signal whether the environment was reset; skipped reset is indistinguishable from a successful one#406
#411

Description

@nikolas-sapa
nikolas-sapa
opened on Oct 4, 2026

Summary

WebArenaInstance.full_reset() gives the caller no way to tell whether a reset happened.

prepare_backend() calls it with no arguments, so skip_if_not_set=True is always in force. With no
reset endpoint configured, the reset is skipped with a logged warning — but the function then falls
through to the warm-start block and returns None, which is exactly what it returns after a
successful reset. So the benchmark proceeds against an environment that was never reset, and the
caller has no signal to distinguish the two cases. The only evidence is one WARNING line in a long
benchmark log.

This matters because the upstream WebArena documentation treats the reset as required for correct
evaluation:

After evaluating the 812 examples, reset the environment to the initial state
— web-arena-x/webarena, README

All 812 shipped task configs also carry require_reset: false, which is consistent with the intended
protocol being one reset per sweep rather than per task. So the single reset that BrowserGym performs
is the one carrying the reproducibility guarantee — and it is the one that can silently not happen.

Where

browsergym/webarena/src/browsergym/webarena/instance.py:59

def full_reset(self, skip_if_not_set: bool = True):
    base_url = os.environ.get(self.RESET_URL_VAR, None)
    if not base_url:
        logger.error(f"Environment variable {self.RESET_URL_VAR} is missing or empty, required for a full instance reset.")
        if skip_if_not_set:
            logger.warning("Skipping automated reset. Make sure the instance has been manually reset.")
        else:
            raise RuntimeError(f"Could not reset instance, aborting.")
    else:
        ...  # performs the reset, polls /status until "Ready for duty!"

    # warm-start the instance (navigate to every domain)   <-- reached in BOTH branches
    retries_left = 3
    while retries_left:
        ...
        self._check_is_reachable(timeout=60)

Note the fall-through: the warm-start block sits outside the if/else, so the skipped branch does not
return early — and neither branch returns a value that identifies itself.

Called from browsergym/experiments/src/browsergym/experiments/benchmark/utils.py in
prepare_backend(), with no arguments, for all three backends:

line backend reset env var
147 webarena WA_FULL_RESET
176 webarena_verified WA_FULL_RESET
205 visualwebarena VWA_FULL_RESET

(VisualWebArenaInstance subclasses WebArenaInstance and overrides RESET_URL_VAR, so it inherits
the same behaviour.)

Reproduction

No WebArena instance needed — this is the control flow, not the network:

git clone https://github.com/ServiceNow/BrowserGym && cd BrowserGym   # verified at 0.14.4, c05d7f3

Every call site uses the silent default:

grep -n "full_reset()" browsergym/experiments/src/browsergym/experiments/benchmark/utils.py
# 147:            default_instance.full_reset()
# 176:            default_instance.full_reset()
# 205:            default_instance.full_reset()

And the branch behaviour, run directly (no live instance needed):

import importlib.util, logging, os
logging.basicConfig(level=logging.INFO)
spec = importlib.util.spec_from_file_location(
    "wa_instance", "browsergym/webarena/src/browsergym/webarena/instance.py")
mod = importlib.util.module_from_spec(spec); spec.loader.exec_module(mod)

os.environ.pop("WA_FULL_RESET", None)
inst = mod.WebArenaInstance.__new__(mod.WebArenaInstance)  # bypass __init__, needs live env vars
inst.urls = {"probe": "https://example.com"}               # stand in for the six sites

print("silent branch returns:", inst.full_reset())         # -> None
inst.full_reset(skip_if_not_set=False)                     # -> RuntimeError, by contrast

Observed on 0.14.4:

ERROR    Environment variable WA_FULL_RESET is missing or empty, required for a full instance reset.
WARNING  Skipping automated reset. Make sure the instance has been manually reset.
silent branch returns: None
fail-closed path raises: Could not reset instance, aborting.

A successful reset also returns None — that is the point. The return value carries no information
about whether the environment was reset.

Why I think it is worth changing

The failure is invisible at the point where it matters. A user who has not set WA_FULL_RESET (or
VWA_FULL_RESET) gets a clean run, a normal-looking return value, and results that depend on whatever
state their instance happened to be in. Because the only evidence is one WARNING in a long benchmark
log, the most likely outcome is that nobody notices the reset was skipped — and there is currently no
way for a downstream script to check.

I am not claiming any published WebArena number is wrong. I have not run the benchmark, and the
reset is available to anyone who configures it — this is about the default being silent rather than
about the mechanism being absent.

Suggested fix (policy call, so I'd rather discuss than impose)

Failing closed — full_reset(skip_if_not_set=False) at the three call sites — would break every
current user who has no reset endpoint, so I don't think it should be done unilaterally. Two smaller
options:

  1. Fail legibly. Have full_reset() return a bool (or a small enum: reset / skipped), and
    have prepare_backend() surface "environment reset: SKIPPED (WA_FULL_RESET unset)" in the run
    header or run metadata, so a run is self-documenting and downstream tooling can assert on it.
  2. Fail closed behind an opt-in. Keep the current default, but add an explicit
    WA_REQUIRE_RESET=1 that turns the silent skip into a hard error, for anyone who wants the
    guarantee enforced.

Either way the current behaviour is preserved by default, so nothing breaks.

Happy to open a PR for whichever you prefer — including just the docs clarification that
WA_FULL_RESET / VWA_FULL_RESET are required for a reproducible run.

Activity

  1. Tharun-Varshan-S commented on Oct 8, 2026

    @Tharun-Varshan-S
    Tharun-Varshan-S
    on Oct 8, 2026
    Contributor

    Hi @Surya-5555 , I investigated and implemented the “fail legibly” approach you suggested for Issue #406.

    Problem

    WebArenaInstance.full_reset() previously returned None in both cases:

    1. The environment was successfully reset.
    2. The reset was skipped because WA_FULL_RESET / VWA_FULL_RESET was not configured.

    Because prepare_backend() called full_reset() with the default skip_if_not_set=True, a missing reset endpoint only produced a warning. The benchmark then continued normally, and the caller had no programmatic way to determine whether the environment had actually been reset.

    This made a skipped reset indistinguishable from a successful reset and could affect reproducibility.

    Solution

    I chose the backward-compatible “fail legibly” approach rather than changing the default behavior to fail closed.

    I modified full_reset() so that it returns an explicit status:

    • True → the reset was successfully performed.
    • False → the reset was intentionally skipped because the reset endpoint was not configured.
    • Existing exceptions remain exceptions when a configured reset attempt fails.

    I then updated the three prepare_backend() call sites for:

    • webarena
    • webarena_verified
    • visualwebarena

    to consume this status and surface it in the run logs.

    The run now clearly reports either:

    Environment reset: DONE

    or:

    Environment reset: SKIPPED (WA_FULL_RESET unset)

    For VisualWebArena, the correct VWA_FULL_RESET configuration is respected.

    How it was implemented

    The important change is that the reset operation no longer relies on None to represent both outcomes.

    Conceptually:

    full_reset()
    │
    ├── reset endpoint configured
    │       ├── reset succeeds → True
    │       └── reset fails    → exception
    │
    └── reset endpoint missing
            └── skip allowed → False

    prepare_backend() then converts that result into an explicit run-status message.

    This preserves the existing default behavior for users who do not have an automated reset endpoint, while making the reset state visible to users and downstream tooling.

    Edge cases checked

    I also considered:

    • Missing reset environment variable.
    • Empty reset environment variable.
    • skip_if_not_set=True.
    • skip_if_not_set=False.
    • Successful reset.
    • Reset endpoint failure.
    • Warm-start failures.
    • VisualWebArenaInstance inheriting full_reset() while overriding RESET_URL_VAR.
    • All three prepare_backend() call sites.
    • Existing callers of full_reset().

    I kept the patch limited to the reset-status handling and did not change benchmark task/scoring behavior.

    The remaining validation is to run the relevant test suite/CI against the repository's Playwright/WebArena environment.
    Can you provide me oppurtunity to work on this issue

  2. Surya-5555 commented on Oct 9, 2026

    @Surya-5555
    Surya-5555
    on Oct 9, 2026
    Collaborator

    Thank you both for your incredible work on this!

    @nikolas-sapa - Your analysis of the fall-through bug is spot on. I agree that the "fail legibly" approach is the right call here to preserve backward compatibility while surfacing the critical state.

    @Tharun-Varshan-S - Thank you for jumping in and implementing this locally! It sounds like you have the exact right approach mapped out.

    Since @nikolas-sapa originally opened the issue and offered to write the PR, I want to give them the first right of refusal. Nikolas, are you still planning to open a PR for this, or are you happy to let Tharun submit their implementation?

    Either way, please go ahead with the "fail legibly" approach returning True/False. Looking forward to reviewing the PR!

  3. Tharun-Varshan-S commented on Oct 21, 2026

    @Tharun-Varshan-S
    Tharun-Varshan-S
    on Oct 21, 2026
    Contributor

    Opened PR #411 implementing the "fail legibly" approach discussed above.

    Summary of changes in #411:

    • full_reset() now returns bool (True when the reset was executed, False when skipped because the reset environment variable is unset/empty).
    • prepare_backend() in utils.py now surfaces the reset state explicitly in the run logs for webarena, webarena_verified, and visualwebarena (Environment reset: DONE or Environment reset: SKIPPED (<VAR> unset)).
    • Added unit tests in tests/webarena/test_instance_reset.py covering all reset states, error conditions, VisualWebArena inheritance, and prepare_backend log outputs.
  4. Surya-5555
    added a commit that references this issue on Oct 28, 2026

    fix: return explicit status from full_reset to prevent silent skips (f…

    0e6830f
  5. Surya-5555
    closed this as completedin #411on Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions