Skip to content

Fix program_html evaluator race condition and empty string handling - #408

Open
Tharun-Varshan-S wants to merge 2 commits into
ServiceNow:mainfrom
Tharun-Varshan-S:fix-issue-407-evaluator-race
Open

Tharun-Varshan-S wants to merge 2 commits into
ServiceNow:mainfrom
Tharun-Varshan-S:fix-issue-407-evaluator-race

Conversation

@Tharun-Varshan-S

Copy link
Copy Markdown

Resolves #407

This PR implements the fixes suggested in #407:

  1. Replaces the hardcoded time.sleep(3) with page.wait_for_load_state("networkidle"), explicitly catching and tolerating TimeoutError so we evaluate pages with heavy background activity gracefully.
  2. Updates the except Exception blocks for the JS locators to assign selected_element = None (instead of ""), ensuring missing elements correctly hit the is None guard and return 0.0 directly rather than vacuously slipping through to the string scorers.
  3. Adds mocked regression tests for these edge cases.

@Surya-5555

Surya-5555 commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Hi @Tharun-Varshan-S - Thanks for the PR
It looks like the code-format check is failing. Could you please run pre-commit locally and push the formatting updates?
Looking forward to reviewing this once the checks are green

@Tharun-Varshan-S

Copy link
Copy Markdown
Author

Hi @Surya-5555,

Thanks for the review! I've run pre-commit run --all-files locally and pushed all the formatting updates. The code format checks should now pass green.

Let me know if you need anything else!

@Tharun-Varshan-S
Tharun-Varshan-S force-pushed the fix-issue-407-evaluator-race branch from f5fa31b to 11b564d Compare October 17, 2026 03:43
@Surya-5555

Copy link
Copy Markdown
Collaborator

Hi @Tharun-Varshan-S,
Thanks for cleaning up the PR! The logic and tests look great. Just one quick fix needed:

Issue: In browsergym/webarenalite/src/browsergym/webarenalite/evaluators.py at line 273, except playwright.sync_api.TimeoutError: will raise NameError: name 'playwright' is not defined because playwright isn't imported at module level.

Fix (line 18): Change to from playwright.sync_api import CDPSession, Page, TimeoutError
Fix (line 273): Change to except TimeoutError: pass

@Tharun-Varshan-S

Copy link
Copy Markdown
Author

Hi @Surya-5555,

Thanks for catching that! I have resolved the issue:

  • Imported TimeoutError from playwright.sync_api on line 18.
  • Updated the exception block to catch TimeoutError directly on line 273.
  • Verified formatting via pre-commit.

The fix has been pushed to the branch. Thanks again for the review!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HTML/program_html evaluators score a failed page-read as 0.0, indistinguishable from a wrong answer (302/812 WebArena tasks take that path)

2 participants