Support testing with multiple versions of Firefox#6294
Conversation
First write a test plan to check if this is likely a webcompat issue, and decide which platforms might be affected. Based on the results of that, run on some combination of nightly, stable and esr builds, and record the results for all of them.
|
CC @ksy36 |
| Literal["not_reproduced"] | ||
| Literal["not_reproducable"] | ||
| | Literal["non_compat"] | ||
| | Literal["unsupported"] |
There was a problem hiding this comment.
should this be unsupported_platform ?
| if isinstance(result, BugReproductionResult): | ||
| if self.initial_repro is not None: | ||
| raise ValueError("Got duplicate steps / summary") | ||
| self.initial_repro = InitialReproduction( |
There was a problem hiding this comment.
Should this only promote successful BugReproductionResults to initial_repro? Right now a failed Nightly BugReproductionResult sets self.initial_repro, even when result.reproduced is false.
And Chrome mask task runs based on checking whether
if repro_results.initial_repro is not None: , so looks like it will run regardless of the result.reproduced
| profile_path: Path, | ||
| steps: str, | ||
| ): | ||
| super().__init__(task_config, run_tracker) |
There was a problem hiding this comment.
I get an error at this step
autowebcompat-repro-agent-1 | File "/opt/venv/lib/python3.12/site-packages/hackbot_agents/autowebcompat_repro/agent.py", line 428, in __init__
autowebcompat-repro-agent-1 | super().__init__(task_config, run_tracker)
autowebcompat-repro-agent-1 | TypeError: StepsReproduction.__init__() missing 3 required positional arguments: 'firefox_path', 'profile_path', and 'steps'
So passed them into init to proceed further:
super().__init__(
task_config,
run_tracker,
firefox_path,
profile_path,
steps,
enable_privileged_context=True,
)
And added enable_privileged_context: bool = False, into StepsReproduction init arguments, as it already has devtools mcp registered, so we won't need to register them again here.
There was a problem hiding this comment.
Or ChromeMaskReproduction was meant to inherit from Task instead of StepsReproduction?
| ): | ||
| super().__init__(task_config, run_tracker) | ||
| self.steps = steps | ||
| self.add_mcp_server( |
There was a problem hiding this comment.
I've only run this on 1 bug, so not sure if having only steps is enough for Chrome mask test as it also needs to know whats is the expected/non broken result?
For example for this bug https://bugzilla.mozilla.org/show_bug.cgi?id=2052489 these are the steps:
autowebcompat-repro-agent-1 | ############################################################
autowebcompat-repro-agent-1 | # 1. Open Firefox with a clean profile.
autowebcompat-repro-agent-1 | 2. Navigate to https://drmeth.com/.
autowebcompat-repro-agent-1 | 3. Wait for the page to load and observe the page.
autowebcompat-repro-agent-1 | 4. Observe that the game content (currency counters, buy/dealer panels, EXPORT/IMPORT/STATS/SAVE buttons, social icons, and copyright text) is not scaled to fill the page; instead it is squished into a small box (~300x160px) in the center of the viewport with overlapping, unreadable elements, while the top header bar (logo, ad banner, Login) renders normally.
autowebcompat-repro-agent-1 | ############################################################
| ): | ||
| with open(self.initial_repro.screenshot_path, "rb") as f: | ||
| return base64.b64encode(f.read()).decode("ascii") | ||
| return None |
There was a problem hiding this comment.
This seems to be writing a large base64 string in the result json. Also I think publish_file might have been dropped in this branch, so the artifact image is not saved
| .system_prompt() | ||
| .format( | ||
| task_details=""" | ||
| 1. Identify the affected URL from the reproduction steps. |
There was a problem hiding this comment.
I think this prompt is not propagated to the agent as it tries to debug the extension background script:
autowebcompat-repro-agent-1 |
autowebcompat-repro-agent-1 | [agent] The extension enables masking per-hostname via `browser.storage.local` (`enabledHostnames`) and reacts to an `enabled_hostnames_changed` message. Let me read the background script to confirm how to trigger it, then enable masking via the extension's privileged context.
autowebcompat-repro-agent-1 |
autowebcompat-repro-agent-1 | --- turn 27 ---
autowebcompat-repro-agent-1 | [agent→tool] Bash({"command": "cd /tmp/cm && cat background_script.js", "description": "Read background script"})
autowebcompat-repro-agent-1 | [tool←ok] const chromeUAStringManager = new ChromeUAStringManager();
autowebcompat-repro-agent-1 | const enabledHostnames = new EnabledHostnamesList();
autowebcompat-repro-agent-1 | let contentScriptHandle = null;
2 participants
First write a test plan to check if this is likely a webcompat issue, and decide which platforms might be affected.
Based on the results of that, run on some combination of nightly, stable and esr builds, and record the results for all of them.