Roadmap: learn-once/compile-once execution model to reduce AI-in-the-loop cost for large regression suites

[qtpsudhakar]

I'm evaluating ARTEMIS for large-scale test automation and have a question about the intended long-term architecture.

ARTEMIS can currently execute end-to-end workflows using AI/agentic interaction. However, for large regression suites, having an AI model reason at every step can introduce latency, cost, and nondeterminism.

Is there (or is there planned) a workflow along these lines?

Learn once → generate/compile a reusable automation → execute cheaply/deterministically → invoke AI only when the learned interaction no longer matches the UI

For example:

1. ARTEMIS explores a workflow such as: Login → Search → Product → Add to Cart → Checkout
2. During exploration, it learns the semantic/UI targets and state transitions.
3. The learned workflow is persisted as a reusable test artifact.
4. Subsequent executions use the learned targets/path without invoking the LLM/VLM for every action.
5. If a target/state cannot be resolved because the application UI has changed, ARTEMIS invokes the AI/VLM to understand the new UI and repair/update the workflow.
6. The repaired workflow is validated and then reused for subsequent executions.

In other words, I'm interested in whether ARTEMIS is intended to become a learned/adaptive automation engine, rather than having the AI remain in the execution loop for every test run.

A few specific questions:

• Does ARTEMIS currently persist knowledge learned during a successful workflow for later deterministic/reduced-AI execution?
• Is there a concept of compiling/serializing an agentic workflow into a fast, reusable test?
• Can AI/VLM invocation be triggered only when deterministic/local grounding fails?
• Is self-healing/re-learning of previously discovered workflows part of the roadmap?
• Is the planned lightweight Edge VLM intended partly to support this low-latency execution/recovery model?
• If this isn't currently supported, is this an architectural direction the team is considering?

I'm particularly interested in how ARTEMIS is expected to scale to thousands of regression tests where AI-per-step latency would become significant.

[somew1nd]

Thank you for raising this question. We agree with your concerns about latency, cost, and nondeterminism in large regression suites. However, we do not think addressing them requires turning ARTEMIS into an engine that learns workflows, compiles them, and automatically repairs them when they fail.

Successfully completing a workflow once is not the same as producing a reliable regression test. A test also needs explicit preconditions, test data, synchronization, and assertions. When a UI changes, finding another path to the goal does not necessarily mean the test has been repaired. The original path failing may be precisely the regression the test was supposed to catch. Automatically navigating around it could defeat the purpose of the test.

Our intended approach has two main use cases, with a separate architectural question about who controls failure recovery.

1. Use ARTEMIS to explore the real application before writing test code.
Coding agents often write end-to-end tests by inferring how an application works from its codebase and general knowledge. The resulting code can look reasonable while being difficult or impossible to run on a real device.
ARTEMIS is designed to address this gap. Before writing a test, the coding agent should use ARTEMIS to perform the workflow on an actual device, observe the screens and interactions, and verify the state transitions. It can then write test code based on what actually happened.
The workflow we recommend is:

Explore and verify with ARTEMIS → generate conventional test code with a coding agent → run regression tests through an existing test framework.

We provide MCP tools so coding agents can operate devices, monitor tasks, and inspect execution details and screenshots. Our mcp_server/rules.md explicitly requires exploration before test authoring. It also directs coding agents to prioritize dynamic locators, such as resource IDs and text, and use coordinates verified during exploration as a fallback when necessary.

The reusable artifact here is conventional test code, including locators, waits, and assertions. Subsequent regression runs do not need a model to decide each action again. ARTEMIS stores execution traces that support this process, but those traces are not automatically compiled into executable tests.

In short, writing stable test code should be the responsibility of the coding agent. Artemis is positioned as a tool that helps coding agents uncover the truth. We will continue to optimize in this direction to make Artemis easier for coding agents to use, and we will also provide prompts to help the coding agent learn how to write stable regression tests; however, we do not intend to step into the coding agent's role in the near future.

2. Embed ARTEMIS through the SDK to handle the parts of a test that are hardest to script reliably.
We do not recommend handing an entire regression test to ARTEMIS. A more practical approach is to execute stable, well-defined steps with conventional code, call ARTEMIS for a limited portion with many possible branches, and then return control to the test code.

For example, login may involve an existing session, an account chooser, or permission dialogs. Bluetooth setup may encounter a device that is unpaired, paired but disconnected, or waiting for confirmation in a system dialog. Bringing all these starting states to the same test setup can require substantial branching logic in a hardcoded script.

Through the SDK, you can give ARTEMIS a specific task and let it work from the device’s current state. You should clearly describe both the goal and the state in which it should hand control back. For example:

“Connect to the specified Bluetooth device, verify that the page shows it as connected, and stop on the device details page without performing any further actions.”

Once ARTEMIS returns, the test code checks the required state and continues with the remaining operations and assertions. This avoids handcrafting a navigation path for every possible starting state while keeping model calls limited to the part that needs them.

This is an intentional SDK call designed into the test. It does not require a script to fail first. ARTEMIS is responsible for a specific, bounded task. This is particularly useful for setup operations such as login or device connection. If those operations are themselves the subject of the test, the relevant path requirements and assertions still need to be explicitly preserved.

Separately, deciding whether ARTEMIS should intervene after a script fails is a test orchestration question.
The previous use case concerns what ARTEMIS can do. This question concerns who decides when to invoke it.

We considered and experimented with putting automatic failure intervention inside ARTEMIS. As we worked through it, however, we found ourselves taking on responsibilities that belong to the test framework: What counts as a failure? Which failures are recoverable? Should the original step be retried, or should an alternative flow run? Where should execution resume? How should recovery affect the test result?
These decisions depend on the intent of the test. A missing button might mean an unexpected setup dialog is covering the screen, or it might indicate a real product regression. The ability to find another route to the goal is not enough to determine whether recovery is appropriate.

We therefore believe the test framework—or the test code calling ARTEMIS—should own that decision. It can execute the conventional script first, detect a failure that meets its recovery criteria, and invoke ARTEMIS through the SDK with a clearly defined recovery task. It then checks the result and decides whether to continue, retry, or report a failure.

ARTEMIS should focus on completing the task it receives quickly and reliably. The test framework should decide when to call it and how to interpret the result. This separation already allows an “invoke AI only when needed” approach without requiring ARTEMIS to become another test framework.