feat: multi Agent OS server support in `AskUiControllerClient` by mlikasam-askui · Pull Request #276 · askui/python-sdk

mlikasam-askui · 2026-05-12T09:00:49Z

Summary

AskUiControllerClient can now manage multiple Agent OS target computers
(local and/or remote) at once. One is active at a time and receives the agent's
actions; callers can switch at runtime or scope a switch to a with block.

New API:

AgentOsTargetComputer (base), LocalAgentOsTargetComputer (replaces the old
AskUiControllerServer — owns the local controller subprocess; auto-detects
the Windows AskuiCoreService and switches to its port),
RemoteAgentOsTargetComputer (already-running remote controller, no process
management).
AgentOsTargetComputerManager — enforces invariants (at most one local,
unique session GUIDs / computer_ids / remote addresses) and owns the gRPC
connection lifecycle. Targets are addressed exclusively by computer_id
(single dict lookup; no secondary indices); the connection dict is keyed by
computer_id too.
On AskUiControllerClient and exposed through AgentOs /
ComputerAgentOsFacade: add_agent_os_target_computer,
add_remote_agent_os_target_computer, list_agent_os_target_computers,
get_current_computer_target_id (returns the computer_id string of the
active target), switch_agent_os_target_computer,
reset_agent_os_target_computers, temporary_select.
AndroidAgentOs / PpadbAgentOs get a sibling temporary_select(device_sn).
ComputerAgent auto-registers three new act() tools so the LLM can drive
multi-target flows: ComputerListAgentOsTargetComputersTool,
ComputerSwitchAgentOsTargetComputerTool,
ComputerGetCurrentComputerTargetIdTool.
LocalAgentOsTargetComputer and RemoteAgentOsTargetComputer are re-exported
from the top-level askui package.

New unit tests cover the target-computer classes, the manager, the multi-target
client, and the new computer tools. The e2e test was updated to the new
constructor.

Replace single-controller config with a TargetComputer / TargetComputerManager abstraction. AskUiControllerClient now manages a list of local and remote controller servers, opens a gRPC connection per target on connect(), and routes agent-os actions to a single active target. Add tools for the agent to list, switch, and inspect the active target computer.

… target session GUID

This reverts commit 7bd98be.

philipph-askui · 2026-05-14T08:44:45Z

 from .set_active_display_tool import ComputerSetActiveDisplayTool
+from .switch_agent_os_server_tool import ComputerSwitchAgentOsServerTool


shouldn't these 3 new tools also be in experimental?

I see them as part of the default tool list, similar to the Android agent.

philipph-askui · 2026-05-14T08:47:29Z

        cursor_position = self.agent_os.get_mouse_position()
-        return f"Mouse is at position ({cursor_position.x}, {cursor_position.y})."
+        return (
+            f"[Server with id '{server.computer_id}']: Mouse is at position "


I am not happy with the name "Server in general". So far, I was using the term "machine" when telling the agent that it is operating multiple devices.
The name of one machine could then be server. If we use the term server here, this will introduce ambiguity that might cause the currently working stuff to break

philipph-askui · 2026-05-14T08:47:44Z

-        return str(self.agent_os.get_system_info().model_dump_json())
+        server = self.agent_os.get_active_agent_os_server(report=False)
+        system_info_json = self.agent_os.get_system_info().model_dump_json()
+        return f"[Server with id '{server.computer_id}']: {system_info_json}"


same here (Server ->machine)

philipph-askui · 2026-05-14T08:48:47Z

+            agent_os=agent_os,
+        )
+
+    def __call__(self, computer_id: str) -> str:


how is this computer_id determined? By the user, e.g. in the init of the agent? Currently, they can set local_machine_name and remote_machine_name

It can be set by the user, or it defaults to the session ID, which is a UUID4.

philipph-askui · 2026-05-14T08:52:02Z

tbh: at the moment I really like the concept with 2 dedicated toolsets to operate 2 machines. Still, I see that this here will scale also to even more machines.

Have you checked if the agent understands that it can operate multiple machines and that it calls the switch tool before executing operations if needed?

Further: we need to automatically adjust the system capabilities and device information prompts if multiple agentOS are added.

Also: we should definitely update the docs with this PR to explain these changes here

programminx-askui · 2026-05-15T07:00:58Z

+        self, required_tags: list[str]
+    ) -> AgentOs | AndroidAgentOs:
+        """
+        Find the first registered agent OS whose tags are a superset of


Suggested change

Find the first registered agent OS whose tags are a superset of

Find the first registered AgentOS whose tags are a superset of

Please find all occurance. in Strings

programminx-askui · 2026-05-15T07:02:48Z

+
+    def temporary_select(self, device_sn: str) -> AbstractContextManager[Self]:


Suggested change

def temporary_select(self, device_sn: str) -> AbstractContextManager[Self]:

@abstractmethod

def temporary_select(self, device_sn: str) -> AbstractContextManager[Self]:

Ok.

Why is this function not implmented?

programminx-askui · 2026-05-15T07:06:14Z

        raise AndroidAgentOsError(msg)

+    @contextmanager
+    def temporary_select(self, device_sn: str) -> Iterator[Self]:


But in general. Why do we need the concept of temporary_select? How does the agent should call it?

temporary_select is not intended to be called directly by the agent. It is designed to allow users to execute commands on a specific device without needing to restore the previous device context afterward.

It can also be used in custom tools to restrict a tool to a specific device. For example, an enableWifiOnCarEmulator tool could internally use temporary_select, ensuring it only operates on the car emulator. This differs from a generic enableWifi tool, which works across all devices, where the agent is responsible for switching to the appropriate device before invoking the tool.

programminx-askui · 2026-05-15T07:07:09Z

+logger = logging.getLogger(__name__)
+
+
+def _generate_session_guid() -> str:


Do we have already such a function?

programminx-askui · 2026-05-15T08:10:48Z

+                return server
+        return None
+
+    def switch(self, computer_id: str) -> AgentOsServer:


We can switch to the same computer? Should this not failing?

programminx-askui · 2026-05-15T08:11:10Z

+        Raises:
+            KeyError: If no server with the given computer id is registered.
+        """
+        server = self.get(computer_id)


Please wrap a proper AgentOSServerManager Error around.

programminx-askui · 2026-05-15T08:15:01Z

+    def agent_os_server_manager(self) -> AgentOsServerManager:
+        """The underlying Agent-OS-server manager."""
+        return self._manager


Why do we have the server_manager inside the client?

THis should not be here

programminx-askui · 2026-05-15T08:17:15Z

        self.is_cacheable = True

    def __call__(self) -> str:
+        server = self.agent_os.get_active_agent_os_server(report=False)


Intresting.

programminx-askui · 2026-05-15T08:19:21Z

+        assert _generate_session_guid() != _generate_session_guid()
+
+
+class TestReplacePort:


…skui-controller-multi-target

programminx-askui

Let's have a talk.

programminx-askui · 2026-06-03T08:41:41Z

-    OS-level operations (screenshot, mouse, keyboard, ...), and is identified by
-    a unique session GUID. Each server also tracks which display it is currently
-    operating against.
+    A target computer runs the server-side counterpart of the `AgentOs` client


programminx-askui · 2026-06-03T08:43:00Z

+                        "sc",
+                        "query",
+                        LocalAgentOsTargetComputer._ASKUI_CORE_SERVICE_NAME,


Which user rights do you need to perfrom the query? Did you tested it without admin rights?

programminx-askui · 2026-06-03T08:46:28Z

+        if self.is_connected:
+            self.connect(target)


~~Why? When you have already on connected target in the list and then you add another unconnected new target. Then it does not connect the new target.~~

Why do we automaticaly connect to the target? This is only an add-function. Then I would prefere a param auto_connect = True. And replace the buggy implementation, when one target is connected, that the next one get's not connected.

programminx-askui · 2026-06-03T08:47:18Z

+    def add_remote(
+        self,
+        address: str,
+        description: str,
+    ) -> RemoteAgentOsTargetComputer:
+        """
+        Convenience method to construct and register a remote Agent OS target
+        computer. Auto-connects when the manager already has at least one open
+        connection.
+
+        Args:
+            address (str): gRPC address of the remote Agent OS target computer.
+            description (str): Human-readable description.
+
+        Returns:
+            RemoteAgentOsTargetComputer: The newly registered target.
+        """
+        target = RemoteAgentOsTargetComputer(address=address, description=description)
+        self.add(target)
+        return target


Can we delete this function? It does not make sense.

I still see no benefit of this funciton. -> please remove

programminx-askui · 2026-06-03T08:49:05Z

+        if self.is_connected and computer_id not in self._connections:
+            self.connect(target)


Here you trying to fix the logic from above.

What should be the Connection Livecycle of an Target? Is the Manager Responsible to connect to the targets?

programminx-askui · 2026-06-03T08:52:06Z

+            for target in self._by_computer_id.values():
+                self.connect(target)
+        except Exception:
+            self.disconnect_all()


Whats happend when an exception is happing on the disconnect function?

programminx-askui · 2026-06-03T08:53:36Z

+            if hasattr(e, "add_note"):
+                e.add_note(
+                    f"While connecting to Agent OS target computer "
+                    f"{target.description!r} "
+                    f"(computer_id={target.computer_id!r}, "
+                    f"session_guid={target.session_guid}, "
+                    f"display={target.display}, "
+                    f"address={target.address})"
+                )
+            raise


Why aer you not creating a new Exception? You should not manipulate a old one.

programminx-askui · 2026-06-03T08:55:47Z

+            target.start()
+            started_process = True
+        channel = grpc.insecure_channel(
+            target.address,
+            options=[
+                ("grpc.max_send_message_length", 2**30),
+                ("grpc.max_receive_message_length", 2**30),
+                ("grpc.default_deadline", 300000),
+            ],
+        )
+        stub = controller_v1.ControllerAPIStub(channel)


Why do we have inside the Manager GRPC Code? This should be moved to the Client. We need to get rid of this.

programminx-askui · 2026-06-03T08:59:23Z

-    started_process: bool
-

 class AskUiControllerClient(AgentOs):


Why does a Client has a Manager with multiple Targets? This should be another way around: Manager has Multiple Target and the Target should have the Client.

Rename to: MultiComputerTargetAgentOS

programminx-askui

Hello @mlikasam-askui ,

I added some changes. But we have to do after this more work.

programminx-askui · 2026-06-03T14:23:13Z

-    started_process: bool
-

 class AskUiControllerClient(AgentOs):


Rename to: MultiComputerTargetAgentOS

programminx-askui · 2026-06-03T14:23:59Z

+    started_process: bool
+
+
+class AgentOsTargetComputerManager:


Rename to `ComputerTargetPool``

programminx-askui · 2026-06-03T14:26:01Z

+logger = logging.getLogger(__name__)
+
+
+class AgentOsTargetComputer:


Suggested change

class AgentOsTargetComputer:

class ComputerTarget

programminx-askui · 2026-06-03T14:26:29Z

+        )
+
+
+class LocalAgentOsTargetComputer(AgentOsTargetComputer):


Suggested change

class LocalAgentOsTargetComputer(AgentOsTargetComputer):

class LocalComputerTarget(ComputerTarget):

programminx-askui · 2026-06-03T14:27:04Z

+            self._process = None
+
+
+class RemoteAgentOsTargetComputer(AgentOsTargetComputer):


Suggested change

class RemoteAgentOsTargetComputer(AgentOsTargetComputer):

class RemoteComputerTarget(ComputerTarget):

programminx-askui · 2026-06-03T14:45:18Z

+            started_process=started_process,
+        )
+
+    def disconnect_all(self) -> None:


programminx-askui · 2026-06-03T14:48:16Z

+
+    @telemetry.record_call()
+    @override
+    def add_remote_agent_os_target_computer(


Can we not have a add function with accepts

Suggested change

def add_remote_agent_os_target_computer(

def add(computer_target: RemoteComputerTargetOptions | LocalComputerTargetOptions

programminx-askui · 2026-06-03T14:49:44Z

+    def reset_agent_os_target_computers(
+        self,
+        agent_os_target_computers: list[AgentOsTargetComputer] | None = None,
+    ) -> None:


This function should not accept a list. It should use the current pool and reset it

Suggested change

def reset_agent_os_target_computers(

self,

agent_os_target_computers: list[AgentOsTargetComputer] | None = None,

) -> None:

def reset_computer_targets(

self,

) -> None:

programminx-askui · 2026-06-03T14:51:41Z

+    def add_agent_os_target_computer(
+        self, agent_os_target_computer: "AgentOsTargetComputer"
+    ) -> "AgentOsTargetComputer":
+        """Register an additional target computer. Auto-connects if connected."""
+        raise NotImplementedError
+
+    def add_remote_agent_os_target_computer(
+        self,
+        address: str,
+        description: str,
+    ) -> "RemoteAgentOsTargetComputer":
+        """Register an additional remote target computer."""
+        raise NotImplementedError
+
+    def reset_agent_os_target_computers(
+        self,
+        agent_os_target_computers: "list[AgentOsTargetComputer] | None" = None,
+    ) -> None:
+        """Disconnect (if connected) and replace the target computer list."""
+        raise NotImplementedError
+
+    def list_agent_os_target_computers(self) -> "list[AgentOsTargetComputer]":
+        """Return all registered target computers."""
+        raise NotImplementedError
+
+    def get_current_computer_target_id(self, report: bool = True) -> str:
+        """Return the `computer_id` of the currently active target computer."""
+        raise NotImplementedError
+
+    def switch_agent_os_target_computer(
+        self, computer_id: str
+    ) -> "AgentOsTargetComputer":
+        """Switch the active target computer by its `computer_id`."""
+        raise NotImplementedError
+
+    def temporary_select(self, computer_id: str) -> AbstractContextManager[Self]:
+        """
+        Temporarily switch the active target computer for the duration of a `with`
+        block, then restore the previously-active target on exit (even if the
+        block raises).
+
+        Args:
+            computer_id (str): Computer id of the target to activate inside the
+                block.
+
+        Returns:
+            AbstractContextManager[Self]: Context manager that yields this
+                `AgentOs` with the selected target active.
+
+        Example:
+            ```python
+            with agent_os.temporary_select('Remote-Machine') as remote_machine:
+                img = remote_machine.screenshot()
+                img.save("remote_machine.png")
+            # previous active target restored here
+            ```
+        """
+        raise NotImplementedError
+


This should not be part of the default AgentOS

-> Please Remove

-> This should be part of an ComputerAgentOS

programminx-askui · 2026-06-03T14:53:09Z

+
+
+class ComputerSwitchAgentOsTargetComputerTool(ComputerBaseTool):
+    def __init__(self, agent_os: AgentOs | None = None) -> None:


The init should either accept only MultiComputerTargetAgentOS or check if it's an instance of,

So all ComputerBaseTools should only accept an ComputerAgentOS

mlikasam-askui and others added 13 commits May 8, 2026 12:17

add reporter messgaes

768653c

fix logger

01e8f4f

refactor: drop tags from target computers and prefix tool output with…

cfb84f6

… target session GUID

fixed reporter

de55cf2

add temporary_select target

c7f9631

add device_id

aa0aba5

improve doc string

78523b4

improve error message

4ca7f80

add unit tests

9379663

improve tool results

e40c02c

fix lint problems

3493c4a

improve-doc-string

9a475c5

mlikasam-askui mentioned this pull request May 12, 2026

feat: multi Agent OS server support in AskUiControllerClient #275

Closed

mlikasam-askui requested review from philipph-askui and programminx-askui May 12, 2026 09:01

mlikasam-askui self-assigned this May 12, 2026

mlikasam-askui and others added 2 commits May 12, 2026 11:47

feat: capture unhandled errors in reporter

7bd98be

Revert "feat: capture unhandled errors in reporter"

24de514

This reverts commit 7bd98be.

philipph-askui reviewed May 14, 2026

View reviewed changes

philipph-askui requested changes May 14, 2026

View reviewed changes

mlikasam-askui requested a review from philipph-askui May 15, 2026 06:55

programminx-askui requested changes May 15, 2026

View reviewed changes

mlikasam-askui requested a review from programminx-askui May 19, 2026 09:54

implement review remarks

55cdac0

mlikasam-askui force-pushed the SOLENG-360-refactor/askui-controller-multi-target branch from c20d744 to 55cdac0 Compare May 19, 2026 10:08

Merge remote-tracking branch 'origin/main' into SOLENG-360-refactor/a…

40873be

…skui-controller-multi-target

programminx-askui requested changes Jun 3, 2026

View reviewed changes

		from .set_active_display_tool import ComputerSetActiveDisplayTool
		from .switch_agent_os_server_tool import ComputerSwitchAgentOsServerTool

	Find the first registered agent OS whose tags are a superset of
	Find the first registered AgentOS whose tags are a superset of


		def temporary_select(self, device_sn: str) -> AbstractContextManager[Self]:

		logger = logging.getLogger(__name__)


		def _generate_session_guid() -> str:

		assert _generate_session_guid() != _generate_session_guid()


		class TestReplacePort:

		if self.is_connected and computer_id not in self._connections:
		self.connect(target)

		logger = logging.getLogger(__name__)


		class AgentOsTargetComputer:

	class LocalAgentOsTargetComputer(AgentOsTargetComputer):
	class LocalComputerTarget(ComputerTarget):

		self._process = None


		class RemoteAgentOsTargetComputer(AgentOsTargetComputer):

	class RemoteAgentOsTargetComputer(AgentOsTargetComputer):
	class RemoteComputerTarget(ComputerTarget):

	def add_remote_agent_os_target_computer(
	def add(computer_target: RemoteComputerTargetOptions \| LocalComputerTargetOptions



		class ComputerSwitchAgentOsTargetComputerTool(ComputerBaseTool):
		def __init__(self, agent_os: AgentOs \| None = None) -> None:

Conversation

mlikasam-askui commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

philipph-askui commented May 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

programminx-askui left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

programminx-askui Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

programminx-askui left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mlikasam-askui commented May 12, 2026 •

edited

Loading

programminx-askui Jun 3, 2026 •

edited

Loading