Add optional Notte provider for semantic browser workflows

.env.example

@@ -11,7 +11,10 @@ SECRET_ENCRYPTION_KEY=
 # Required for browser execution outside the Vercel one-click flow.
 # The Kernel Marketplace resource injects this automatically on Vercel.
 # For manual setup, create a key at https://kernel.sh.
+BROWSER_PROVIDER=kernel
 KERNEL_API_KEY=
+# Alternatively set BROWSER_PROVIDER=notte and provide only NOTTE_API_KEY.
+NOTTE_API_KEY=
 # Required for model inference outside Vercel. Vercel deployments use project
 # OIDC for AI Gateway access instead.
 AI_GATEWAY_API_KEY=

AGENTS.md

@@ -60,7 +60,7 @@ Run the validation the task requests. When it does not establish the behavior yo
 - The workspace manager lives on `/` and the agent chat on `/chat`; browser execution belongs only to the declared browser-agent subagent's flat tool surface under `agent/subagents/browser-agent/tools`.
 - Keep each worker browser tool's schema and implementation together. Share the Kernel SDK client through `agent/subagents/browser-agent/lib/kernel.ts`; do not add a Kernel extension or root browser connection.
 - `agent/subagents/browser-agent/lib` is for code genuinely shared by worker tools. Group a shared worker domain in a lower-case folder, such as `trace/domains.ts` or `autofill/provider.ts`; do not use it as a holding area for a tool's one-off logic.
-- Validate runtime environment variables through `shared/environment/env.ts`. `KERNEL_API_KEY` is required by the worker browser tools.
+- Validate runtime environment variables through `shared/environment/env.ts`. `BROWSER_PROVIDER` defaults to `kernel`; worker browser tools require `KERNEL_API_KEY` for Kernel or `NOTTE_API_KEY` for Notte.
 - Run `pnpm check` and `pnpm build` before handing off changes.
 
 ## Code organization

README.md

@@ -64,6 +64,41 @@ Treat the private Blob store as production key material: deleting it loses the
 automatically generated encryption key, and rotating that key requires
 re-encrypting existing vault values.
 
+### Notte browser provider
+
+To use [Notte](https://www.notte.cc/) cloud browsers instead of Kernel, set:
+
+```dotenv
+BROWSER_PROVIDER=notte
+NOTTE_API_KEY=your-notte-api-key
+```
+
+`BROWSER_PROVIDER` defaults to `kernel`. With Notte selected, `KERNEL_API_KEY`
+is not required. The one-click Vercel button still provisions Kernel; configure
+these variables yourself for an existing deployment or a manual installation.
+
+Notte sessions support the semantic browser tools (`browser_snapshot`,
+`browser_text`, `browser_find`, `browser_act`, `browser_wait_for`), live viewing,
+and the existing secure vault autofill over CDP. Each workspace has its own
+Notte profile; `save_changes: true` persists login state when the browser is
+closed, and only one writer can be active. Read-only sessions can run in parallel.
+CDP connection credentials are kept out of tool results.
+
+Live validation confirmed cookie-backed profile restoration, but a profile
+containing only localStorage did not restore that state. Treat storage-only
+login persistence as unverified until that issue is resolved.
+
+Kernel's remote `playwright_execute`, desktop `computer_action`, and
+`capture_browser_image` tools are omitted from the Notte tool set. Notte workers
+use semantic actions instead and return no image attachments. Set the viewport
+at creation; resizing is not supported. Idle timeouts and maximum session lifetimes are 15–30 minutes (default
+15), subject to the Notte account’s plan limits. These sessions explicitly use direct
+connections without proxies and enable CAPTCHA solving. Trace domains currently
+include the starting URL only for Notte.
+
+Profiles and open sessions stay with their original provider. Finish/delete open
+sessions before switching providers; switching does not migrate saved logins.
+
 ### Blob storage
 
 The one-click deploy creates and connects a private Blob store automatically.
@@ -190,8 +225,8 @@ development is a manual path and requires:
 
 - Node.js 24 and pnpm 11.24.0
 - Docker Desktop or another running Docker Compose installation
-- Kernel credentials from a [Kernel API key](https://kernel.sh) or a linked
-  Vercel Marketplace resource
+- Browser credentials: a [Kernel API key](https://kernel.sh), a linked
+  Vercel Marketplace resource, or `BROWSER_PROVIDER=notte` with `NOTTE_API_KEY`
 - AI Gateway access from an API key or a linked Vercel project's OIDC token
 
 First clone and install the application:
@@ -202,13 +237,14 @@ cd OpenInstinct
 pnpm install --frozen-lockfile
 ```
 
-For fully manual setup, copy the environment template and add your Kernel and AI
+For fully manual setup, copy the environment template and add your browser provider and AI
 Gateway keys:
 
 ```bash
 cp .env.example .env.local
 
 # Set KERNEL_API_KEY and AI_GATEWAY_API_KEY in .env.local.
+# For Notte, set BROWSER_PROVIDER=notte and NOTTE_API_KEY instead of KERNEL_API_KEY.
 ```
 
 If you already use a Vercel project, link it to pull AI Gateway access. If that
@@ -230,11 +266,11 @@ pnpm dev
 migrations, and starts the application. Stopping the development process also
 stops and removes the PostgreSQL container; its data remains in the
 `postgres-data` volume for the next run. Run `pnpm dev:app` when intentionally
-using an externally managed database instead. If `KERNEL_API_KEY` is missing,
+using an externally managed database instead. If the selected browser provider's API key is missing,
 `pnpm dev` stops before starting Docker and points back to the recommended
 Vercel flow or the manual `.env.local` setup.
 
-Local development otherwise uses the same vault, Kernel browser, and AI Gateway
+Local development otherwise uses the same vault, selected browser provider, and AI Gateway
 path as the Vercel deployment. Better Auth and vault encryption use stable
 local-only defaults when their variables are unset. Vercel deployments
 provision them automatically in private Blob; other production hosts require

agent/subagents/browser-agent/instructions.md

@@ -23,10 +23,11 @@ You are `browser-agent`, the root coordinator's dedicated browser executor. Comp
 
 # Execution
 
-- Use `playwright_execute` as the primary browser execution surface. Prefer one bounded program per page state that inspects, performs related safe actions, verifies the meaningful outcome, and returns a compact result. When Playwright is unreliable or semantic interaction is more suitable, inspect with `browser_snapshot`, `browser_text`, or `browser_find`, then use `browser_act` for a short relaxed action plan. `browser_act` dispatches actions and returns the successor state without waiting for model-authored postconditions; do not repeat an action merely because strict causal verification is absent. Use `browser_wait_for` only when the next operation truly depends on a delayed user-visible state. Use current refs only, and snapshot again after navigation, a stale-ref error, or an unavailable successor.
-- Use `computer_action` only when the page requires visual reasoning or coordinate input that the semantic browser tools cannot express. Never use fixed multi-second sleeps; use `browser_wait_for` with a specific semantic state, URL, title, value, or element condition.
+- When `playwright_execute` is available, use it as the primary browser execution surface. Prefer one bounded program per page state that inspects, performs related safe actions, verifies the meaningful outcome, and returns a compact result. When Playwright is unreliable or semantic interaction is more suitable, inspect with `browser_snapshot`, `browser_text`, or `browser_find`, then use `browser_act` for a short relaxed action plan. `browser_act` dispatches actions and returns the successor state without waiting for model-authored postconditions; do not repeat an action merely because strict causal verification is absent. Use `browser_wait_for` only when the next operation truly depends on a delayed user-visible state. Use current refs only, and snapshot again after navigation, a stale-ref error, or an unavailable successor.
+- With Notte, use `browser_snapshot`, `browser_text`, `browser_find`, `browser_act`, and `browser_wait_for` as the primary surface. Remote Playwright programs, desktop actions, and durable image capture are unavailable; return an empty `images` array. Use the live viewer when a task requires human visual interaction.
+- Use `computer_action`, when available, only when the page requires visual reasoning or coordinate input that the semantic browser tools cannot express. Never use fixed multi-second sleeps; use `browser_wait_for` with a specific semantic state, URL, title, value, or element condition.
 - Create one browser and reuse it. Pass a known target as `start_url`. Start read-only; immediately before a saved login is needed, replace it at the same URL with `save_changes: true`, and delete that writer as soon as authentication succeeds so the profile is saved. Only one writable workspace browser may exist.
-- Kernel stealth includes managed CAPTCHA solving. Leave a challenge untouched and make one bounded wait of at most 20 seconds. If it remains, preserve the browser and return the takeover blocker and live-view URL. Never bypass authentication, CAPTCHAs, paywalls, or other access controls.
+- Browser sessions enable managed CAPTCHA solving. Leave a challenge untouched and make one bounded wait of at most 20 seconds. If it remains, preserve the browser and return the takeover blocker and live-view URL. Never bypass authentication, CAPTCHAs, paywalls, or other access controls.
 - Keep ordinary `computer_action` screenshots temporary and model-visible only. Use `capture_browser_image` only when the assignment requests an image or visual evidence materially improves the final result. Prefer an `image_resource` for a requested item photo, and return only descriptors actually produced by the capture tool.
 - Re-read the page after coordinator-approved continuation or human takeover because the browser state may have changed.
 - Delete the browser when the assignment succeeds or ends without a pending approval or human action. Keep it open only when approval, authentication, CAPTCHA, or takeover is the sole remaining blocker.

agent/subagents/browser-agent/lib/autofill/native.ts

@@ -1,3 +1,4 @@
+import { notteCdpUrl } from "../notte";
 import Kernel from "@onkernel/sdk";
 import { z } from "zod";
 import { env } from "@shared/environment";
@@ -611,10 +612,17 @@ async function withKernelPage<T>(
     readonly sessionId: readonly string[];
   }) => Promise<T>
 ) {
-  const browser = await new Kernel({
-    apiKey: env.KERNEL_API_KEY,
-  }).browsers.retrieve(browserSessionId, {}, { signal });
-  const connection = await CdpConnection.connect(browser.cdp_ws_url, signal);
+  const cdpUrl =
+    env.BROWSER_PROVIDER === "notte"
+      ? await notteCdpUrl(browserSessionId, signal)
+      : (
+          await new Kernel({ apiKey: env.KERNEL_API_KEY }).browsers.retrieve(
+            browserSessionId,
+            {},
+            { signal }
+          )
+        ).cdp_ws_url;
+  const connection = await CdpConnection.connect(cdpUrl, signal);
 
   try {
     const { targetInfos } = targetListSchema.parse(

agent/subagents/browser-agent/lib/kernel.ts

@@ -1,4 +1,5 @@
 import Kernel from "@onkernel/sdk";
 import { env } from "@shared/environment";
 
-export const kernel = new Kernel({ apiKey: env.KERNEL_API_KEY });
+// Notte uses CDP directly and does not require a Kernel credential.
+export const kernel = new Kernel({ apiKey: env.KERNEL_API_KEY ?? "" });

agent/subagents/browser-agent/lib/notte.ts

@@ -0,0 +1,168 @@
+import { createHash } from "node:crypto";
+import { z } from "zod";
+import { env } from "@shared/environment";
+
+const sessionSchema = z.object({
+  session_id: z.string().min(1),
+  created_at: z.string(),
+  status: z.enum(["active", "closed", "error", "timed_out"]),
+  cdp_url: z.url().nullish(),
+  viewer_url: z.url().nullish(),
+  viewport_width: z.number().nullish(),
+  viewport_height: z.number().nullish(),
+});
+const profileSchema = z.object({
+  profile_id: z.string(),
+  name: z.string().nullable(),
+});
+
+// Keep provider and write mode in the existing opaque session ID. This also
+// prevents a provider switch from sending old sessions to the wrong service.
+export function notteSessionId(id: string, writable: boolean) {
+  return `notte:${writable ? "write" : "read"}:${id}`;
+}
+
+export function isNotteSession(id: string) {
+  return id.startsWith("notte:");
+}
+
+function remoteSessionId(id: string) {
+  const match = /^notte:(?:read|write):(.+)$/u.exec(id);
+  if (!match?.[1]) throw new Error("Invalid Notte browser session ID.");
+  return encodeURIComponent(match[1]);
+}
+
+async function request(
+  path: string,
+  method: string,
+  body?: z.infer<ReturnType<typeof z.json>>,
+  signal?: AbortSignal
+) {
+  if (!env.NOTTE_API_KEY)
+    throw new Error("NOTTE_API_KEY is required for Notte browsers.");
+  const response = await fetch(`https://api.notte.cc${path}`, {
+    method,
+    headers: {
+      Authorization: `Bearer ${env.NOTTE_API_KEY}`,
+      "Content-Type": "application/json",
+    },
+    body: body === undefined ? undefined : JSON.stringify(body),
+    signal: signal
+      ? AbortSignal.any([signal, AbortSignal.timeout(60_000)])
+      : AbortSignal.timeout(60_000),
+  });
+  if (!response.ok) {
+    // Do not expose response bodies, which can include connection credentials.
+    throw Object.assign(
+      new Error(`Notte request failed (HTTP ${String(response.status)}).`),
+      { status: response.status }
+    );
+  }
+  const value: unknown = await response.json();
+  return value;
+}
+
+export async function ensureNotteProfile(
+  workspaceId: string,
+  signal?: AbortSignal
+) {
+  const name = `openinstinct-${createHash("sha256").update(workspaceId).digest("hex").slice(0, 40)}`;
+  const profiles = z
+    .object({ items: z.array(profileSchema) })
+    .parse(
+      await request(
+        `/profiles?name=${name}&page_size=100`,
+        "GET",
+        undefined,
+        signal
+      )
+    );
+  const existing = profiles.items.find((profile) => profile.name === name);
+  if (existing) return existing.profile_id;
+  return profileSchema.parse(
+    await request("/profiles/create", "POST", { name }, signal)
+  ).profile_id;
+}
+
+export async function startNotteBrowser(
+  input: {
+    profileId: string;
+    writable: boolean;
+    timeoutSeconds: number;
+    viewport?: { width: number; height: number };
+  },
+  signal?: AbortSignal
+) {
+  if (input.timeoutSeconds > 1800)
+    throw new Error("Notte idle timeout cannot exceed 1800 seconds.");
+  const session = sessionSchema.parse(
+    await request(
+      "/sessions/start",
+      "POST",
+      {
+        browser_type: "chromium",
+        proxies: false,
+        solve_captchas: true,
+        idle_timeout_minutes: Math.ceil(input.timeoutSeconds / 60),
+        max_duration_minutes: Math.ceil(input.timeoutSeconds / 60),
+        profile: { id: input.profileId, persist: input.writable },
+        viewport_width: input.viewport?.width ?? null,
+        viewport_height: input.viewport?.height ?? null,
+      },
+      signal
+    )
+  );
+  return descriptor(
+    session,
+    notteSessionId(session.session_id, input.writable)
+  );
+}
+
+export async function retrieveNotteBrowser(id: string, signal?: AbortSignal) {
+  const session = sessionSchema.parse(
+    await request(`/sessions/${remoteSessionId(id)}`, "GET", undefined, signal)
+  );
+  return descriptor(session, id);
+}
+
+export async function stopNotteBrowser(id: string, signal?: AbortSignal) {
+  await request(
+    `/sessions/${remoteSessionId(id)}/stop`,
+    "DELETE",
+    undefined,
+    signal
+  );
+}
+
+function descriptor(session: z.infer<typeof sessionSchema>, id: string) {
+  return {
+    session_id: id,
+    created_at: session.created_at,
+    cdp_ws_url: session.cdp_url ?? undefined,
+    browser_live_view_url: session.viewer_url ?? undefined,
+    status: session.status === "active" ? "active" : "deleted",
+    profile_save_changes: id.startsWith("notte:write:"),
+    viewport:
+      session.viewport_width && session.viewport_height
+        ? { width: session.viewport_width, height: session.viewport_height }
+        : undefined,
+  };
+}
+
+export async function notteCdpUrl(id: string, signal?: AbortSignal) {
+  const browser = await retrieveNotteBrowser(id, signal);
+  if (browser.status !== "active")
+    throw new Error("Notte browser is closed. Create a new session.");
+  if (browser.cdp_ws_url) return browser.cdp_ws_url;
+  const debug = z
+    .object({ ws: z.object({ cdp: z.url() }) })
+    .parse(
+      await request(
+        `/sessions/${remoteSessionId(id)}/debug`,
+        "GET",
+        undefined,
+        signal
+      )
+    );
+  return debug.ws.cdp;
+}