feat: release getmnemo-migrate 0.2.0

README.md

@@ -26,6 +26,8 @@ getmnemo-migrate from raw-csv \
   --container pilot:customer-name
 ```
 
+Small documents are sent through Mnemo's bulk endpoint in bounded batches. If the destination is running an older API without that endpoint, the CLI falls back to the existing single-document calls automatically.
+
 The `content=*` mapping serializes each complete CSV or JSON record. For cleaner source data, map a specific field instead, such as `content=notes,id=record_id,company=company`.
 
 ## Import Lanes
@@ -56,13 +58,15 @@ getmnemo-migrate from raw-json \
   --container pilot:customer-name
 ```
 
+Map `container=<field>` when one approved export contains multiple customers. Each record is then written to its own isolated container; `--container` remains the fallback.
+
 ## Provider Sources
 
 | Source | Source credentials | Useful options |
 | --- | --- | --- |
 | `mem0` | `MEM0_API_KEY` | `--user`, `--base-url` |
 | `zep` | `ZEP_API_KEY` | `--user` is required, `--base-url` |
-| `supermemory` | `SUPERMEMORY_API_KEY` | `--base-url` |
+| `supermemory` | `SUPERMEMORY_API_KEY` | `--source-container` is required, `--base-url` |
 | `letta` | `LETTA_API_KEY`, `LETTA_BASE_URL` | Imports core and archival memory |
 
 ## Safe Operating Flow
@@ -72,15 +76,40 @@ getmnemo-migrate from raw-json \
 3. Import into a customer-specific container with conservative concurrency.
 4. Keep the printed migration job ID.
 5. Run `status <jobId>` or `reconcile <jobId>` to obtain the final completion report.
-6. Validate retrieval against a small set of customer-approved questions before connecting an agent.
+6. Retry isolated write failures with `retry-failed <jobId>`.
+7. Validate retrieval against a small set of customer-approved questions before connecting an agent.
 
 ```bash
 getmnemo-migrate status
 getmnemo-migrate cancel mig_...
 getmnemo-migrate resume mig_...
 getmnemo-migrate reconcile mig_...
+getmnemo-migrate retry-failed mig_...
+```
+
+Use `--json` with planning, imports, resume, retry, reconciliation, status, and cancellation when another system needs to consume the report.
+
+Create a verification file after agreeing expected answers with the customer:
+
+```json
+[
+  {
+    "question": "What is the current support response time?",
+    "expect": ["two business hours"],
+    "searchMode": "hybrid",
+    "limit": 10
+  }
+]
 ```
 
+Then run retrieval acceptance against the same isolated container:
+
+```bash
+getmnemo-migrate verify mig_... --questions ./acceptance-questions.json --json
+```
+
+The command exits non-zero when an expected phrase is missing. For an import that mapped records to several containers, pass `--container` to verify one customer boundary at a time.
+
 Cancellation pauses after the current bounded batch. Resume skips successful source records and retries unresolved ones. Mnemo custom IDs and idempotency keys also protect against duplicate writes if local state is lost.
 
 Job state, processed IDs, remote job IDs, and failure journals are stored under `~/.getmnemo/migrations` with owner-only permissions. Set `GETMNEMO_MIGRATION_DIR` to use another state directory.
@@ -89,6 +118,9 @@ Job state, processed IDs, remote job IDs, and failure journals are stored under
 
 - Concurrency is capped at 25 even if a larger value is requested.
 - HTTP 429 and server/network failures retry up to five times with backoff.
+- Malformed JSONL rows are journaled while valid rows continue.
+- Failed writes can be retried in a new job without replaying successful records.
+- Bulk requests contain at most 50 documents and stay below a conservative request-size ceiling.
 - Documents larger than the API limit are split deterministically near line boundaries.
 - Atomic memories longer than the memory endpoint limit are split with part metadata.
 - JSON arrays are capped at 50 MB to avoid loading unbounded files into memory; JSONL and CSV stream.

docs/pilot-integration.md

@@ -43,17 +43,24 @@ getmnemo-migrate from raw-csv \
   --map 'content=*,id=Record ID' \
   --container pilot:customer-name \
   --concurrency 5
+
+getmnemo-migrate reconcile mig_... --json
+getmnemo-migrate retry-failed mig_... --json
+getmnemo-migrate verify mig_... \
+  --questions ./acceptance-questions.json \
+  --json
 ```
 
-Record the migration job ID and keep the local migration directory until acceptance is complete. A paused or interrupted import is resumed with `getmnemo-migrate resume <jobId>`; it does not restart successful records.
+Record the migration job ID and keep the local migration directory until acceptance is complete. A paused or interrupted import is resumed with `getmnemo-migrate resume <jobId>`; it does not restart successful records. Malformed rows and write failures are isolated in the job journal, and `retry-failed` creates a separate auditable retry job for unresolved writes.
 
 ## Runtime API Contract
 
 The integrated system needs three Mnemo operations:
 
 | Operation | Endpoint | Purpose |
 | --- | --- | --- |
-| Add source material | `POST /v1/documents` | Asynchronous ingestion with provenance and deterministic `customId`. |
+| Add source material | `POST /v1/documents/batch` | Bounded asynchronous ingestion with per-record results, provenance, and deterministic `customId`. |
+| Single-write fallback | `POST /v1/documents` | Used automatically for oversized records or older API deployments. |
 | Check ingestion | `GET /v1/jobs?ids=...` | Completion and failure reconciliation in groups of at most 50 job IDs. |
 | Retrieve context | `POST /v1/search` | Scoped retrieval for the governance layer before an agent acts. |
 
@@ -69,6 +76,7 @@ The pilot is ready for agent integration when all of the following are true:
 - the final report has no pending jobs and any failed records are explained or retried;
 - rerunning or resuming does not create duplicate source records;
 - customer-approved test questions retrieve the expected current context and provenance;
+- the machine-readable verification report has zero failed questions;
 - searches cannot cross the pilot customer’s workspace and container boundary;
 - removing Mnemo access from the governance service prevents further reads and writes.
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "getmnemo-migrate",
-  "version": "0.1.1",
+  "version": "0.2.0",
   "description": "Resumable CLI for importing historical documents and memories into Mnemo.",
   "type": "module",
   "bin": {

src/adapters/base.ts

@@ -9,8 +9,17 @@ export interface SourceRecord {
   metadata?: Record<string, unknown>;
   /** Optional source-provided content classification. */
   contentType?: string;
+  /** Optional per-record destination container resolved from source data. */
+  containerTag?: string;
 }
 
+export interface AdapterIssue {
+  sourceId: string;
+  error: string;
+}
+
+export type AdapterIssueHandler = (issue: AdapterIssue) => void | Promise<void>;
+
 export interface AdapterConfig {
   /** Optional path to a JSONL file (used by raw-jsonl adapter). */
   file?: string;
@@ -22,14 +31,26 @@ export interface AdapterConfig {
   baseUrl?: string;
   /** Page size hint for paginated providers. */
   pageSize?: number;
+  /** Source-side containers to read from when the provider requires scoping. */
+  sourceContainers?: string[];
 }
 
 export abstract class Adapter {
   abstract readonly name: string;
   abstract readonly defaultLane: ImportLane;
 
+  private issueHandler?: AdapterIssueHandler;
+
   constructor(protected readonly cfg: AdapterConfig = {}) {}
 
+  setIssueHandler(handler: AdapterIssueHandler): void {
+    this.issueHandler = handler;
+  }
+
+  protected async reportIssue(issue: AdapterIssue): Promise<void> {
+    await this.issueHandler?.(issue);
+  }
+
   /** Best-effort total record count. Return null if unknown. */
   abstract count(): Promise<number | null>;
 

src/adapters/file-adapters.test.ts

@@ -5,6 +5,7 @@ import { afterEach, beforeEach, describe, expect, it } from "vitest";
 import { DirectoryAdapter } from "./directory.js";
 import { RawCsvAdapter } from "./raw-csv.js";
 import { RawJsonAdapter } from "./raw-json.js";
+import { RawJsonlAdapter } from "./raw-jsonl.js";
 
 async function collect(adapter: { iterate(): AsyncIterable<unknown> }): Promise<unknown[]> {
   const records: unknown[] = [];
@@ -96,4 +97,44 @@ describe("file adapters", () => {
       Status: "Customer",
     });
   });
+
+  it("maps each record to its own destination container", async () => {
+    const file = join(directory, "customers.csv");
+    await fs.writeFile(file, "id,notes,customer\n1,Alpha note,customer:alpha\n");
+    const adapter = new RawCsvAdapter({
+      file,
+      map: "content=notes,id=id,container=customer",
+    });
+
+    await expect(collect(adapter)).resolves.toEqual([
+      {
+        sourceId: "1",
+        content: "Alpha note",
+        contentType: undefined,
+        containerTag: "customer:alpha",
+        metadata: { source: "raw-csv", file },
+      },
+    ]);
+  });
+
+  it("reports malformed JSONL records and continues with valid records", async () => {
+    const file = join(directory, "mixed.jsonl");
+    await fs.writeFile(
+      file,
+      '{"id":"1","content":"first"}\nnot-json\n{"id":"3","content":"third"}\n',
+    );
+    const adapter = new RawJsonlAdapter({ file });
+    const issues: Array<{ sourceId: string; error: string }> = [];
+    adapter.setIssueHandler((issue) => issues.push(issue));
+
+    const records = await collect(adapter) as Array<{ sourceId?: string }>;
+
+    expect(records.map((record) => record.sourceId)).toEqual(["1", "3"]);
+    expect(issues).toEqual([
+      {
+        sourceId: "line:2",
+        error: expect.stringContaining("Invalid JSON on line 2"),
+      },
+    ]);
+  });
 });

src/adapters/raw-jsonl.ts

@@ -40,7 +40,11 @@ export class RawJsonlAdapter extends Adapter {
       try {
         parsed = JSON.parse(trimmed) as Record<string, unknown>;
       } catch (err) {
-        throw new Error(`Invalid JSON on line ${lineNo} of ${this.filePath}: ${(err as Error).message}`);
+        await this.reportIssue({
+          sourceId: `line:${lineNo}`,
+          error: `Invalid JSON on line ${lineNo} of ${this.filePath}: ${(err as Error).message}`,
+        });
+        continue;
       }
       const record = recordFromObject(parsed, this.map, {
         source: "raw-jsonl",

src/adapters/record-mapping.ts

@@ -44,9 +44,17 @@ export function recordFromObject(
 
   const id = pluck(input, map.id!);
   const contentType = map.contentType ? pluck(input, map.contentType) : undefined;
+  const container = map.container ? pluck(input, map.container) : undefined;
   const metadata: Record<string, unknown> = { ...baseMetadata };
   for (const [target, source] of Object.entries(map)) {
-    if (target === "content" || target === "id" || target === "contentType") continue;
+    if (
+      target === "content" ||
+      target === "id" ||
+      target === "contentType" ||
+      target === "container"
+    ) {
+      continue;
+    }
     const value = pluck(input, source);
     if (value !== undefined) metadata[target] = value;
   }
@@ -56,5 +64,9 @@ export function recordFromObject(
     content,
     metadata,
     contentType: typeof contentType === "string" ? contentType : undefined,
+    containerTag:
+      typeof container === "string" || typeof container === "number"
+        ? String(container)
+        : undefined,
   };
 }

src/adapters/supermemory.test.ts

@@ -0,0 +1,49 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { SupermemoryAdapter } from "./supermemory.js";
+
+afterEach(() => {
+  delete process.env.SUPERMEMORY_API_KEY;
+  vi.unstubAllGlobals();
+});
+
+describe("SupermemoryAdapter", () => {
+  it("reads current v4 memory pages for an explicitly scoped source container", async () => {
+    process.env.SUPERMEMORY_API_KEY = "source-key";
+    const requests: Array<{ url: string; body: Record<string, unknown> }> = [];
+    vi.stubGlobal(
+      "fetch",
+      vi.fn(async (url: string | URL | Request, init?: RequestInit) => {
+        const body = JSON.parse(String(init?.body)) as Record<string, unknown>;
+        requests.push({ url: String(url), body });
+        const page = Number(body.page);
+        return new Response(
+          JSON.stringify({
+            memoryEntries: [
+              {
+                id: `memory-${page}`,
+                memory: page === 1 ? "first memory" : "second memory",
+                createdAt: "2026-07-22T00:00:00.000Z",
+              },
+            ],
+            pagination: { currentPage: page, totalPages: 2, totalItems: 2, limit: 1 },
+          }),
+          { status: 200, headers: { "content-type": "application/json" } },
+        );
+      }),
+    );
+
+    const adapter = new SupermemoryAdapter({
+      sourceContainers: ["customer:alpha"],
+      pageSize: 1,
+    });
+    const records = [];
+    for await (const record of adapter.iterate()) records.push(record);
+
+    expect(requests).toHaveLength(2);
+    expect(requests[0]).toEqual({
+      url: "https://api.supermemory.ai/v4/memories/list",
+      body: { containerTags: ["customer:alpha"], limit: 1, page: 1 },
+    });
+    expect(records.map((record) => record.content)).toEqual(["first memory", "second memory"]);
+  });
+});