Polish agent answers: one-round write errors, no-match not an error, shorter instructions, UTF-8

CHANGELOG.md

@@ -2,6 +2,14 @@
 
 ## Unreleased
 
+### Polish: agent answers
+
+- **One round of write errors:** `create_item` and `update_item` no longer stop at the tool's own argument check. Wrong types, missing required fields, unknown fields, bad dates and versions, bad labels and entity keys all come back in one message. The argument list still advertises types, required fields and limits.
+- **No match is not an error:** `list_items` with no match (by `q` or `entity_key`) returns a normal answer telling the agent to say the item is not in the library.
+- **Date order** (review date on or after effective date) is checked only when both dates are real, so an invalid date no longer brings a misleading second message.
+- **Shorter agent instructions:** about half the length (publisher, Review Lock: 5,662 → 2,975 characters; Reader key: 1,563 → 1,299; write tool descriptions about 30% shorter). The safety rules are unchanged: cite or say it is not there, published items only, Review Lock drafts and human-only publish/revert.
+- **UTF-8:** MCP responses declare `charset=utf-8`, so strict clients no longer garble em dashes and accents.
+
 ### Stricter, clearer item writes over MCP
 
 - **Labels are normalized everywhere:** `create_item` and `update_item` now store labels trimmed, lower case and unique, like the web form. Migration `024_normalize_labels.sql` cleans existing labels, including the published copy and the previous published copy (without counting as a new publish: citation version and review date are unchanged).

docs/concepts.md

@@ -67,7 +67,8 @@ Each Connect link is one library. Tools offered depend on the caller's role in t
 - **Lengths:** name 200, summary 160, labels 40 each and at most 20. A field without its own limit uses one default per type (text 200, longtext 8,000, markdown 50,000), the same in the tool's argument list and in the save check.
 - **Labels** are stored trimmed, lower case and without duplicates, whether they come from an agent, the form or `suggest_labels`. Labels are free-form.
 - **Summary** on a plain General library comes from the body; an agent that sends `summary` there is told to put the text in `body`.
-- **Errors** list every problem in one message (with the library's field list), so an agent can fix them in one retry.
+- **Errors** list every problem in one message (with the library's field list), so an agent can fix them in one retry. The tool's argument list shows types, required fields and limits, but the save check enforces them, so a wrong type, a missing field and an unknown field all come back together. Review date versus effective date is compared only when both are real dates.
+- **No match** from `list_items` (by `q` or `entity_key`) is a normal answer that says the item is not in the library, not a tool error.
 
 **How `list_items q=` searches.**
 1. Every word must match (name, labels and aliases, summary, then body; the full-text index applies English stemming). Aliases score exactly like labels.

src/http/app.ts

@@ -11,6 +11,7 @@ import {
   requireBearerAuth,
 } from "@modelcontextprotocol/express";
 import { toNodeHandler } from "@modelcontextprotocol/node";
+import { declareUtf8 } from "./utf8-charset.ts";
 import {
   buildOAuthProtectedResourceMetadata,
   createMcpHandler,
@@ -343,6 +344,7 @@ export function createApp(input: {
       if (!info) throw new Error("MCP handler invoked without authInfo");
       return buildMcpServer(db, info, scoped, { publicBaseUrl: config.publicBaseUrl });
     });
+    declareUtf8(res);
     void toNodeHandler(mcpHandler)(req, res, req.body);
   }
 
@@ -457,6 +459,7 @@ export function createApp(input: {
           });
         });
         const nodeMcp = toNodeHandler(mcpHandler);
+        declareUtf8(res);
         void nodeMcp(req, res, req.body);
       })().catch(next);
     });

src/http/utf8-charset.ts

@@ -0,0 +1,44 @@
+import type { ServerResponse } from "node:http";
+
+/** text/event-stream and application/json without a charset get "; charset=utf-8". */
+export function withUtf8(value: string): string {
+  if (/;\s*charset=/i.test(value)) return value;
+  if (/^\s*(text\/event-stream|application\/json)\s*(;|$)/i.test(value)) return `${value.trim()}; charset=utf-8`;
+  return value;
+}
+
+/**
+ * MCP responses are UTF-8, but the SDK sends `text/event-stream` with no charset,
+ * so strict clients decode em dashes and accents as Latin-1. Add the charset on
+ * whatever path the SDK uses to set the header.
+ */
+export function declareUtf8(res: ServerResponse): void {
+  const setHeader = res.setHeader.bind(res);
+  res.setHeader = ((name: string, value: number | string | readonly string[]) =>
+    setHeader(name, name.toLowerCase() === "content-type" && typeof value === "string" ? withUtf8(value) : value)) as typeof res.setHeader;
+  const writeHead = res.writeHead.bind(res) as (...args: unknown[]) => ServerResponse;
+  res.writeHead = ((...args: unknown[]) => {
+    for (const arg of args) {
+      if (Array.isArray(arg)) {
+        // Raw [name, value, name, value, ...] or [[name, value], ...].
+        if (arg.every((entry) => Array.isArray(entry))) {
+          for (const pair of arg as [string, unknown][]) {
+            if (String(pair[0]).toLowerCase() === "content-type" && typeof pair[1] === "string") pair[1] = withUtf8(pair[1]);
+          }
+        } else {
+          for (let i = 0; i + 1 < arg.length; i += 2) {
+            if (String(arg[i]).toLowerCase() === "content-type" && typeof arg[i + 1] === "string") arg[i + 1] = withUtf8(arg[i + 1]);
+          }
+        }
+      } else if (arg && typeof arg === "object") {
+        const headers = arg as Record<string, unknown>;
+        for (const key of Object.keys(headers)) {
+          if (key.toLowerCase() === "content-type" && typeof headers[key] === "string") headers[key] = withUtf8(headers[key] as string);
+        }
+      }
+    }
+    const current = res.getHeader("content-type");
+    if (typeof current === "string") setHeader("content-type", withUtf8(current));
+    return writeHead(...args);
+  }) as typeof res.writeHead;
+}

src/items/library-schema.ts

@@ -297,55 +297,54 @@ export function schemaAgentBlurb(schema: LibrarySchema): string {
  * MCP tool argument types for create_item / update_item on this library only.
  * Built from the stored schema so clients do not see other starters' fields.
  */
+/**
+ * Tool arguments are advertised with their JSON type but not enforced by the SDK:
+ * the save check enforces types, required fields and limits, and reports every
+ * problem in one answer (the SDK would stop at its own first pass).
+ */
+function advertisedString(description: string): z.ZodType {
+  return z.unknown().optional().meta({ type: "string", description });
+}
+
 function mcpZodForField(field: SchemaFieldDef): z.ZodType {
   const max = fieldMax(field);
   let description = field.label;
   if (field.type === "date") description = `${field.label} (YYYY-MM-DD)`;
   if (field.type === "version") description = `${field.label} (for example 1.0)`;
   if (field.type === "markdown") description = `${field.label} (markdown)`;
-  const base = z.string().trim().max(max).describe(description);
-  if (field.required) return base.min(1, `${field.label} is required`);
-  return base.optional();
+  const rule = field.required ? `Required. At most ${max} characters.` : `Optional. At most ${max} characters.`;
+  return advertisedString(`${description}. ${rule}`);
 }
 
 /** Field shape for create_item (no id). */
 export function mcpItemWriteShape(schema: LibrarySchema): Record<string, z.ZodType> {
   const shape: Record<string, z.ZodType> = {
-    name: z.string().trim().min(1).max(200).describe("Item name"),
-    entity_key: entityKeySchema,
-    labels: z
-      .array(z.string().trim().min(1).max(40))
-      .max(20)
-      .optional()
-      .describe("Labels (optional)"),
+    // Advertised, not enforced (see advertisedString): the save check reports every problem at once.
+    name: advertisedString("Item name. Required. At most 200 characters."),
+    entity_key: advertisedString(
+      "Optional stable id for this real-world entity (for example host:contoso-01). Letters, digits and . _ : / - ; at most 120 characters.",
+    ),
+    labels: z.unknown().optional().meta({
+      type: "array",
+      items: { type: "string" },
+      description: "Optional labels: at most 20, each at most 40 characters. Stored lower case.",
+    }),
   };
   if (summaryIsDerived(schema)) {
     if (!isPlainGeneral(schema)) {
-      shape.summary = z
-        .string()
-        .trim()
-        .max(160)
-        .optional()
-        .describe("Optional one-line summary (list/search excerpt). Derived from the body when omitted");
+      shape.summary = advertisedString(
+        "Optional one-line summary (list/search excerpt), at most 160 characters. Derived from the body when omitted",
+      );
     }
   } else {
-    shape.summary = z
-      .string()
-      .trim()
-      .min(1)
-      .max(160)
-      .describe("Required one-line summary (list/search excerpt)");
+    shape.summary = advertisedString("One-line summary (list/search excerpt). Required. At most 160 characters.");
   }
   for (const field of schema.fields) {
     shape[field.key] = mcpZodForField(field);
   }
   // Legacy Policy alias — optional, only when this library has a policy field.
   if (schema.fields.some((field) => field.key === "policy")) {
-    shape.rule = z
-      .string()
-      .max(20_000)
-      .optional()
-      .describe("Deprecated alias for policy; prefer policy");
+    shape.rule = advertisedString("Deprecated alias for policy; prefer policy");
   }
   return shape;
 }
@@ -399,14 +398,18 @@ export function mcpWriteToolDescription(
   const head =
     writeMode === "direct"
       ? kind === "create"
-        ? "Add a library item. This library is Review Unlock — the item is live immediately. Set entity_key for the real-world entity: if an item with that entity_key already exists, it is updated instead (fields you leave out keep their value). Give a stable name, a one-line summary, and useful labels."
-        : "Update one library item by id after list_items found it (prefer entity_key). Fields you leave out keep their current value; pass replace: true to replace the whole item. This library is Review Unlock — the saved version is live immediately. Keep entity_key and name stable."
+        ? "Add an item; Review Unlock, so it is live immediately. Set entity_key for the real-world entity: if an item with that entity_key already exists, it is updated instead. Give a stable name, a one-line summary, and useful labels."
+        : "Update one item by id (find it with list_items, prefer entity_key). Fields you leave out keep their current value; replace: true replaces the whole item. Review Unlock: live immediately."
       : kind === "create"
-        ? "Add a library item as a draft (Pending). Not visible to readers or Reader keys until a human Publishes it in the publisher UI. Optional entity_key is a stable id: if an item with that entity_key already exists, it is updated instead (fields you leave out keep their value)."
-        : "Update one library item by id. Fields you leave out keep their current value; pass replace: true to replace the whole item. Saves as draft: if published, the edit is Pending until a human Publishes it, and Readers keep the published version.";
+        ? "Add an item as a Pending draft; readers see it only after a human Publishes. If an item with the optional entity_key already exists, it is updated instead."
+        : "Update one item by id. Fields you leave out keep their current value; replace: true replaces the whole item. Saves as Pending; readers keep the published version until a human Publishes.";
   const summaryBit = summaryIsDerived(schema) ? "" : ", summary";
   const shape = schema.custom ? "custom" : schema.starter;
-  return `${head} This library uses the ${shape} schema. Send name${summaryBit}${fieldList ? `, ${fieldList}` : ""}. Optional: entity_key, labels. Do not invent fields. To add a field, propose_field after the human confirms — they Approve in the publisher. Use only when the human asked.`;
+  return `${head} ${shape} schema: send name${summaryBit}${fieldList ? `, ${fieldList}` : ""}; optional entity_key, labels. Do not invent fields. Every problem comes back in one answer.`;
+}
+
+function isRealDate(value: unknown): boolean {
+  return typeof value === "string" && isoDate("Date").safeParse(value).success;
 }
 
 function isoDate(label: string) {
@@ -477,6 +480,9 @@ function unexpectedFields(schema: LibrarySchema, input: ItemWriteInput): string[
     extras.push("body");
   }
   const issues: string[] = [];
+  if (input.labels !== undefined && !Array.isArray(input.labels)) {
+    issues.push("labels must be an array of strings, for example [\"vpn\", \"security\"].");
+  }
   if (extras.length > 0) issues.push(`Unknown field(s): ${[...new Set(extras)].join(", ")}.`);
   return issues;
 }
@@ -525,6 +531,10 @@ export function itemForLibrarySchema(
 
   // Every problem is collected and returned together, not just the first.
   const issues = unexpectedFields(schema, input);
+  if (input.entity_key !== undefined) {
+    const key = entityKeySchema.safeParse(input.entity_key);
+    if (!key.success) issues.push(issueLines(key.error.issues).join(" "));
+  }
   const failed = (more: string[]) => ({
     ok: false as const,
     message: `${[...issues, ...more].join(" ")} ${schemaFieldMessage(schema)}`.trim(),
@@ -536,7 +546,6 @@ export function itemForLibrarySchema(
       name: input.name,
       body: input.body ?? "",
       labels: normalizeLabels(input.labels),
-      entity_key: input.entity_key,
     });
     if (!parsed.success) return failed(issueLines(parsed.error.issues));
     if (issues.length > 0) return failed([]);
@@ -573,11 +582,10 @@ export function itemForLibrarySchema(
     const raw = payload as Record<string, unknown>;
     if (
       schema.starter === "policy" &&
-      typeof raw.review_date === "string" &&
-      typeof raw.effective_date === "string" &&
-      /^\d{4}-\d{2}-\d{2}$/.test(raw.review_date.trim()) &&
-      /^\d{4}-\d{2}-\d{2}$/.test(raw.effective_date.trim()) &&
-      raw.review_date.trim() < raw.effective_date.trim()
+      // Only compare two real dates; a bad date already has its own message.
+      isRealDate(raw.review_date) &&
+      isRealDate(raw.effective_date) &&
+      (raw.review_date as string).trim() < (raw.effective_date as string).trim()
     ) {
       lines.push("Review date must be on or after the effective date.");
     }

src/items/schema.ts

@@ -118,10 +118,11 @@ export type CreateLibraryInput = z.infer<typeof createLibraryInputSchema>;
  * as the web form (parseLabelField) but without truncating, so an over-long label
  * or too many labels is still refused by validation instead of being cut silently.
  */
-export function normalizeLabels(labels: readonly string[] | undefined): string[] {
+export function normalizeLabels(labels: readonly string[] | string | undefined): string[] {
   const seen = new Set<string>();
   const out: string[] = [];
-  for (const raw of labels ?? []) {
+  if (!Array.isArray(labels)) return out;
+  for (const raw of labels) {
     const label = raw.trim().toLowerCase();
     if (!label || seen.has(label)) continue;
     seen.add(label);

src/mcp/library-design.ts

@@ -3,24 +3,12 @@
  * The human hears one question at a time. Tools run only after they confirm.
  */
 export function libraryDesignGuide(): string {
-  return `Designing a library — publishers only. Guide the human. This is a conversation, not a form.
-
-Speak like a careful editor of a library. One question at a time. Wait for the answer. Use their words when you reflect the purpose back. Do not call create_library, propose_table, or propose_field until they have confirmed the design.
-
-1. Purpose. Ask what this library is for, and what question it should answer. Restate that purpose in one sentence and let them correct it.
-2. Name. Offer a short library name from that purpose. Let them change it.
-3. How writes go live. Ask whether a human should Publish before items are visible (Review Lock), or whether saves should be live immediately for readers and agents (Review Unlock). Recommend Review Lock for policies and procedures. Recommend Review Unlock only for automation or task output they want live without a Publish step. Do not guess.
-4. Shape. Recommend one shape from the purpose. Starters are guides, not sealed forever. Do not ask them to pick a schema code.
-   - A policy, when they are keeping a rule people must follow. Suggest the usual fields in plain language: who owns it, the version, when it starts and must be reviewed, why it exists, who it covers, and what it requires. They may add more later with propose_field.
-   - A procedure, when they are keeping the steps that carry out a policy. Suggest: who owns the steps, which policy they serve, when to start, the steps, and who does each step.
-   - A table, when they are keeping rows of facts — a roster, a list, a calendar. One item is one row. Design the columns with them.
-5. Table design, only for a table. Take one column at a time. Ask for the label, then whether it is a short fact (text), a date, or a longer note (long text), then whether every row needs it. Suggest the next column from the purpose. Keep the table small. Stop when they say it is enough. Never go past 12 columns. Offer one sample row so they can see a filled-in example. Sample rows are for review. They are not saved as items.
-6. Read-back. Say the library name, Review Lock or Review Unlock, the purpose in one line, and the columns (or the policy or procedure shape). Ask them to confirm. Change whatever they correct. Do not call a tool on this turn.
-7. Only after they confirm:
-   - Policy: create_library with schema policy and the confirmed write_mode. Do not send a table.
-   - Procedure: create_library with schema procedure and the confirmed write_mode. Do not send a table.
-   - A new table: create_library with schema general, the confirmed write_mode, and the table set to the confirmed design. general is only the placeholder until they approve the table in the publisher.
-   - A table for this library, when you are already connected to it and it has no items: propose_table with the confirmed design. Do not create a second library.
-   - This library already has items: do not propose a table here. To add one optional field after the schema is locked, confirm with the human then propose_field. Do not invent rename or remove. Cap 12 fields.
-8. After the tool returns, tell them the human step in everyday words. Open the publisher and Approve the table or field when one is waiting. For a new library, connect that library’s own link before adding items. After they Approve a field, tell them to reconnect the MCP link. Do not invent an approve, reject, or write-mode tool. In Review Lock libraries, new items stay Pending until they Publish. In Review Unlock libraries, saves are live immediately. Only a human can change Review Lock / Review Unlock in the publisher UI.`;
+  return `Designing a library (publishers only) is a conversation, not a form. One question at a time; wait for the answer. Do not call create_library, propose_table, or propose_field until the human has confirmed the design.
+1. Purpose: ask what it is for; restate it in one sentence.
+2. Name: offer a short one from the purpose.
+3. How writes go live: Review Lock (a human Publishes first; recommend for policies and procedures) or Review Unlock (saves live at once; only for automation output). Ask; do not guess.
+4. Shape: a policy (owner, version, start and review dates, why, who it covers, what it requires), a procedure (owner, related policy, when to start, steps, who does each step), or a table of rows. For a table, take one column at a time (label; text, date or long text; required?), at most 12, and show one sample row.
+5. Read back name, write mode, purpose and fields. Change what they correct. No tool call on this turn.
+6. After they confirm: create_library (schema policy or procedure, or general with the table). For this library: propose_table only while it has no items; once it has items, propose_field for one optional field. Never rename or remove fields.
+7. Then tell them the human step: Approve the table or field in the publisher; connect a new library's own link before adding items. Do not invent an approve, reject, or write-mode tool.`;
 }