Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions .github/workflows/provider-ux-copy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
name: Provider UX verification
on:
pull_request:
branches: [main]
paths:
- "packages/app/src/renderer/libs/provider-status*"
- "packages/app/src/renderer/libs/hooks/use-local-ai-providers*"
- "packages/app/src/renderer/components/chat/ChannelEmptyState.tsx"
- "packages/app/src/renderer/components/settings/pages/general-page.tsx"
- "packages/app/src/renderer/components/settings/pages/agents-page.tsx"
- "packages/app/automation/verify-provider-guidance.mts"
- ".github/workflows/provider-ux-copy.yml"
jobs:
provider-ux:
runs-on: macos-latest
timeout-minutes: 30
env:
NODE_OPTIONS: --max-old-space-size=4096
permissions:
contents: read
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-node@v4
with:
node-version: "22"
- uses: pnpm/action-setup@v4
- name: Install locked dependencies
run: pnpm install --frozen-lockfile
- name: Test provider status and refresh recovery
working-directory: packages/app
run: pnpm exec vitest run src/renderer/libs/provider-status.test.ts src/renderer/libs/hooks/use-local-ai-providers.test.ts
- name: Prepare actual Electron app
run: |
set -euo pipefail
mkdir -p packages/app/.automation/ux-evidence
pnpm --filter @convera/app automation:prepare 2>&1 | tee packages/app/.automation/ux-evidence/after-build.log
! grep -E 'FATAL ERROR|heap out of memory' packages/app/.automation/ux-evidence/after-build.log
test -s packages/app/.vite/renderer/main_window/index.html
test -s packages/app/.vite/build/preload.js
test -s packages/app/.vite/build/main.js
- name: Verify provider setup in isolated Electron profile
working-directory: packages/app
env:
UX_VARIANT: after
run: pnpm exec tsx automation/verify-provider-guidance.mts
- name: Prepare baseline checkout
env:
BASE_SHA: ${{ github.event.pull_request.base.sha }}
run: |
git worktree add "$RUNNER_TEMP/convera-before" "$BASE_SHA"
ln -s "$GITHUB_WORKSPACE/node_modules" "$RUNNER_TEMP/convera-before/node_modules"
if [ -d "$GITHUB_WORKSPACE/packages/app/node_modules" ]; then
ln -s "$GITHUB_WORKSPACE/packages/app/node_modules" "$RUNNER_TEMP/convera-before/packages/app/node_modules"
fi
- name: Prepare baseline Electron app
working-directory: ${{ runner.temp }}/convera-before
run: |
set -euo pipefail
pnpm --filter @convera/app automation:prepare 2>&1 | tee "$GITHUB_WORKSPACE/packages/app/.automation/ux-evidence/before-build.log"
! grep -E 'FATAL ERROR|heap out of memory' "$GITHUB_WORKSPACE/packages/app/.automation/ux-evidence/before-build.log"
test -s packages/app/.vite/renderer/main_window/index.html
test -s packages/app/.vite/build/preload.js
test -s packages/app/.vite/build/main.js
- name: Capture actual baseline with the same scenario
working-directory: ${{ runner.temp }}/convera-before/packages/app
env:
UX_VARIANT: before
run: EVIDENCE_DIR="$GITHUB_WORKSPACE/packages/app/.automation/ux-evidence" pnpm exec tsx "$GITHUB_WORKSPACE/packages/app/automation/verify-provider-guidance.mts"
- name: Upload native verification evidence
if: always()
uses: actions/upload-artifact@v4
with:
name: convera-provider-ux-evidence
path: |
packages/app/.automation/ux-evidence
packages/app/.automation/logs
if-no-files-found: error
27 changes: 27 additions & 0 deletions docs/ux-review-providers.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Provider and memory guidance review

Scope: first-use channel copy, provider selection/checking, settings, memory privacy/cost expectations. Source baseline: `0e21210e7f9bcfb6164c95b53338621a98e46cb6`. The UX-writing skill at https://www.skills.sh/content-designer/ux-writing-skill/ux-writing was read, including its full upstream SKILL.md. Copy was reviewed for purpose, concision, conversational tone and factual clarity.

## Before → after

- “Local AI Providers” implied local inference → “AI providers”, with local history and external provider processing explained
- Missing API key / CLI status was a dead end → names the required environment variable or install/sign-in step and how to re-check
- Merely present API keys showed “Ready” → “Configured”, with usage-charge context, without claiming key validation
- Re-check did not re-enter loading, rejected checks were unhandled and could leave stale ready badges → visible checking/error/retry states; failed checks clear stale availability and older results cannot overwrite a newer check
- “Selected/Select” left the scope unclear → “Default/Use by default/Setup required”, with separate colleague-level provider choice explained
- Channel empty state claimed “Nobody here can answer” based only on the default provider → names that provider’s setup need and explains colleague-specific choices; checking no longer displays a false failure
- Suggestions did not explain their behavior → “Choose a suggestion to edit before sending”
- Local memory copy obscured background AI processing → explains that completed turns are sent to the chosen curator and provider charges may apply
- Missing memory status displayed zero pending/failed jobs → “Status unavailable”; memory load errors provide Reload memory settings

## Verification

- Eight new tests exercise provider status guidance, missing bridge, rejected/unsuccessful IPC, stale-ready clearing, retry, checking, and out-of-order responses
- Full unit suite: 586 passed, 3 existing skipped in the cloud checkout
- `Provider UX verification` builds and drives the actual Electron app through the repository’s automation MCP on macOS, in isolated before/after profiles
- Native scenario: open settings, observe unavailable OpenAI API and its disabled choice, re-check, inspect memory guidance and capture screenshots; no credentials or billable model requests
- Evidence: `{before,after}-{workspace,provider-settings,memory-settings}.png` and matching `*-automation.json`

## Boundaries

Cloud-local `automation:prepare` is blocked by a Unix-socket EPERM; the CI scenario supplies the real-machine check. Local whole-app type-check was terminated by the environment, so unit results do not imply that check passed. No configured-provider chat, AI output quality, MCP service account, memory curation request, production rollout or macOS security change is exercised. CI errors fail visibly; they are not converted into a success result.
193 changes: 193 additions & 0 deletions packages/app/automation/verify-provider-guidance.mts
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
import assert from "node:assert/strict";
import { copyFile, mkdir, rm, writeFile } from "node:fs/promises";
import path from "node:path";
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";

const appRoot = process.cwd();
const variant = process.env.UX_VARIANT ?? "after";
const profileId = `ux-copy-${variant}`;
const evidenceDir = path.resolve(
process.env.EVIDENCE_DIR ?? ".automation/ux-evidence",
);
await mkdir(evidenceDir, { recursive: true });
const env = Object.fromEntries(
Object.entries(process.env).filter(
(entry): entry is [string, string] => typeof entry[1] === "string",
),
);
// This verifies provider setup without credentials or billable model requests.
delete env.OPENAI_API_KEY;
delete env.FIREWORKS_API_KEY;
const transport = new StdioClientTransport({
command: "pnpm",
args: ["automation"],
cwd: appRoot,
env,
stderr: "pipe",
});
const client = new Client({
name: "convera-ux-verification",
version: "1.0.0",
});
const log: unknown[] = [];
async function call(name: string, args: Record<string, unknown>) {
const result = await client.callTool({ name, arguments: args }, undefined, {
timeout: 120000,
});
const blocks = result.content as Array<{ type: string; text?: string }>;
const body = blocks.find((block) => block.type === "text")?.text;
const parsed = body ? JSON.parse(body) : null;
log.push({ name, args, result: parsed });
await writeFile(
path.join(evidenceDir, `${variant}-automation.json`),
JSON.stringify(log, null, 2),
);
assert.ok(!result.isError && parsed?.ok, JSON.stringify(parsed));
return parsed.data;
}
async function capture(name: string) {
const shot = await call("convera_observe", { action: "screenshot" });
await copyFile(
shot.filePath,
path.join(evidenceDir, `${variant}-${name}.png`),
);
}
async function find(name: string) {
const elements = await call("convera_observe", {
action: "snapshot",
max_elements: 500,
});
const element = elements.find((item: { name: string }) => item.name === name);
assert.ok(element, `Missing visible control: ${name}`);
return element.selector as string;
}
let launched = false;
try {
await client.connect(transport);
await call("convera_session", { action: "launch", profile_id: profileId });
launched = true;
// The semantic toolset has no resize action. Resize only this isolated
// Electron window so screenshots include the complete settings section.
await call("convera_execute", {
context: "main",
script:
"const window = electron.BrowserWindow.getAllWindows()[0]; window.setSize(1100, 850); return window.getSize();",
});
await call("convera_observe", { action: "snapshot", max_elements: 500 });
await call("convera_wait", {
condition: "displayed",
selector: 'button[aria-label="Open settings"]',
timeout_ms: 60000,
});
await call("convera_observe", { action: "snapshot", max_elements: 500 });
await capture("workspace");
await call("convera_interact", {
action: "click",
selector: await find("Open settings"),
});
await call("convera_wait", {
condition: "exists",
selector: '[data-testid="local-ai-provider-openai-api"]',
timeout_ms: 60000,
});
const providerSelector = '[data-testid="local-ai-provider-openai-api"]';
await call("convera_wait", {
condition: "text_contains",
selector: providerSelector,
text: "No API key",
timeout_ms: 60000,
});
const settingsBody = await call("convera_observe", {
action: "element",
selector: "body",
});
const providerHeading =
variant === "before" ? "Local AI Providers" : "AI providers";
assert.ok(settingsBody.text.includes(providerHeading));
await call("convera_interact", {
action: "scroll",
selector: `//h2[normalize-space()="${providerHeading}"]`,
});
const provider = await call("convera_observe", {
action: "element",
selector: providerSelector,
});
assert.equal(provider.enabled, false);
if (variant === "after")
assert.match(provider.text, /Set OPENAI_API_KEY.*restart Convera/);
await capture("provider-settings");
const refreshSelector = await find("Re-check providers");
await call("convera_interact", {
action: "click",
selector: refreshSelector,
});
await call("convera_wait", {
condition: "enabled",
selector: refreshSelector,
timeout_ms: 60000,
});
await call("convera_wait", {
condition: "text_contains",
selector: providerSelector,
text: "No API key",
timeout_ms: 60000,
});
await find("Background curator");
const memoryHeading =
variant === "before" ? "Memory and Context" : "Memory and context";
assert.ok(settingsBody.text.includes(memoryHeading));
await call("convera_interact", {
action: "scroll",
selector: `//h2[normalize-space()="${memoryHeading}"]`,
});
const body = await call("convera_observe", {
action: "element",
selector: "body",
});
if (variant === "after")
assert.match(body.text, /completed turns are sent|Sends completed turns/);
await capture("memory-settings");
// Leave settings and return without changing any provider or memory choice.
const snapshot = await call("convera_observe", {
action: "snapshot",
max_elements: 500,
});
const back = snapshot.find((item: { name: string }) =>
/back to (workspace|chat)/i.test(item.name),
);
if (back) {
await call("convera_interact", {
action: "click",
selector: back.selector,
});
await call("convera_observe", { action: "snapshot", max_elements: 500 });
}
} catch (error) {
if (launched) {
// Keep diagnostic evidence without turning a failed assertion into success.
for (const action of ["snapshot", "logs"] as const) {
try {
await call("convera_observe", { action, max_elements: 500 });
} catch (diagnosticError) {
log.push({ diagnosticError: String(diagnosticError) });
}
}
try {
await capture("failure");
} catch (diagnosticError) {
log.push({ diagnosticError: String(diagnosticError) });
}
}
throw error;
} finally {
try {
if (launched) await call("convera_session", { action: "close" });
} finally {
await client.close();
}
await rm(path.join(appRoot, ".automation", "profiles", profileId), {
recursive: true,
force: true,
});
}
22 changes: 13 additions & 9 deletions packages/app/src/renderer/components/chat/ChannelEmptyState.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ export function ChannelEmptyState({
// Every starter colleague is pinned to this provider at hire time, so a
// missing key here means nobody in the room can answer — worth saying before
// the user types into a room that will stay silent.
const { providers, loading } = useLocalAIProviders();
const { providers, loading, error } = useLocalAIProviders();
const provider = providers.find(
(candidate) => candidate.id === DEFAULT_LOCAL_AI_PROVIDER_ID,
);
Expand Down Expand Up @@ -95,16 +95,19 @@ export function ChannelEmptyState({
</div>
))}
<p className="text-xs leading-relaxed text-muted-foreground">
They read this room like you do and answer when spoken to. Mention
one by name to ask them directly, or say something to nobody in
particular and whoever fits will pick it up.
Mention a colleague by name to ask them directly. For a general
message, each colleague decides whether to reply. Their AI provider
must be configured before they can answer.
</p>
</div>
)}

<div className="space-y-2">
<p className="text-xs text-muted-foreground">
Choose a suggestion to edit before sending.
</p>
<p className="text-xs font-medium uppercase tracking-wide text-muted-foreground">
Try
Add a starter message
</p>
{prompts.map((prompt) => (
<button
Expand All @@ -118,20 +121,21 @@ export function ChannelEmptyState({
))}
</div>

{status && !status.ready && (
{!loading && (error || (status && !status.ready)) && (
<div className="flex items-start gap-2 rounded-lg border border-destructive px-3 py-2 text-xs text-destructive">
<AlertTriangle size={13} className="mt-0.5 flex-shrink-0" />
<p className="min-w-0 flex-1 leading-relaxed">
Nobody here can answer yet — {status.label}
{status.hint ? `: ${status.hint}` : "."}{" "}
{error ??
`${provider?.name ?? "The default provider"} needs setup. ${status?.hint ?? "Check provider settings."}`}{" "}
<button
type="button"
onClick={onOpenSettings}
className="underline pointer-events-auto hover:no-underline"
>
Open Settings → General
</button>{" "}
to set one up.
to check providers. Colleagues may have a different provider set in
Agents.
</p>
</div>
)}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -337,8 +337,8 @@ const AgentFormFields = ({
</p>
) : (
<p className="text-xs text-muted-foreground">
Each colleague can run on its own model — a reviewer on a stronger
one, a note-taker on something cheap.
This choice applies to this colleague. Messages and context go to
the selected provider; its terms and usage charges apply.
</p>
)}
</div>
Expand Down
Loading
Loading