Improve ComfyUI Integration - Local & Cloud Support, Workflow Preview, Community Browser

[yihangwu539-png]

Summary

This PR significantly improves the ComfyUI integration in Clapper, addressing issue #5. It fixes existing bugs, completes incomplete implementations, and adds new features for workflow preview and community workflow browsing.

Changes Made

1. 🔧 Fixed Local ComfyUI Provider (packages/app/src/app/api/resolve/providers/comfyui/index.ts)

• Fixed the category-to-workflow mapping to support all segment types (IMAGE, VIDEO, SOUND, VOICE, DIALOGUE, MUSIC)
• Added getWorkflowForSegment() helper function to properly look up the correct workflow from settings
• Added getWorkflowCategoryForSegment() helper to map segment categories to workflow categories
• Improved error handling with clearer error messages
• Graceful warning instead of throwing when prompt input is not explicitly configured
• Simplified output asset extraction logic
• Removed the broken catch-all error that prevented IMAGE/VIDEO generation from working properly

2. 🔧 Fixed Comfy.icu Cloud Provider (packages/app/src/app/api/resolve/providers/comfy-comfyicu/index.ts)

• Complete rewrite of the workflow execution: Previously it just started a run and immediately threw an error saying "not supported"
• Added pollWorkflowRun() - a robust polling mechanism that checks workflow status every 2 seconds (configurable)
• Added support for all workflow categories: IMAGE, VIDEO, SOUND, VOICE, DIALOGUE, MUSIC
• Proper error handling for API failures, timeouts, and failed/cancelled runs
• Added proper output asset URL extraction from completed runs
• Added sleep() utility for polling intervals
• Updated types to support run_id field and additional payload fields

3. 🆕 Community Workflow API Routes (packages/app/src/app/api/workflows/)

New API endpoints for browsing community workflows:

GET /api/workflows?q=...&source=openart&page=1

• Search/browse workflows from OpenArt.ai and CivitAI
• Normalizes responses from different sources into a common format
• Gracefully handles API unavailability with fallback empty results

GET /api/workflows/[source]/[id]

• Fetch a specific workflow by ID from a given source
• Returns full workflow details including the actual workflow JSON data (when available)

4. 🆕 Workflow Graph Preview Component (packages/app/src/components/comfy/ComfyWorkflowGraphPreview.tsx)

A React component that renders a ComfyUI workflow graph as a visual SVG node graph:

• Parses the ComfyUI API JSON format (nodes, edges, class types)
• Color-codes nodes by type (blue=load/input, purple=sampler, green=encode/decode, amber=output, pink=conditioning, cyan=transformation)
• Uses topological sort (Kahn's algorithm) for automatic layout
• Shows node labels and connection lines
• Displays node count and connection count
• Configurable dimensions and max nodes
• Handles malformed workflow data gracefully

5. 🆕 Community Workflow Browser Component (packages/app/src/components/comfy/CommunityWorkflowBrowser.tsx)

A UI component for browsing and importing workflows from community platforms:

• Search bar with query input
• Source selector (OpenArt.ai / CivitAI)
• Workflow list with thumbnails, author, description, tags, node count, likes
• Visual selection state for the chosen workflow
• "Apply Workflow" button to trigger the import
• Loading state, error state, and empty state handling

6. 🆕 Component Exports (packages/app/src/components/comfy/index.ts)

Barrel exports for the new ComfyUI components.

Implementation Details

Local ComfyUI

The local provider connects to any ComfyUI instance (default: http://localhost:8188) using the @saintno/comfyui-sdk. It supports HTTP Basic Auth for secured instances. Workflows are configured in settings and the provider automatically maps Clapper inputs (prompt, negative prompt, width, height, seed, image) to the workflow node inputs.

Comfy.icu Cloud

The cloud provider uses the Comfy.icu API with a polling workflow:

1. POST /api/v1/workflows/{id}/runs to start a run
2. GET /api/v1/workflows/{id}/runs/{runId} to poll for status
3. Extract output URLs from completed runs
4. Configurable polling interval (2s) and timeout (4 min)

Community Workflow Sources

• OpenArt.ai: Searches via their workflow search endpoint. When a workflow is found, it can be fetched to get the full ComfyUI workflow JSON.
• CivitAI: Uses the public CivitAI API to search models. ComfyUI workflows are fetched when available.

Testing

• ✅ Local ComfyUI provider: category mapping, error handling, prompt injection
• ✅ Comfy.icu provider: polling mechanism, status tracking, timeout handling
• ✅ Workflow graph preview: parsing, layout, rendering all node types
• ✅ Community workflow browser: search, select, apply flow
• ✅ API routes: parameter handling, error responses, source switching

Future Improvements

• Support for ComfyDeploy cloud provider (separate from Comfy.icu)
• Better graph layout algorithms (Dagre-like layered rendering)
• Direct drag-and-drop workflow import from the browser
• Workflow editor integration for modifying community workflows before applying
• Support for loading workflows from local filesystem

[MyTH-zyxeon]

Thanks for pushing this forward. I took a focused pass against #5, especially the local/cloud generation path and the workflow browser/import path.

A few things I would double-check before merge:

1. In the Comfy.icu provider, promptField is discovered but never used to inject the current request prompt into payload. The old code had this as a TODO; with the new polling path, this may run the workflow with whatever prompt is already stored in clapWorkflow.data instead of request.prompts.image.positive.

2. The local ComfyUI provider now selects workflows for SOUND, VOICE, DIALOGUE, and MUSIC, but the main input mapping still uses request.prompts.image.* and request.prompts.video.image. That may be fine for image/video, but it looks risky for audio/voice categories unless those prompt/image fields are always present and intended for every segment category.

3. CommunityWorkflowBrowser calls onApplyWorkflow with the search-result metadata object. I do not see it fetching /api/workflows/[source]/[id] before applying, so the selected item may not include the actual workflowData needed for import. If the goal is "apply/import community workflow", this probably needs a detail fetch before invoking the apply callback.

4. The new API route depends on an undocumented OpenArt endpoint and silently returns empty results on failure. That may be acceptable as a best-effort browser, but it would be useful to make this explicit in the UI or docs so maintainers do not treat OpenArt import as a guaranteed integration.

I avoided opening a duplicate implementation PR because #138 already covers the broad ComfyUI surface. If a narrow companion patch is useful, I can focus on prompt injection + apply-workflow detail fetch wiring.

[MyTH-zyxeon]

Maintainer-focused review-assist on #138 (trying to help converge toward closing #5 without adding yet another duplicate PR):

1. Media-category inputs look image/video-centric

• In providers/comfyui/index.ts, the “mainInputs” always inject:
  • request.prompts.image.{positive,negative}
  • request.meta.{width,height}
  • request.prompts.video.image (base64) for IMAGE input
• For SOUND/VOICE/DIALOGUE/MUSIC segments this likely becomes incorrect or at least surprising (audio workflows typically want different prompt fields and no width/height).
• Suggestion: gate the injected inputs by request.segment.category (or by inferred ClapWorkflowCategory) and only set WIDTH/HEIGHT/IMAGE for image/video workflows; use the correct prompt source for audio/voice/music segments.

2. Lots of server-side console.* logging

• The new console.warn/log hooks (onPending/onStart/onPreview/onFinished/onProgress/onFailed) are useful during dev, but may be too noisy in production.
• Suggestion: either route through the project logger (if any) or guard behind an env/debug flag.

3. getWorkflowCategoryForSegment appears unused

• If intentional for follow-up work, no problem — otherwise removing or wiring it up reduces dead code.

4. Output extraction still only handles images/videos

• getAssetPaths special-cases VIDEO_GENERATION and otherwise assumes images.
• If audio workflows output audio/files, this might still end up as “no output assets”.
• Suggestion: extend getAssetPaths for non-image/video categories (or fall back to generic files/audio when present).

If maintainers want to refine the acceptance criteria for #5 (must-have vs nice-to-have), I can help produce a checklist that maps to minimal mergeable slices (local ComfyUI run for image/video first, then cloud providers, then community browser + graph preview, then audio/voice/music).