ComfyUI Video Workflow Setup

ComfyUI and Video Generation

ComfyUI is a node-based AI image and video tool. You build workflows by connecting nodes. Compared to Web UI, ComfyUI is more flexible and scriptable, suited for advanced users and batch production.

Video-Related Nodes

Input

• Load Image: Load single frame or image sequence as condition
• Load Video: Load video; extract frames or use as reference
• Empty Latent: Create blank latent for pure text-to-video

Model and Sampling

• Checkpoint Loader: Load main model (e.g., AnimateDiff, SVD-compatible)
• VAE Encode / Decode: Latent encode/decode
• KSampler / KSampler Advanced: Sampler; control steps, CFG, seed

Conditioning and Prompts

• CLIP Text Encode: Text encoding; supports positive/negative prompts
• Conditioning: Combine conditions (e.g., ControlNet, IP-Adapter)

Output

• Save Image: Save single frame
• Save Video: Save video sequence (convert frame sequence to video)
• Preview: Live preview

Video-Specific

• AnimateDiff: Motion modules, context length
• SVD (Stable Video Diffusion): Image-to-video nodes
• Frame Interpolation: Frame interpolation for smoother playback

Basic Workflow Setup

Load model: Checkpoint Loader → connect to Sampler and VAE

Set prompts: CLIP Text Encode (pos/neg) → Conditioning → Sampler

Prepare latent: Empty Latent or Load Image + VAE Encode

Sample: Sampler output → VAE Decode → Save Image/Video

For video, treat the latent batch dimension as time, or use dedicated video sampling nodes.

Custom Workflows

Subgraphs and Groups

• Group common nodes into a Group; collapse into a submodule
• Expose parameters via input/output interfaces for reuse

Parameterization

• Make Prompt, seed, steps, etc. adjustable
• Use ComfyUI API or custom nodes to pass values from outside

Save and Load

• Save stores the current workflow as .json
• Load loads an existing workflow
• Build a template library for quick switching by project type

Batch Generation

Option 1: Queue

• Add multiple parameter sets to the Queue
• Each set can change Prompt, seed, etc.; they run in order

Option 2: Scripts and API

• Submit jobs via ComfyUI HTTP API
• Use Python or other scripts to loop and change inputs
• Integrate with upstream systems (CMS, task queues)

Option 3: Batch Nodes

• Some nodes support batch input for multiple items at once
• Watch VRAM; large batches can cause OOM

Performance and Stability

• VRAM: Video generation is memory-heavy; lower resolution, shorten length, use --lowvram
• Models: Ensure Checkpoint and AnimateDiff/SVD versions are compatible
• Cache: Use model caching to avoid repeated loads

Summary

ComfyUI’s node-based, scriptable approach gives flexible video workflows. Once you know the core nodes, subgraph grouping, and batch generation, you can build pipelines from single runs to automated production.