neurocanvas.net 
How to use ControlNet with Flux AI model
In this article, we’ll walk through the steps to install and use ControlNet with the Flux.1 dev model on ComfyUI. ControlNet allows you to manipulate and control the generation process in AI models using reference images, enhancing the potential of the Flux model for high-quality AI art and video generation.
Table of Contents
Software
We will be using ComfyUI, an alternative to the popular AUTOMATIC1111 for AI image and video generation. If you are new to ComfyUI, make sure to read the [ComfyUI Installation Guide] and [ComfyUI Beginner’s Guide]. These resources will help you get familiar with the basic setup. You can also explore the Quick Start Guide if you are completely new to AI image generation.
For in-depth knowledge, taking the ComfyUI course is highly recommended.
Install ControlNet for Flux.1 dev model
To run ControlNet on the Flux.1 dev model within ComfyUI, we’ll need custom nodes developed by Xlabs AI along with specific ControlNet models. These custom nodes will be integrated into ComfyUI’s workflow.
Step 1: Update ComfyUI
Ensure that your ComfyUI version is up-to-date to avoid compatibility issues with custom nodes.
- In ComfyUI, go to Manager and click on Update ComfyUI.
Step 2: Load ControlNet workflow
Next, download the ControlNet workflow JSON file for Flux.1 dev.
- Import the workflow by uploading the file in ComfyUI.
Step 3: Install missing nodes
After loading the workflow, you may notice some red nodes in the interface, indicating that custom nodes are missing. You can install these nodes directly from ComfyUI.
- Click Manager > Install missing custom nodes.
- Once the installation is complete, restart ComfyUI and refresh the page.
Step 4: Download the Flux.1 dev model
To run the Flux model, you’ll need the following files:
- Flux.1 dev model: Download
flux1-dev-fp8.safetensorsand place it inComfyUI/models/unet. - CLIP models: Download both
clip_l.safetensorsandt5xxl_fp8_e4m3fn.safetensors, and move them toComfyUI/models/clip. - Flux VAE model: Download the VAE model and place it in
ComfyUI/models/vae.
Step 5: Download the Canny ControlNet model
ControlNet relies on edge detection for image generation. Download the Canny ControlNet model:
- File:
flux-canny-controlnet-v3.safetensors - Path:
ComfyUI/models/xlabs/controlnets
Step 6: Run the workflow
With all components in place, your ControlNet setup for Flux.1 dev should be ready to run.
- Upload a reference image to the Load Image node or use a test image provided with the workflow.
- Run the workflow and see the results generated by the model.
Tips for using ControlNet for Flux
To get the best results, follow these tips:
- Adjust strength value: In the Apply Flux ControlNet node, keep the strength value moderate. High values can cause visual artifacts in the generated images.
- Canny Edge node settings: Tweak the
low_thresholdandhigh_thresholdvalues in the Canny Edge node to control how much detail is carried over from the reference image. - Match image sizes: Ensure that the image in the Empty Latent Image node has the same aspect ratio as the reference image for optimal output.
Additional ControlNets for Flux
Once you have successfully set up the Canny ControlNet, you can add other ControlNets for more creative control over image generation. Let’s explore a couple of additional ControlNets below.
Depth ControlNet for Flux
The Depth ControlNet extracts and uses depth information from a reference image, giving you control over the perceived 3D composition of the generated image.
To install:
- Download the Depth ControlNet model:
flux-depth-controlnet-v3.safetensors - Place it in
ComfyUI/models/xlabs/controlnets. - Run the workflow as described above by uploading a reference image.
HED ControlNet for Flux
The HED ControlNet focuses on copying the rough outlines of the reference image, allowing you to replicate the general shape and structure without capturing fine details.
To install:
- Download the HED ControlNet model:
flux-hed-controlnet-v3.safetensors - Place it in
ComfyUI/models/xlabs/controlnets. - Wire the workflow as described and upload a reference image to start the generation process.
By following this guide, you should now be able to install and use ControlNet with the Flux AI model on ComfyUI. Experiment with different ControlNets to achieve diverse and artistic results, and don’t hesitate to tweak the workflow settings for even more control over your AI-generated images.