Orthogonal Finetuning (OFT) : OFT effectively stabilizes text-to-image diffusion models during fine-tuning

Orthogonal Finetuning (OFT)

Image Generation Model Training and Deployment #Text-to-Image Generation #Image Synthesis #Generative Model #Fine-tuning Standard Picks Paid

Overview :

The study 'Controlling Text-to-Image Diffusion' explores how to effectively guide or control powerful text-to-image generation models for various downstream tasks. The orthogonal finetuning (OFT) method is proposed, which maintains the model's generative ability. OFT preserves the hypershell energy between neurons, preventing the model from collapsing. The authors consider two important fine-tuning tasks: subject-driven generation and controllable generation. Results show that the OFT method outperforms existing methods in terms of generation quality and convergence speed.

Target Users :

["Text-to-Image Generation","Image Synthesis","Conditional Image Generation"]

Total Visits： 0

Website Views ： 60.4K

Use Cases

Fine-tune DALL-E with a small number of samples to generate images of specified themes

Combine text, reference images, and control signals to generate high-quality, controllable images

Prevent the degeneration of the model's generative ability during the fine-tuning process

Features

Maintain the hypershell energy between neurons

Prevent the degradation or collapse of the generative ability of the text-to-image model

Achieve subject-driven image generation