# Diffusion model

Diffusion as Shader (DaS) is an innovative video generation control model designed to achieve diverse control over video generation through a 3D perception diffusion process. This model utilizes 3D tracking videos as control inputs, supporting multiple video control tasks under a unified architecture, including mesh-to-video generation, camera control, motion transfer, and object manipulation. The main advantage of DaS lies in its 3D perception capabilities, significantly enhancing the temporal consistency of generated videos and demonstrating powerful control abilities with minimal data and short tuning times. Developed collaboratively by research teams from institutions like the Hong Kong University of Science and Technology, the model aims to advance video generation technology, providing more flexible and efficient solutions for fields such as filmmaking and virtual reality.

DynamicControl is a framework designed to enhance the control of text-to-image diffusion models. It dynamically combines various control signals and supports adaptive selection of different numbers and types of conditions to synthesize images more reliably and in detail. The framework first utilizes a dual-loop controller, employing pre-trained conditional generation and discriminator models to generate initial real score rankings for all input conditions. Then, a multimodal large language model (MLLM) constructs an efficient condition evaluator to optimize the condition ordering. DynamicControl jointly optimizes MLLM and the diffusion model, leveraging the inference capabilities of MLLM to facilitate multi-condition text-to-image tasks. The final ordered conditions are input into a parallel multi-control adapter to learn dynamic visual condition feature maps and integrate them to adjust ControlNet, enhancing control over the generated images.

Stable Diffusion 3.5 Large Turbo is a multi-modal diffusion transformer (MMDiT) model for text-to-image generation, employing Adversarial Diffusion Distillation (ADD) technology to enhance image quality, layout, understanding of complex prompts, and resource efficiency, with a particular focus on reducing inference steps. This model excels in image generation, capable of understanding and generating complex text prompts, making it suitable for various image generation scenarios. It is published on the Hugging Face platform under the Stability Community License, allowing for free use by researchers, non-commercial use, and organizations or individuals with annual revenue under $1 million.

TCAN is a novel human character animation framework based on diffusion models that maintains temporal consistency and generalizes well to unseen domains. The framework ensures that generated videos maintain the original image's appearance while adhering to the posture of the driving video and maintaining background consistency through unique modules such as the Appearance-Posture Adaptation Layer (APPA), temporal control networks, and posture-driven temperature maps.

Hallo, a portrait image animation technology developed by Fudan University, utilizes diffusion models to generate realistic and dynamic portrait animations. Unlike traditional methods relying on parametric models for intermediate facial representations, Hallo adopts an end-to-end diffusion paradigm and introduces a hierarchical audio-driven visual synthesis module to enhance the alignment precision between audio input and visual output, including lip, facial expression, and pose movements. This technology offers adaptive control over facial expression and pose diversity, enabling more effective personalized customization for individuals of different identities.

MuseV is a diffusion model-based virtual human video generation framework that supports the generation of unlimited length videos. It utilizes a novel visual conditional parallel denoising approach. It provides a pre-trained virtual human video generation model, supporting functionalities like Image2Video, Text2Image2Video, and Video2Video. MuseV is compatible with the Stable Diffusion ecosystem, including base models, LoRA, ControlNet, and more. It supports multi-reference image techniques such as IPAdapter, ReferenceOnly, ReferenceNet, and IPAdapterFaceID. MuseV's strength lies in its ability to generate high-fidelity, unlimited length videos, specifically targeting the video generation domain.

DistriFusion is an algorithm that requires no training and takes advantage of multiple GPUs to accelerate diffusion model inference without compromising image quality. DistriFusion can reduce delay based on the number of devices used while maintaining visual fidelity.

DiffusionMat is a novel image matting framework that uses a diffusion model to transform rough to fine alpha matting. Unlike traditional methods, our approach treats image matting as a gradual learning process, starting from adding noise to the trimmed map and iteratively denoising through a pre-trained diffusion model, gradually guiding the prediction towards a clean alpha matting. A key innovation in our framework is a correction module, which adjusts the output in each denoising step to ensure that the final result aligns with the structure of the input image. We also introduce Alpha Reliability Propagation, a novel technique aimed at maximizing the utility of available guidance by selectively enhancing the alpha information in the trimmed map regions with confidence, thus simplifying the correction task. To train the correction module, we have designed a specific loss function to target the accuracy of alpha matting edges and the consistency of opaque and transparent areas. We have evaluated our model on several image matting benchmarks, and the results show that DiffusionMat always outperforms existing methods.

GraphMaker is a diffusion model designed for generating large property graphs. It can generate both node attributes and graph structures simultaneously, or generate node attributes first and then the graph structure. Supported datasets include cora, citeseer, amazon_photo, and amazon_computer. You can leverage pre-trained models to generate graphs.