# Diffusion models

text-to-pose is a research project aimed at generating character poses from text descriptions and using these poses to create images. This technology combines natural language processing and computer vision, achieving text-to-image generation by enhancing the control and quality of diffusion models. The project is based on a paper published at the NeurIPS 2024 Workshop, showcasing innovation and cutting-edge advancements. The key advantages of this technology include improved accuracy and controllability in image generation, as well as potential applications in artistic creation and virtual reality.

GameNGen is a fully neural model-driven game engine capable of real-time interaction with complex environments while maintaining high quality over extended trajectories. It can interactively simulate the classic game 'DOOM' at over 20 frames per second, with its next-frame prediction achieving a PSNR of 29.4, comparable to lossy JPEG compression. Human evaluators only slightly outperform random chance in distinguishing between game clips and simulated clips. GameNGen is trained through two phases: (1) an RL-agent learns to play the game and records the actions and observations from the training sessions, which become the training data for the generative model; (2) a diffusion model is trained to predict the next frame, conditioned on the past actions and observation sequences. Conditional enhancement allows for stable autoregressive generation over long trajectories.

Bootstrap3D is a framework aimed at improving 3D content creation by leveraging synthetic data generation techniques to address the scarcity of high-quality 3D assets. It utilizes 2D and video diffusion models to generate multi-view images based on text prompts and employs the 3D-perceptive MV-LLaVA model to filter high-quality data and rewrite inaccurate titles. The framework has generated 1 million high-quality synthetic multi-view images with dense descriptive captions to alleviate the shortage of high-quality 3D data. It also introduces a Training Timestep Reschedule (TTR) strategy that leverages the denoising process to learn multi-view consistency while preserving the original 2D diffusion prior.

Make-An-Audio 2 is a text-to-audio generation technology based on diffusion models, co-developed by researchers from Zhejiang University, ByteDance, and the Chinese University of Hong Kong. This technology utilizes pre-trained large language models (LLMs) to parse text, optimizing for semantic alignment and temporal consistency, thereby improving the quality of generated audio. It also incorporates a feed-forward Transformer-based diffusion denoiser to enhance performance in generating variable-length audio and bolster the extraction of temporal information. Furthermore, by leveraging LLMs to convert abundant audio label data into audio-text datasets, the issue of time data scarcity is addressed.

InstructVideo is a method for training text-to-video diffusion models using reward fine-tuning guided by human feedback. It employs an editing-based reward fine-tuning approach, which reduces fine-tuning cost while enhancing efficiency. Leveraging pre-established image reward models, it provides reward signals through segment-wise sparse sampling and temporal decay rewards, significantly improving the visual quality of generated videos. InstructVideo not only enhances the visual quality of generated videos but also maintains strong generalization capabilities. For more information, please visit the official website.

FreeU is a method that can significantly improve the sampling quality of diffusion models without any additional cost: no training, no extra parameters, no increased memory or sampling time. It achieves this by reweighting the contributions of U-Net's skip connections and main branch feature maps, leveraging the advantages of both components of the U-Net architecture to enhance generation quality. Experiments on image and video generation tasks demonstrate that FreeU can be easily integrated into existing diffusion models such as Stable Diffusion, DreamBooth, ModelScope, Rerender, and ReVersion with just a few lines of code to improve generation quality.