# Zero-Shot

DiffPortrait3D is a conditional difficulty model capable of synthesizing realistic 3D consistent new perspectives even from a single portrait photo taken in the wild. Specifically, given a single RGB input image, our goal is to synthesize facial details rendered from a novel camera perspective while preserving identity and facial expression. Our zero-shot approach generalizes well to various facial portraits with non-poses camera viewpoints, extreme facial expressions, and multiple artistic renderings. At its core, we utilize the generative prior of a pre-trained 2D difficulty model on large-scale image datasets as our presentation backbone, while guiding denoising through a disentangled attention control over appearance and camera posture. To this end, we first inject appearance context from the reference image into the frozen self-attention layers of UNet. Then, we manipulate the presentation view through a novel conditional control module that interprets camera posture by watching conditional images from the same view. Additionally, we insert a trainable cross-view attention module to enhance view consistency, which further enhances consistency by adopting a new 3D perception noise generation process during inference. We have demonstrated state-of-the-art results qualitatively and quantitatively on challenging wild and multi-view benchmarks.

AnyDoor is a diffusion-based image generation model that seamlessly transports target objects to new scenes at user-specified locations. Our model requires only a single training, allowing it to be easily generalized to diverse object-scene combinations without parameter adjustment for each object. To comprehensively describe a specific object, we incorporate detailed features in addition to standard identity features. These carefully designed features preserve texture details while enabling a variety of local variations (like lighting, direction, and pose), ensuring the object blends harmoniously with different environments. We also introduce a method to leverage knowledge from video datasets, where various forms of the same object can be observed along a temporal axis, thereby enhancing the model's generalization capacity and robustness. Extensive experiments validate the superiority of our approach and its immense potential in practical applications like virtual try-on and object placement.

VideoPoet is a large language model that can transform any autoregressive language model into a high-quality video generator. It can generate videos based on input textual descriptions without any visual or audio guidance. VideoPoet is capable of generating various types of videos, including text-to-video, image-to-video, video editing, stylization, and repair. It can be applied in fields such as film production, animation, advertising, and virtual reality. VideoPoet boasts high-quality video generation capabilities and can be flexibly applied to diverse scenarios.

RERENDER A VIDEO is a novel zero-shot text-guided video-to-video translation framework that applies image modeling techniques to the video domain. The framework consists of two main parts: keyframe translation and full video translation. The first part utilizes an adaptive diffusion model to generate keyframes and employs hierarchical cross-frame constraints to ensure consistency in shape, texture, and color. The second part propagates these keyframes to other frames through time-aware patch matching and frame blending. Our framework achieves global style and local texture temporal coherence at a low cost without the need for retraining or optimization. This adaptability allows our framework to leverage existing image diffusion techniques, such as customizing for specific themes using LoRA and introducing additional spatial guidance with ControlNet. Extensive experiments demonstrate the effectiveness of our proposed framework in generating high-quality and temporally coherent videos.