CogVideoX is an open-source video generation model that shares lineage with commercial models, enabling the generation of video content through textual descriptions. It represents the latest advancements in text-to-video generation technology, capable of producing high-quality videos applicable in various fields including entertainment, education, and commercial promotion.

The Video Mamba Suite is a new state-space model suite for video understanding designed to explore and assess the potential of Mamba in video modeling. It contains 14 models/modules, covering 12 video understanding tasks, demonstrating efficient performance and superiority in both video and video-language tasks.

Open-Sora-Plan is an open-source project dedicated to replicating OpenAI's Sora (T2V model) and constructing knowledge about Video-VQVAE (VideoGPT) + DiT. Initiated by the Peking University-Tuizhan AIGC Joint Laboratory, the project currently has limited resources and seeks contributions from the open-source community. The project provides training code and welcomes Pull Requests.

TrailBlazer is a diffusion-based video generation model that enables customized video creation through trajectory control. Users can guide the subject in a video using simple bounding boxes without requiring additional model training or online optimization. TrailBlazer supports the editing of spatial and temporal attention maps, as well as controlling the trajectory and appearance of the subject through keyframe bounding boxes and prompts. This model is simple and efficient to operate, generating natural and smooth video effects. TrailBlazer is suitable for a variety of user scenarios, providing an unlimited creative experience for video generation.

FlowVid is an optical flow guided video synthesis model. By utilizing the spatial and temporal information of optical flow, it achieves temporal consistency between video frames. It seamlessly integrates with existing image synthesis models to enable various modification operations, including stylization, object swapping, and local editing. FlowVid boasts fast generation speed; a 4-second, 30FPS, 512×512 resolution video can be generated in just 1.5 minutes, outperforming CoDeF (3.1x), Rerender (7.2x), and TokenFlow (10.5x) respectively. In user evaluations, FlowVid achieved a quality score of 45.7%, significantly surpassing CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%).

Traditional 3D content creation tools empower users with direct control over scene geometry, appearance, actions, and camera paths, transforming their imagination into reality. However, creating computer-generated videos remains a tedious manual process, which can be automated through emerging text-to-video diffusion models. Despite promising prospects, the lack of control in video diffusion models limits users' ability to apply their own creativity, rather than expanding it. To address this challenge, we propose a novel approach that combines the controllability of dynamic 3D meshes with the expressiveness and editability of emerging diffusion models. Our method takes animated low-fidelity rendering meshes as input and injectsground truth correspondences derived from the dynamic mesh into various stages of a pre-trained text-to-image generation model, resulting in high-quality, temporally consistent frames. We demonstrate our method on various examples, where actions are achieved through animating bound assets or altering camera paths.

SparseCtrl is developed to enhance the controllability of text-to-video generation. It enables flexible structural control by combining sparse signals with only one or a few inputs. It includes an additional conditional encoder to process these sparse signals without affecting the pre-trained text-to-video model. This method is compatible with various formats, including sketches, depth, and RGB images, providing more practical control for video generation and pushing applications like storyboarding, depth rendering, keyframe animation, and interpolation. Extensive experiments demonstrate SparseCtrl's generalization ability on both original and personalized text-to-video generators.

CogVideoX is an open-source video generation model that shares lineage with commercial models, enabling the generation of video content through textual descriptions. It represents the latest advancements in text-to-video generation technology, capable of producing high-quality videos applicable in various fields including entertainment, education, and commercial promotion.

RunwayML is a leading next-generation creative suite that offers a rich set of tools enabling users to transform any idea into reality. With its unique text-to-video generation technology, users can create videos from text descriptions directly on their mobile devices. Key advantages include: 1. Text-to-Video Generation: Users can generate videos simply by inputting text descriptions. 2. Real-Time Updates: Regularly released new features and updates ensure users always have access to the latest AI video and image tools. 3. Seamless Asset Transfer: Users can effortlessly move assets between mobile devices and computers. 4. Multiple Subscription Options: Offering standard, professional, and plans that include 1000 video generations per month.

Developed by Kuaishou Technology, Kling AI is a text-to-video generation model that can produce highly realistic videos based on text prompts. It boasts efficient video generation capabilities, generating up to 2 minutes of 30 frames-per-second video, along with advanced technologies like a 3D Spatiotemporal Joint Attention mechanism and physics-world simulation, giving it a significant competitive edge in the AI video generation field.

The Video Mamba Suite is a new state-space model suite for video understanding designed to explore and assess the potential of Mamba in video modeling. It contains 14 models/modules, covering 12 video understanding tasks, demonstrating efficient performance and superiority in both video and video-language tasks.

MiniGPT4-Video is a multimodal large model designed for video understanding. It can process temporal visual data and text data, generate captions and slogans, and is suitable for video question answering. Based on MiniGPT-v2, it incorporates the visual backbone EVA-CLIP and undergoes multi-stage training, including large-scale video-text pre-training and video question-answering fine-tuning. It achieves significant improvements on benchmarks such as MSVD, MSRVTT, TGIF, and TVQA. The pricing is currently unknown.

Open-Sora-Plan is an open-source project dedicated to replicating OpenAI's Sora (T2V model) and constructing knowledge about Video-VQVAE (VideoGPT) + DiT. Initiated by the Peking University-Tuizhan AIGC Joint Laboratory, the project currently has limited resources and seeks contributions from the open-source community. The project provides training code and welcomes Pull Requests.

TrailBlazer is a diffusion-based video generation model that enables customized video creation through trajectory control. Users can guide the subject in a video using simple bounding boxes without requiring additional model training or online optimization. TrailBlazer supports the editing of spatial and temporal attention maps, as well as controlling the trajectory and appearance of the subject through keyframe bounding boxes and prompts. This model is simple and efficient to operate, generating natural and smooth video effects. TrailBlazer is suitable for a variety of user scenarios, providing an unlimited creative experience for video generation.

FlowVid is an optical flow guided video synthesis model. By utilizing the spatial and temporal information of optical flow, it achieves temporal consistency between video frames. It seamlessly integrates with existing image synthesis models to enable various modification operations, including stylization, object swapping, and local editing. FlowVid boasts fast generation speed; a 4-second, 30FPS, 512×512 resolution video can be generated in just 1.5 minutes, outperforming CoDeF (3.1x), Rerender (7.2x), and TokenFlow (10.5x) respectively. In user evaluations, FlowVid achieved a quality score of 45.7%, significantly surpassing CoDeF (3.5%), Rerender (10.2%), and TokenFlow (40.4%).