# Virtual Reality

Phantom is an advanced video generation technology that achieves subject-consistent video generation through cross-modal alignment. It can generate vivid video content from single or multiple reference images while strictly preserving the identity features of the subject. This technology has significant application value in areas such as content creation, virtual reality, and advertising, providing creators with efficient and creative video generation solutions. Key advantages of Phantom include high subject consistency, rich video details, and powerful multimodal interaction capabilities.

Pippo, developed in collaboration between Meta Reality Labs and various universities, is a generative model capable of producing high-resolution, multi-view videos from a single ordinary photograph. Its core advantage lies in generating high-quality 1K resolution videos without any additional input (such as parameterized models or camera parameters). Based on a multi-view diffusion transformer architecture, it has broad application prospects in areas like virtual reality and film production. Pippo's code is open-source, but pre-trained weights are not included; users need to train the model themselves.

GameFactory is an innovative general-purpose world model that focuses on learning from a limited amount of Minecraft gameplay video data and leverages prior knowledge from a pre-trained video diffusion model to generate new game content. Its core advantage lies in its open-domain generative ability, allowing it to create diverse game scenes and interactive experiences based on user-input text prompts and operational commands. It not only demonstrates strong scene generation capabilities but also achieves high-quality interactive video generation through a multi-stage training strategy and plug-in action control modules. This technology holds great promise in fields such as game development, virtual reality, and creative content generation. The pricing and commercial positioning are currently undefined.

SCENIC is a text-conditioned scene interaction model that adapts to complex environments with varying terrains, supporting user-specified semantic control through natural language. The model navigates 3D scenes using user-defined trajectories as sub-goals and textual prompts. SCENIC employs hierarchical reasoning methods in scene understanding, achieving seamless transitions between different motion styles through frame alignment of movement and text. This technology is significant as it generates character navigation movements that comply with real-world physics and user instructions, playing a crucial role in virtual reality, augmented reality, and game development.

GenEx is an AI model capable of creating a fully explorable 360° 3D world from a single image. Users can interactively explore this generated world. GenEx advances embodied AI in imaginative spaces and has the potential to extend these capabilities into real-world exploration.

SOLAMI is an end-to-end Social Visual-Language-Action (VLA) modeling framework for immersive interaction with 3D autonomous characters. The framework constructs 3D autonomous characters by integrating three main components: a social VLA architecture, interactive multimodal data, and an immersive VR interface. Key benefits of SOLAMI include more accurate and natural character responses (including voice and actions) that align with user expectations, resulting in lower latency. The significance of this technology lies in its ability to endow 3D autonomous characters with human-like social intelligence, enabling them to perceive, understand, and interact with humans, which remains an open foundational question in the field of artificial intelligence.

CAT4D is a cutting-edge technology that generates 4D scenes from monocular videos using multi-view video diffusion models. It transforms input monocular videos into multi-perspective video and reconstructs dynamic 3D scenes. The significance of this technology lies in its ability to extract and reconstruct complete spatial and temporal information from single-view video footage, providing robust technical support for virtual reality, augmented reality, and 3D modeling. Background information indicates that CAT4D is a collaborative project developed by researchers from Google DeepMind, Columbia University, and UC San Diego, representing a successful case of turning advanced research outcomes into practical applications.

The Matrix is a pioneering project aimed at creating a fully immersive and interactive digital universe through AI technology, blurring the lines between reality and illusion. This project transcends existing video model limits by providing frame-level precision in user interaction, AAA-level visuals, and infinite generation capabilities, offering users endless exploration experiences. The Matrix is co-developed by Alibaba Group, The University of Hong Kong, The University of Waterloo, and the Vector Institute, representing a new pinnacle in world simulation technology.

TANGO is a co-speech gesture video reproduction technology based on hierarchical audio-motion embedding and diffusion interpolation. It utilizes advanced artificial intelligence algorithms to convert voice signals into corresponding gesture animations, enabling the natural reproduction of gestures in videos. This technology has broad application prospects in video production, virtual reality, and augmented reality, significantly enhancing the interactivity and realism of video content. TANGO was jointly developed by the University of Tokyo and CyberAgent AI Lab, representing the cutting edge of artificial intelligence in gesture recognition and motion generation.

The Meta Quest 3S is a mixed reality headset that offers an immersive gaming experience along with fitness and entertainment features. It supports applications such as Facebook, Instagram, and WhatsApp and features the 'Hey Meta' wake word to invoke Meta AI. With high-resolution display, lightweight design, innovative controller design, and enhanced haptic feedback, the Meta Quest 3S is designed to deliver unprecedented virtual experiences while ensuring comfort in wear and high-performance graphics processing.

GVHMR is an innovative human motion recovery technology that uses a gravity perspective coordinate system to address the challenge of recovering human motion from monocular videos. This technology reduces ambiguity in learning image-pose mappings and avoids cumulative errors in consecutive images associated with autoregressive methods. GVHMR has shown exceptional performance in field benchmarks, surpassing existing state-of-the-art techniques in both accuracy and speed. Additionally, its training process and model weights are publicly accessible, providing high research and practical value.

World Labs is a company focused on spatial intelligence, dedicated to constructing large world models (Large World Models) to perceive, generate, and interact with the 3D world. The company was founded by renowned scientists, professors, scholars, and industry leaders in the AI field, including Professor Fei-Fei Li from Stanford University and Professor Justin Johnson from the University of Michigan. They have advanced 3D scene reconstruction and novel perspective synthesis through innovative techniques like Neural Radiance Fields (NeRF). World Labs is supported by notable investors such as Marc Benioff and Jim Breyer, and its technology has significant application value and commercial potential in the AI domain.

OmniRe is a comprehensive method for efficiently reconstructing high-fidelity dynamic urban scenes from device logs. This technology achieves a complete reconstruction of different objects in the scene by constructing a dynamic neural scene graph based on Gaussian representations and building multiple local canonical spaces to simulate various dynamic actors, including vehicles, pedestrians, and cyclists. OmniRe enables comprehensive reconstruction of different objects present in a scene, allowing for real-time simulation of reconstructed scenes involving all participants. Extensive evaluations on the Waymo dataset show that OmniRe significantly outperforms previous state-of-the-art methods both quantitatively and qualitatively.

This is an open-source project designed for remote control of the humanoid robot Unitree H1_2. Utilizing Apple Vision Pro technology, it enables users to control robots through a virtual reality environment. The project has been tested on Ubuntu 20.04 and Ubuntu 22.04 and provides detailed installation and configuration guidance. The main advantages of this technology include offering an immersive remote control experience and supporting testing in a simulated environment, providing new solutions for the robotics remote control field.

ControlMM is a full-body motion generation framework equipped with plug-and-play multimodal control capabilities. It can robustly generate movements across various domains, including Text-to-Motion, Speech-to-Gesture, and Music-to-Dance. The model has significant advantages in controllability, sequence coherence, and motion realism, providing a new motion generation solution for the field of artificial intelligence.

HoloDreamer is a text-driven 3D scene generation framework capable of producing immersive and view-consistent fully enclosed 3D scenes. It consists of two fundamental modules: stylized rectangular panoramic generation and enhanced two-phase panoramic reconstruction. This framework first generates high-definition panoramic images as a complete initialization for the 3D scene, then quickly reconstructs the 3D scene using 3D Gaussian scattering (3D-GS) technology, resulting in view-consistent and fully enclosed 3D scene generation. HoloDreamer's main advantages include high visual consistency, harmony, and robust reconstruction quality and rendering.

Aiuni is a platform that delivers immersive experiences in a 3D virtual world, where users can create and explore personalized 3D models while enjoying an engaging cosmic adventure. With its innovative 3D technology, rich interactivity, and high degree of customization, Aiuni offers users a brand new space for virtual experiences.

EgoGaussian is an advanced 3D scene reconstruction and dynamic object tracking technology. It can reconstruct 3D scenes and track the movement of objects dynamically using only RGB first-person perspective input. This technology leverages the unique discrete properties of Gaussian scattering to segment dynamic interactions from the background. Through a piece-wise online learning process, it utilizes the dynamic characteristics of human activities to reconstruct the evolution of the scene in chronological order and track the movement of rigid objects. EgoGaussian outperforms previous NeRF and dynamic Gaussian methods in wild video challenges and delivers exceptional quality in reconstructed models.

WonderWorld is an innovative 3D scene expansion framework that allows users to explore and shape virtual environments based on a single input image and user-specified text. Through fast Gaussian voxel and guided diffusion depth estimation methods, it significantly reduces computing time and generates geometry-consistent expansions, resulting in 3D scene generation times of less than 10 seconds. It supports real-time user interaction and exploration. This opens up possibilities for rapidly generating and navigating immersive virtual worlds in fields like virtual reality, gaming, and creative design.

Developed by a team from Tsinghua University, Unique3D is a technology that can generate high-fidelity textured 3D mesh models from a single image. This technology has significant implications for image processing and 3D modeling, enabling users to quickly convert 2D images into 3D models, providing powerful technical support for game development, animation production, and virtual reality.

Rokoko is a sensor-based motion capture system that offers high-quality body, finger, and facial animation solutions for 3D digital creators. With its user-friendly interface and affordable price, it allows users to easily achieve realistic character animation.

Immerse is an expert-designed virtual reality language immersion learning platform. It helps adults learn new languages fluently by providing language courses and AI-assisted practice. Its main advantages include: providing an immersive language learning experience through virtual reality technology; providing personalized language exercises through AI technology; guidance from professional teachers and real-time feedback.

PhysDreamer is a method based on physics, which endows静态 3D objects with interactive dynamics by utilizing the object dynamics prior learned from video generation models. This approach allows for the simulation of realistic responses to novel interactions (such as external forces or agent operations) in the absence of real physical property data of objects. PhysDreamer promotes the development of more engaging and realistic virtual experiences through user studies to evaluate the realism of synthetic interactions.

Lixel CyberColor (LCC), an advanced technology product developed by XGRIDS, revolutionizes the creation of 3D scenes. LCC can automatically generate infinite 3D scenes with cinematic-quality effects using Multi-SLAM and Gaussian Splash technology. Its core advantage lies in its precise capture and reproduction of real-world details, bringing realistic experiences to fields like virtual reality, game development, and film production. XGRIDS, as an integrated hardware-software solution, showcases its powerful capabilities in high-precision 3D reconstruction and intelligent space computing at scales ranging from micrometers to kilometers. Utilizing the Multi-SLAM algorithm and optimized 3DGS technology, it automatically creates hyper-realistic large-scale 3D models for an immersive experience. Optimized algorithms achieve realistic rendering effects, while data compression technology reduces model size by 90%. Integrated LiDAR technology achieves centimeter-level model precision, and AI-driven dynamic object removal algorithms are provided. LCC plugins and SDKs are released for use in Unity, UE, Web, and mobile platforms, providing powerful support for 3D content."

VIGGLE is a controllable video generation tool based on the JST-1 video-3D base model. It allows any character to move according to your requirements. JST-1 is the first video-3D base model with practical physical understanding capabilities. VIGGLE's strengths lie in its powerful video generation and control capabilities, enabling it to generate videos of various actions and plots based on user needs. It is targeted at professional users such as video creators, animators, and content creators, helping them produce video content more efficiently. VIGGLE is currently in the testing phase and may release a paid subscription version in the future.

Wooorld is an immersive virtual reality exploration and social platform. Users can explore the hundreds of cities, landmarks, and natural landscapes around the globe with friends in the virtual world. Wooorld offers highly realistic and detailed 3D maps, allowing users to pan and zoom just by grabbing the map with their hands. Users can also engage in voice conversations, use avatars with facial and body motion capture, play virtual reality games, and collaborate using creative tools. This is a unique social experience.

UltrAvatar is a realistic and movable 3D avatar generation model designed to bridge the gap between virtual and real-world experiences. It utilizes Score Distillation Sampling (SDS) loss, a differentiable renderer, and text conditioning to guide the diffusion model in generating 3D avatars. Compared to existing works, UltrAvatar presents a novel approach to 3D avatar generation by enhancing geometric fidelity and offering superior physical rendering texture quality. It employs a diffusion color extraction model and a realism-guided texture diffusion model to remove unnecessary lighting effects, presenting genuine diffusion colors, enabling the generated avatars to render realistically under various lighting conditions. Our experiments have proven the effectiveness and robustness of this method, significantly outperforming existing state-of-the-art approaches.

DL3DV-10K is a large-scale real-world dataset containing over 10,000 high-quality videos. Each video is manually annotated with key scene points and complexity, and also provides camera pose, NeRF depth estimation, point clouds, and 3D meshes. The dataset can be used for general NeRF research, scene consistency tracking, visual language models, and other computer vision studies.

ZeroNVS is a tool for synthesizing zero-shot 360-degree panoramas from a single real image. It provides 3D SDS distillation code, evaluation code, and a pre-trained model. Users can utilize this tool for their own NeRF model distillation and evaluation, and experiment on various datasets. ZeroNVS boasts high-quality synthesis effects and supports customized image data. The tool is primarily used in virtual reality, augmented reality, and panoramic video production.

Genie is a research preview of Luma's 3D generation foundation model. It can generate a variety of 3D models for use in design, creation, and entertainment. Genie offers rich functionalities, including shape generation, texture painting, and animation creation. It can be applied in multiple fields such as game development, virtual reality, and film special effects. Pricing and positioning for Genie will be determined before its formal release.