# End-to-End Model

OpenEMMA is an open-source project that replicates Waymo's EMMA model, providing an end-to-end framework for motion planning of autonomous vehicles. This model utilizes pre-trained Vision-Language Models (VLMs) such as GPT-4 and LLaVA to integrate text and forward-facing camera inputs, achieving precise predictions of future path points and providing decision rationale. OpenEMMA aims to provide accessible tools for researchers and developers to advance research and applications in autonomous driving.

LLaMA-Omni is a low-latency, high-quality end-to-end speech interaction model built on the Llama-3.1-8B-Instruct architecture, aimed at achieving speech capabilities comparable to GPT-4o. The model supports low-latency speech interactions, generating text and speech responses simultaneously. It completed training in less than 3 days using only 4 GPUs, demonstrating its efficient training capabilities.

OptiSpeech is an efficient, lightweight, and fast text-to-speech model specifically designed for device-side text-to-speech conversion. Leveraging advanced deep learning techniques, it converts text into naturally sounding speech, making it suitable for applications that require speech synthesis on mobile devices or embedded systems. The development of OptiSpeech was significantly accelerated by GPU resources provided by Pneuma Solutions.

CyberHost is an end-to-end audio-driven human animation framework that employs a region codebook attention mechanism to generate complete hand integrity, identity consistency, and natural movement. The model utilizes a dual U-Net architecture as its foundational structure and implements motion frame strategies for temporal continuity, establishing a baseline for audio-driven human animation. CyberHost enhances the quality of synthesized results through a series of human-centered training strategies, including body movement maps, hand clarity scoring, reference features for pose alignment, and local enhancement supervision. It is the first audio-driven human diffusion model capable of zero-shot video generation within the human domain.

SpeechGPT2 is an end-to-end speech dialogue language model developed by the School of Computer Science at Fudan University. It can perceive and express emotions while providing appropriate voice responses in various styles based on context and human instructions. The model uses ultra-low bitrate speech codec (750bps) to simulate semantic and acoustic information and is initialized via a Multi-Input Multi-Output Language Model (MIMO-LM). Currently, SpeechGPT2 is a turn-based dialogue system, with development underway for a full-duplex real-time version that has shown promising progress. Despite limitations in computational and data resources, SpeechGPT2 has room for improvement regarding noise robustness in speech understanding and stability in speech generation quality, with plans for future open-source technical reports, code, and model weights.