%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
Liteavatar
Overview :
LiteAvatar is an audio-driven real-time 2D avatar generation model primarily designed for real-time chat scenarios. Through efficient speech recognition and viseme parameter prediction technology combined with a lightweight 2D face generation model, it achieves 30fps real-time inference using only CPU. Key advantages include efficient audio feature extraction, a lightweight model design, and mobile device-friendly support. This technology is suitable for real-time interactive virtual avatar generation scenarios such as online meetings and virtual live streaming. It was developed based on the need for real-time interaction and low hardware requirements. Currently, it is open-source and free, positioned as an efficient, low-resource-consuming real-time avatar generation solution.
Target Users :
LiteAvatar is targeted towards application developers needing real-time virtual avatar generation, virtual live streaming platforms, and businesses requiring real-time interaction. This technology is suitable for scenarios where efficient real-time interaction is desired with low hardware costs, such as online education, virtual meetings, and virtual social platforms. It helps users enhance interaction experiences and lowers technical barriers.
Use Cases
Online education platforms use this model to provide real-time virtual teacher avatars to students, enhancing interactivity.
Virtual live streaming platforms use LiteAvatar to generate real-time virtual avatars for streamers, reducing hardware costs.
Enterprise internal video conferencing systems integrate this technology to enable virtual avatar participation, enhancing privacy protection.
Features
Audio Feature Extraction: Extracts features from audio using an efficient ASR model.
Viseme Parameter Prediction: Generates lip-sync parameters synchronized with speech based on audio features.
2D Avatar Generation: Real-time rendering of lip movements, supporting lightweight deployment.
Real-time Interaction Support: Achieves 30fps real-time inference on CPU-only devices.
Open-source & Easy-to-use: Provides complete code and documentation for easy integration and expansion by developers.
How to Use
1. Prepare sample data and extract it to the specified path.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%230080ff;%20}%20%3c/style%3e%3c/defs%3e%3cpath%20class='st0'%20d='M16.2,11.1c.4.5.4,1.2,0,1.8l-4.7,7.1h-3.8l5.3-8L7.6,4h3.8l4.7,7.1Z'/%3e%3c/svg%3e)
2. Install a Python environment (preferably 3.10) and run `pip install -r requirements.txt` to install dependencies.
3. Run inference using `python lite_avatar.py --data_dir /path/to/sample_data --audio_file /path/to/audio.wav --result_dir /path/to/result`.
4. The inference result will be saved as an MP4 video file.
5. Refer to the `OpenAvatarChat` project to implement real-time interactive video chat functionality.
Featured AI Tools
Chinese Picks
Wenxin Yiyian
Wenxin Yiyian is Baidu's new generation of knowledge-enhanced large language model. It can interact with people in dialogue, answer questions, assist in creation, and help people efficiently and conveniently access information, knowledge, and inspiration. Based on the FlyingPaddle deep learning platform and Wenxin Knowledge Enhancement Large Language Model, it continuously integrates learning from massive data and large-scale knowledge, featuring knowledge enhancement, retrieval enhancement, and dialogue enhancement. We look forward to your feedback to help Wenxin Yiyian continue to improve.
Chatbot
5.4M
English Picks
Bot3 AI
Bot3 AI is your ultimate destination for AI conversational robots. Experience unprecedented levels of intelligent dialogue participation by interacting with AI characters.
Chatbot
2.7M