# TensorRT

FasterLivePortrait is a real-time portrait animation project based on deep learning. It achieves real-time running speed of 30+ FPS, including pre-processing and post-processing, by using TensorRT on the RTX 3090 GPU. The project also implements the conversion of the LivePortrait model to an Onnx model and achieves a >70ms/frame inference speed using onnxruntime-gpu on the RTX 3090, supporting cross-platform deployment. In addition, the project supports native gradio apps, enhancing inference speed by several times and supporting simultaneous inference for multiple faces. The code structure has been restructured, no longer relying on PyTorch, all models use onnx or tensorrt for inference.

WhisperFusion is a product powered by WhisperLive and WhisperSpeech functionalities. It enables seamless AI conversation by integrating the Mistral large language model (LLM) into the real-time speech-to-text process. Both Whisper and LLM are optimized with the TensorRT engine to maximize performance and real-time processing capabilities. WhisperSpeech utilizes torch.compile for optimization. The product is focused on delivering an ultra-low latency AI real-time conversation experience.

SwiftInfer is an LLM inference acceleration library based on Nvidia TensorRT. It significantly boosts the inference speed of LLMs in production environments by leveraging GPU acceleration. The project implements the Attention Sink mechanism proposed for streaming language models, supporting the generation of infinitely long texts. The code is concise, easy to run, and supports mainstream large language models.

Stable-Diffusion-WebUI-TensorRT is a TensorRT accelerated extension for Stable Diffusion, enabling optimal performance on NVIDIA RTX GPUs. This extension requires installation and generation of optimized engines for usage. It supports Stable Diffusion 1.5 and 2.1 versions. Please refer to the official website for installation instructions. When using, you can generate a default engine, select the TRT model, and accelerate image generation. You can generate multiple optimized engines as needed. For detailed usage instructions and FAQs, please refer to the official documentation.