# Multimodal Learning

OmniTalker is a unified framework proposed by Alibaba's Tongyi Lab with the aim of generating audio and video in real time to enhance human-computer interaction experiences. Its innovation lies in solving common issues in traditional text-to-speech and speech-driven video generation methods, such as out-of-sync audio-video, inconsistent styles, and system complexity. OmniTalker adopts a dual-branch diffusion transformer architecture, achieving high-fidelity audio-video outputs while maintaining efficiency. Its real-time inference speed reaches 25 frames per second, making it suitable for various interactive video chat applications and enhancing user experiences.

DeepSeek-VL2 is a series of advanced large-scale mixture of experts (MoE) visual language models, significantly improved compared to its predecessor DeepSeek-VL. This model series demonstrates exceptional capabilities across various tasks, including visual question answering, optical character recognition, document/table/chart understanding, and visual localization. Comprising three variants: DeepSeek-VL2-Tiny, DeepSeek-VL2-Small, and DeepSeek-VL2, with 1 billion, 2.8 billion, and 4.5 billion active parameters respectively, DeepSeek-VL2 achieves competitive or state-of-the-art performance against existing dense and MoE-based open-source models, even with a similar or fewer number of active parameters.

MMAudio is a multimodal joint training technology aimed at high-quality video-to-audio synthesis. This technology can generate synchronized audio from video and text inputs, suitable for various applications such as film production and game development. Its significance lies in improving the efficiency and quality of audio generation, making it ideal for creators and developers in need of audio synthesis.

InternViT-300M-448px-V2_5 is an enhanced version of InternViT-300M-448px, utilizing incremental learning with ViT and NTP loss (Stage 1.5) to enhance the visual encoder's capability to extract visual features. It is particularly effective in underrepresented domains in large-scale network datasets, such as multilingual OCR data and mathematical graphs. This model is part of the InternViT 2.5 series and retains the same 'ViT-MLP-LLM' architecture as its predecessors while integrating incrementally pre-trained InternViT with various pre-trained LLMs, such as InternLM 2.5 and Qwen 2.5, using randomly initialized MLP projectors.

Florence-VL is a visual language model that enhances the processing capabilities of visual and language information by introducing generative visual encoders and deep breadth fusion technology. The significance of this technology lies in its ability to improve machines' understanding of images and text, achieving better performance in multimodal tasks. Florence-VL is developed based on the LLaVA project, providing code for pre-training and fine-tuning, model checkpoints, and demonstrations.

LLaVA-o1 is a visual language model developed by the Yuan Group at Peking University, capable of spontaneous and systematic reasoning, similar to GPT-01. This model has outperformed others in six challenging multimodal benchmarks, including Gemini-1.5-pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct. LLaVA-o1 demonstrates its unique advantages in visual language modeling by solving problems through step-by-step reasoning.

PPLLaVA is an efficient large-scale video language model that combines fine-grained visual prompt alignment, a convolutional-style pooling mechanism for visual token compression based on user instructions, and CLIP context extension. This model has achieved new state-of-the-art results on datasets such as VideoMME, MVBench, VideoChatGPT Bench, and VideoQA Bench, using only 1024 visual tokens, achieving an 8-fold improvement in throughput.

Agent S is an open agent framework designed for autonomous interaction with computers through a graphical user interface (GUI). It transforms human-computer interaction by automating complex, multi-step tasks. The framework introduces an experience-enhanced hierarchical planning approach that leverages online network knowledge and narrative memory, extracting high-level experiences from past interactions to decompose complex tasks into manageable subtasks and provide step-by-step guidance using situational memory. Agent S continuously optimizes its actions and learns from experience, achieving adaptive and effective task planning. In the OSWorld benchmark, Agent S outperformed the baseline with a success rate increase of 9.37% (an 83.6% relative improvement), demonstrating extensive versatility in the WindowsAgentArena benchmark.

FakeShield is a multimodal framework designed to address two primary challenges in the field of Image Forensics Detection and Localization (IFDL): the black-box nature of detection mechanisms and the limited generalization across different tampering methods. By leveraging GPT-4o to enhance existing IFDL datasets, FakeShield has created a Multimodal Tampering Description Dataset (MMTD-Set) to train its tampering analysis capabilities. The framework includes domain label-guided interpretable detection modules (DTE-FDM) and localization modules (MFLM) that can interpret various types of tampering detection and guide localization through detailed textual descriptions. FakeShield outperforms other methods in detection accuracy and F1 scores, providing a superior and interpretable solution.

LLaVA-Video is a large multimodal model (LMM) focused on video instruction tuning, addressing the challenge of acquiring high-quality raw data from the internet by creating a high-quality synthetic dataset, LLaVA-Video-178K. This dataset includes detailed video descriptions, open-ended questions, and multiple-choice questions, aimed at enhancing the understanding and reasoning capabilities of video language models. The LLaVA-Video model has demonstrated outstanding performance across various video benchmarks, validating the effectiveness of its dataset.

NVLM 1.0 is a series of advanced multimodal large language models (LLMs) that have achieved state-of-the-art results on visual-language tasks, comparable to leading proprietary and open-access models. Notably, NVLM 1.0 surpasses its LLM backbone model in text performance following multimodal training. We have made the model weights and code open-source for the community.

LongLLaVA is a multimodal large language model that extends efficiently to 1,000 images through a hybrid architecture, aimed at enhancing image processing and understanding capabilities. The model achieves effective learning and inference on large-scale image data through innovative architecture design, making it significant for fields like image recognition, classification, and analysis.

EAGLE is a series of high-resolution, vision-centric multimodal large language models (LLMs) designed to enhance the perception capabilities of multimodal LLMs through a combination of visual encoders and varied input resolutions. The model features a 'CLIP+X' fusion based on channel connections, suitable for visual experts trained on different architectures (ViT/ConvNets) and domains (detection/segmentation/OCR/SSL). The EAGLE model family supports input resolutions over 1K and excels in multimodal LLM benchmarks, particularly in resolution-sensitive tasks such as optical character recognition and document understanding.

SlowFast-LLaVA is a zero-training multimodal large language model designed for video understanding and reasoning. It achieves performance comparable to or even better than state-of-the-art video large language models across various video question-answering tasks and benchmarks, without the need for fine-tuning on any data.

Llama3-s v0.2 is a multimodal checkpoint developed by Homebrew Computer Company, focusing on improving speech comprehension capabilities. This model enhances its performance through early integration of semantic tagging and community feedback to streamline its structure, improve compression efficiency, and ensure consistent feature extraction from speech. Llama3-s v0.2 demonstrates stable performance across multiple speech understanding benchmarks and offers a live demo for users to experience its functionalities firsthand. Although the model is still in early development and has certain limitations—such as sensitivity to audio compression and a maximum handling time of 10 seconds for audio—the team intends to address these issues in future updates.

llama3-s is an open and ongoing research experiment aiming to extend large language models (LLMs) based on text to have native 'hearing' abilities. The project draws inspiration from Meta's Chameleon paper, focusing on token passability by incorporating audio tokens into the LLM vocabulary, potentially expanding to various input types in the future. As an open-source scientific experiment, both the codebase and datasets are publicly available.

MAVIS is a mathematical visual instruction tuning model designed for multimodal large language models (MLLMs). It enhances MLLMs' capabilities in visual mathematical problem-solving by improving visual encoding of mathematical graphs, graph-language alignment, and mathematical reasoning skills. The model includes two newly curated datasets, a mathematical visual encoder, and a mathematical MLLM, achieving leading performance in the MathVerse benchmark test through a three-phase training paradigm.

LongVA is a long context transformer model capable of processing over 2000 frames or 200K visual tokens. It achieves leading performance in Video-MME among 7B models. The model is tested on CUDA 11.8 and A100-SXM-80G and can be quickly deployed and used through the Hugging Face platform.

MG-LLaVA is a machine learning language model (MLLM) designed to enhance the visual processing capabilities of models. It achieves this by incorporating a multi-granularity visual pipeline, encompassing low-resolution, high-resolution, and object-centric features. An additional high-resolution visual encoder is introduced to capture finer details, and a Conv-Gate fusion network is used to integrate these high-resolution features with the base visual features. Furthermore, object-level features derived from offline detector bounding boxes are integrated to further refine the model's object recognition abilities. Trained via instruction tuning on publicly available multimodal data, MG-LLaVA exhibits exceptional perceptual skills.

Stable Diffusion 3, developed by Stability AI, is the latest text-to-image generation model with significantly improved image fidelity, multi-subject handling, and text alignment capabilities. Utilizing the Multimodal Diffusion Transformer (MMDiT) architecture, it provides separate image and language representations, supporting API, download, and online platform access for diverse application scenarios.

emo-visual-data is a publicly available emoji visual annotation dataset. It collects 5329 emojis through visual annotation completed using the glm-4v and step-free-api projects. This dataset can be used to train and test multimodal large models and is crucial for understanding the relationship between image content and textual descriptions.

CuMo is an extension architecture for multimodal large language models (LLMs). It enhances model scalability by incorporating sparse Top-K gated expert-mixing (MoE) blocks within both the visual encoder and MLP connector, while adding virtually no activation parameters during inference. CuMo pre-trains MLP blocks and initializes experts within the MoE blocks, utilizing auxiliary loss during the visual instruction fine-tuning stage to ensure balanced expert loading. CuMo outperforms other similar models on various VQA and visual instruction following benchmarks, trained entirely on open-source datasets.

Bunny is a series of lightweight yet powerful multimodal models, providing various plug-and-play view encoders and language backbone networks. By selecting from a broader and more carefully curated dataset, we have constructed richer training data to compensate for the reduced model size. The Bunny-v1.0-3B model outperforms similar-sized and even larger MLLMs (7B) in performance and matches the levels of a 13B model.

llava-llama-3-8b-v1_1 is an optimized LLaVA model by XTuner, based on meta-llama/Meta-Llama-3-8B-Instruct and CLIP-ViT-Large-patch14-336. It has been fine-tuned with ShareGPT4V-PT and InternVL-SFT. Designed for the combination of image and text processing, the model features strong multimodal learning capabilities and is suitable for various downstream deployment and evaluation toolkits.

MyGO is a tool for multimodal knowledge graph completion. It processes discrete modal information as fine-grained labels to enhance completion accuracy. MyGO utilizes the transformers library to embed text labels and trains and evaluates on multimodal datasets. It supports custom datasets and provides training scripts for replicating experimental results.

Stable Diffusion 3 is an advanced text-to-image generation system that rivals or surpasses top-tier systems like DALL-E 3 and Midjourney v6 in terms of layout and prompt following. This system employs a novel Multimodal Diffusion Transformer (MMDiT) architecture, utilizing different weight sets to enhance the representation of both images and language, thereby improving text understanding and spelling ability. The Stable Diffusion 3 API is now available on the Stability AI developer platform, offering fast and reliable API services in collaboration with Fireworks AI. It also promises to release model weights for self-hosting through Stability AI membership in the near future.

The MATHVERSE project aims to evaluate the ability of multimodal large language models to handle and understand visual math problems, particularly in parsing and comprehending graphical information within problems.