ModernBERT-base
M
Modernbert Base
Overview :
ModernBERT-base is a modern bidirectional encoder Transformer model pretrained on 2 trillion English and code samples, natively supporting up to 8192 tokens of context. The model incorporates cutting-edge architectural improvements such as Rotary Positional Embeddings (RoPE), Local-Global Alternating Attention, and Unpadding, showing exceptional performance on long-text processing tasks. It is ideal for processing long documents for tasks such as retrieval, classification, and semantic search within large corpuses. Since the training data is primarily in English and code, its performance may be reduced when handling other languages.
Target Users :
The target audience includes developers, data scientists, and researchers who need to handle long textual data. ModernBERT-base is particularly suited for natural language processing, code retrieval, and hybrid (text + code) semantic search scenarios due to its capabilities in processing lengthy texts and optimization for English and code data.
Total Visits: 29.7M
Top Region: US(17.94%)
Website Views : 51.1K
Use Cases
Information retrieval from large-scale documents
Semantic search within codebases to find related functions or modules
Text classification and semantic search within large corpuses
Features
Supports long-text processing for sequences up to 8192 tokens
Rotary Positional Embeddings (RoPE) for extended context support
Local-Global Alternating Attention for improved efficiency on long inputs
Unpadding and Flash Attention for optimized inference speed
Pretrained on extensive text and code datasets
Eliminates the need for token type IDs, simplifying downstream tasks
Supports Flash Attention 2 for greater efficiency
How to Use
1. Install the transformers library: Use pip to install git+https://github.com/huggingface/transformers.git.
2. Load the model and tokenizer: Use AutoTokenizer and AutoModelForMaskedLM to load the tokenizer and model from the pretrained model.
3. Prepare the input text: Feed the text to be processed into the tokenizer to obtain the input format required by the model.
4. Model inference: Pass the processed input data to the model for inference.
5. Obtain prediction results: For Masked Language Model tasks, retrieve the model's predictions for the [MASK] position.
6. Apply downstream tasks: Fine-tune ModernBERT for specific tasks like classification, retrieval, or question answering.
7. Optimize efficiency with Flash Attention 2: If supported by your GPU, install the flash-attn library and use it for enhanced inference efficiency.
AIbase
Empowering the Future, Your AI Solution Knowledge Base
© 2025AIbase