AIbase
EN
Home
All Categories
AI Search
Modernbert Large
ModernBERT-large
M
Modernbert Large
AI SearchCode Assistant#BERT#Transformer#Long context#Encoder model#Pre-trainedStandard PicksOpen Source
Overview :
ModernBERT-large is a state-of-the-art bidirectional encoder Transformer model (BERT style) pre-trained on 2 trillion tokens of English and code data, with a native context length of up to 8192 tokens. This model incorporates the latest architectural improvements such as Rotary Positional Embeddings (RoPE) for long-context support, local-global alternating attention for enhanced efficiency with long inputs, and padding-free and Flash Attention for improved inference speed. ModernBERT-long is suitable for tasks involving the handling of long documents, such as retrieval, classification, and semantic search within large corpora. The training data primarily consists of English and code, which may result in lower performance with other languages.
Target Users :
The target audience includes researchers and developers in the field of Natural Language Processing (NLP), particularly those who need to handle long texts and code data. The long-context processing capabilities and high efficiency of ModernBERT-large make it an ideal choice for large corpora and complex NLP tasks.
Total Visits： 29.7M
Top Region： US(17.94%)
Website Views ： 50.0K
Use Cases
Perform semantic search for text and code in large-scale corpora.
Retrieval and classification tasks for long documents.
Achieve new state-of-the-art performance in code retrieval tasks such as code search and StackQA.
Features
? Rotary Positional Embeddings (RoPE): Supports long-context processing.
? Local-global alternating attention: Increases processing efficiency for long inputs.
? Padding-free and Flash Attention: Enhances model inference efficiency.
? Long context length: Natively supports context lengths of up to 8192 tokens.
? Multi-task applicability: Suitable for retrieval, classification, and semantic search tasks for both text and code.
? High performance: Outperforms other similarly sized encoder models across multiple tasks.
? Rich pre-training data: Pre-trained on an extensive dataset consisting of 2 trillion tokens of English and code.
How to Use
1. Install the transformers library: Use pip to install the latest transformers library.
2. Load the model and tokenizer: Use AutoTokenizer and AutoModelForMaskedLM to load the tokenizer and model from the pre-trained model.
3. Process input text: Tokenize the input text to convert it into the format required by the model.
4. Model inference: Pass the processed input text to the model for inference.
5. Obtain prediction results: Extract prediction results from the model output, such as the predicted tokens for [mask].
6. Fine-tune the model: Fine-tune the model based on downstream tasks to enhance performance on specific tasks.
AIbase
Empowering the Future, Your AI Solution Knowledge Base
English简体中文繁體中文にほんご
© 2025AIbase