# BERT

ModernBERT-large is a state-of-the-art bidirectional encoder Transformer model (BERT style) pre-trained on 2 trillion tokens of English and code data, with a native context length of up to 8192 tokens. This model incorporates the latest architectural improvements such as Rotary Positional Embeddings (RoPE) for long-context support, local-global alternating attention for enhanced efficiency with long inputs, and padding-free and Flash Attention for improved inference speed. ModernBERT-long is suitable for tasks involving the handling of long documents, such as retrieval, classification, and semantic search within large corpora. The training data primarily consists of English and code, which may result in lower performance with other languages.

ModernBERT is a next-generation encoder model co-released by Answer.AI and LightOn. It is a comprehensive upgrade of the BERT model, offering longer sequence lengths, better downstream performance, and faster processing speeds. ModernBERT leverages the latest Transformer architecture improvements, with a strong focus on efficiency, trained on modern data scales and sources. As an encoder model, it excels in a variety of natural language processing tasks, particularly in code search and understanding. It is available in two model sizes: a base version (139M parameters) and a large version (395M parameters), suitable for diverse application needs.

ModernBERT-base is a modern bidirectional encoder Transformer model pretrained on 2 trillion English and code samples, natively supporting up to 8192 tokens of context. The model incorporates cutting-edge architectural improvements such as Rotary Positional Embeddings (RoPE), Local-Global Alternating Attention, and Unpadding, showing exceptional performance on long-text processing tasks. It is ideal for processing long documents for tasks such as retrieval, classification, and semantic search within large corpuses. Since the training data is primarily in English and code, its performance may be reduced when handling other languages.

kelindar/search is a Go language library that provides functionality for embedded vector search and semantic embeddings, built on llama.cpp. This library is particularly suitable for small to medium-sized projects that require powerful semantic search capabilities while maintaining a simple and efficient implementation. It supports GGUF BERT models, allowing users to leverage advanced embedding techniques without delving into the complexities of traditional search systems. The library also offers GPU acceleration for fast computations on supported hardware. If your dataset contains fewer than 100,000 entries, this library can be easily integrated into your Go application to implement semantic search features.