%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%23061b40;%20}%20.st1%20{%20fill:%20%23306af1;%20}%20.st2%20{%20fill:%20%235ce5cf;%20}%20%3c/style%3e%3c/defs%3e%3cg%3e%3cpath%20class='st0'%20d='M55,10.5h9v3h-9v9h12V7.5h-12v3ZM64,19.5h-6v-3h6v3Z'/%3e%3cpolygon%20class='st0'%20points='69%2016.5%2078%2016.5%2078%2019.5%2069%2019.5%2069%2022.5%2081%2022.5%2081%2013.5%2072%2013.5%2072%2010.5%2081%2010.5%2081%207.5%2069%207.5%2069%2016.5'/%3e%3cpolygon%20class='st0'%20points='95%2010.5%2095%207.5%2083%207.5%2083%2022.5%2095%2022.5%2095%2019.5%2086%2019.5%2086%2016.5%2095%2016.5%2095%2013.5%2086%2013.5%2086%2010.5%2095%2010.5'/%3e%3cpath%20class='st0'%20d='M40,1.5v21h11.6l1.4-1.4v-7.6h0c0,0-1.4-1.5-1.4-1.5l1.4-1.4V2.9l-1.4-1.4h-11.6ZM50,19.5h-7v-6h7v6ZM50,10.5h-7v-6h7v6Z'/%3e%3c/g%3e%3cpath%20class='st1'%20d='M23.1,24L14.7,4.8l-4.9,11.2h3.8l-1.8,4H3.3L12.1,0H2C.9,0,0,.9,0,2v20c0,1.1.9,2,2,2h21.1Z'/%3e%3cpath%20class='st2'%20d='M34,0h-16.8l10.6,24h6.2c1.1,0,2-.9,2-2V2C36,.9,35.1,0,34,0ZM32.5,20h-4V4h4v16Z'/%3e%3c/svg%3e)
3FS
Overview :
3FS is a high-performance distributed file system designed for AI training and inference workloads. Leveraging modern SSDs and RDMA networks, it provides a shared storage layer, simplifying distributed application development. Its core advantages include high performance, strong consistency, and support for various workloads, significantly improving AI development and deployment efficiency. This system is suitable for large-scale AI projects, particularly excelling in data preparation, training, and inference stages.
Target Users :
3FS is ideal for AI developers and research teams requiring high-performance storage solutions, especially those handling massive datasets and complex model training. It significantly improves data processing efficiency and reduces development and deployment costs.
Use Cases
In large-scale AI training, 3FS is used to store and quickly access training data, significantly improving training speed.
During the inference stage, 3FS's KVCache function provides efficient caching support for LLM inference, reducing computational overhead.
3FS is used in the data preparation stage to efficiently manage intermediate outputs of the data pipeline, optimizing the data processing workflow.
Features
High Performance: Combines the throughput of thousands of SSDs and the network bandwidth of hundreds of storage nodes to support large-scale parallel access.
Strong Consistency: Employs the CRAQ protocol to ensure strong data consistency, simplifying application development.
Support for Multiple Workloads: Suitable for scenarios such as data preparation, training sample loading, checkpoint saving, and inference caching.
Ease of Use: Provides standard file interfaces, eliminating the need to learn new storage APIs.
High Scalability: Supports large-scale cluster deployments to meet the needs of AI projects of various sizes.
How to Use
1. Clone the 3FS repository from GitHub: `git clone https://github.com/deepseek-ai/3fs`.
%20--%3e%3cdefs%3e%3cstyle%3e%20.st0%20{%20fill:%20%230080ff;%20}%20%3c/style%3e%3c/defs%3e%3cpath%20class='st0'%20d='M16.2,11.1c.4.5.4,1.2,0,1.8l-4.7,7.1h-3.8l5.3-8L7.6,4h3.8l4.7,7.1Z'/%3e%3c/svg%3e)
2. Initialize submodules: `cd 3fs && git submodule update --init --recursive`.
3. Install dependencies such as CMake, libuv, liblz4, etc. (see documentation for specific dependencies).
4. Build 3FS: `cmake -S . -B build -DCMAKE_BUILD_TYPE=RelWithDebInfo`, then run `cmake --build build`.
5. Deploy a test cluster, configuring storage nodes and clients according to the deployment guide.
Featured AI Tools
Pseudoeditor
PseudoEditor is a free online pseudocode editor. It features syntax highlighting and auto-completion, making it easier for you to write pseudocode. You can also use our pseudocode compiler feature to test your code. No download is required, start using it immediately.
Development & Tools
3.8M
Coze
Coze is a next-generation AI chatbot building platform that enables the rapid creation, debugging, and optimization of AI chatbot applications. Users can quickly build bots without writing code and deploy them across multiple platforms. Coze also offers a rich set of plugins that can extend the capabilities of bots, allowing them to interact with data, turn ideas into bot skills, equip bots with long-term memory, and enable bots to initiate conversations.
Development & Tools
3.8M