SpacTor-T5
S
Spactor T5
Overview :
SpacTor is a new training procedure that includes (1) a mixed objective combining span corruption (SC) and replacement tag detection (RTD), and (2) a two-stage curriculum that optimizes the mixed objective in the initial \tau iterations and then transitions to standard SC loss. Experiments on various NLP tasks, using the encoder-decoder architecture (T5), show that SpacTor-T5 achieves comparable downstream performance to standard SC pre-training while reducing the pre-training iterations by 50% and the total FLOPs by 40%. Additionally, under the same computational budget, we find that SpacTor can significantly improve downstream benchmark performance.
Target Users :
A pre-trained model for natural language processing (NLP) tasks
Total Visits: 29.7M
Top Region: US(17.94%)
Website Views : 43.9K
Use Cases
Use SpacTor-T5 for text generation in NLP tasks
Leverage SpacTor-T5 for sentiment analysis
Apply SpacTor-T5 for question answering in a question answering system
Features
Mixed objective training procedure
Span corruption and replacement tag detection
Two-stage curriculum optimization
AIbase
Empowering the Future, Your AI Solution Knowledge Base
© 2025AIbase