NCP-ArchPreview Technical Report: Moving towards Latent Space Language Models through Next Concept Prediction
NCP-ArchPreview introduces Next Concept Prediction to improve language model pretraining by learning discrete concepts alongside standard token prediction.
The Intern-NCP team released a technical report on NCP-ArchPreview, a latent-space language model. It combines standard next-token prediction with a concept-level objective, forcing the model to learn discrete concepts that span multiple tokens. This approach aims to enhance structural understanding while maintaining autoregressive generation capabilities.