VKORE-325M: A 325M LLM Pretrained From Zero on a Rebellions ATOM-Max NPU, Equal to a B200 Twin
aimachinelearningnpullm
TL;DR
VIDRAFT has pretrained a 325.6M parameter language model, VKORE-325M-NPU-Base, from random initialization on the Rebellions ATOM-Max, a commercially available inference NPU released in 2025. The same model trained on a B200 GPU serves as the reference. On 1,024 windows of text that neither model saw in training, the NPU model scores 0.7993 and the GPU model 0.7994. Both weights and the evaluation script are public.
What was announced?
A base language model pretrained entirely on hardware sold for inference. The run used 5 billion byte tokens of FineWeb-Edu, took 76,292 steps over about four days, and finished without divergence. No pretrained weights were used.
| Item | Value |
|---|---|
| Model | VKORE-325M-NPU-Base, 325.6M parameters, 20 layers |
| Hardware | Rebellions ATOM-Max NPU (2025, inference-only) |
| Data | FineWeb-Edu, 5.0B byte tokens |
| Run | 76,292 steps, about four days |
| Reference | identical model on a B200 GPU |
How was GPU quality measured?
By holding everything fixed except the hardware: the same initialization, data, batch and learning-rate schedule on both machines.
The training curves overlap for the whole run.
What do the unseen-text results show?
| Model | Loss, unseen text | Bits per byte |
|---|---|---|
| Rebellions ATOM-Max NPU | 0.7993 | 1.1531 |
| B200 GPU reference | 0.7994 | 1.1533 |
Window by window, the difference is -0.0001 with a standard error of 0.0007, and the NPU model is lower on 50.1% of windows.
How does it extend VKORE-46M?
The model is 7 times larger and trained on 222 times more data than VKORE-46M-NPU. The stability technique that keeps long runs on track was developed in house and is under patent application.
Does buying the NPU enable training?
No. The ATOM-Max and its standard software are built for inference. Training depends on VIDRAFT's own training stack and stability technique.
FAQ
Which NPU was used? The Rebellions ATOM-Max, a commercial inference NPU released in 2025.
Is the NPU model as good as the GPU model? On unseen FineWeb-Edu text, yes: 0.7993 vs 0.7994.
Can the result be reproduced? Yes. Both weights and evaluate_unseen.py are in the model repository.
Is it a chat model? No. It is a research-scale base model released as a hardware result.
Sources
- Technical report: https://huggingface.co/blog/FINAL-Bench/pretraining-a-325m-language-model-from-zero-on-an
- Models: https://huggingface.co/FINAL-Bench/VKORE-325M-NPU-Base