#LLMs trained on synthetic data
---
date:
· theme: eval
· words: 107
---
Now that some very strong LLMs like DeepSeek-R1 are out as open source, a new phase is taking shape: training the next LLMs on high-quality datasets that are… synthetic.
A startup called Prime Intellect used DeepSeek-R1 to generate 1.4M reasoning examples in math, code and science, domains where answers are easy to verify. The dataset is meant for training even better reasoning models.
It’s a fascinating development:
- LLMs no longer depend solely on humans to produce high-quality training data.
- Human-annotated data is slow to produce. Needing less of it speeds up model improvement.
- This “recursive” development of AI should mean faster iteration cycles.
Like robots building robots.
