🚀 Introducing our new tech report: Muon is Scalable for LLM Training We found that Muon optimizer can be scaled up using the follow techniques: • Adding weight decay • Carefully adjusting the per-parameter update scale ✨ Highlights: • ~2x computational efficiency vs AdamW…
🚀 Introducing DeepSeek-V3! Biggest leap forward yet: ⚡ 60 tokens/second (3x faster than V2!) 💪 Enhanced capabilities 🛠 API compatibility intact 🌍 Fully open-source models & papers 🐋 1/n
Highly recommend this user-friendly project if you start with LM pretraining and want to build your own model/optimizer. The repo is easy to understand, easy to edit and easy to implement new ideas with minimum workloads. Well done Keller! Looking forward to your records on VIT:)
I enjoy getting NanoGPT training speed records. I’m also interested in making my formulation of NanoGPT speedrunning an accessible benchmark on which other people find it easy to try new ideas. To that end, I have tried to keep the code of the current record short, and minimize…
Fixed a bug which caused all training losses to diverge for large gradient accumulation sizes. 1. First reported by @bnjmn_marie, GA is supposed to be mathematically equivalent to full batch training, but losses did not match. 2. We reproed the issue, and further investigation…
Harm's Law of Smol Models (HLSM) tells us how much we need to scale up the data size (k_D) as we scale down the model size (k_N), if we wish to preserve the loss of a Chinchilla-optimal model. harmdevries.com/post/model-siz…
Afraid of sum-of-squares (SOS) relaxations? Read this new blog post for a smooth ride in the Fourier domain. francisbach.com/sums-of-square…
United States Trends
- 1. Michigan 105K posts
- 2. Ryan Day 4,175 posts
- 3. Bo Jackson 1,022 posts
- 4. Buckeyes 7,256 posts
- 5. #GoBlue 7,122 posts
- 6. #TheGame 3,099 posts
- 7. Barham 1,215 posts
- 8. Florida 105K posts
- 9. Texas 186K posts
- 10. Donaldson 1,325 posts
- 11. #SmallBusinessSaturday 2,862 posts
- 12. Julian Sayin 1,289 posts
- 13. Kentucky 17.3K posts
- 14. Gus Johnson N/A
- 15. #GoBucks 4,705 posts
- 16. Ann Arbor 3,389 posts
- 17. Leeds 32.9K posts
- 18. Stoops 1,055 posts
- 19. Grade 3 3,236 posts
- 20. Foden 21.1K posts
Something went wrong.
Something went wrong.