🚀 Introducing our new tech report: Muon is Scalable for LLM Training We found that Muon optimizer can be scaled up using the follow techniques: • Adding weight decay • Carefully adjusting the per-parameter update scale ✨ Highlights: • ~2x computational efficiency vs AdamW…
🚀 Introducing DeepSeek-V3! Biggest leap forward yet: ⚡ 60 tokens/second (3x faster than V2!) 💪 Enhanced capabilities 🛠 API compatibility intact 🌍 Fully open-source models & papers 🐋 1/n
Highly recommend this user-friendly project if you start with LM pretraining and want to build your own model/optimizer. The repo is easy to understand, easy to edit and easy to implement new ideas with minimum workloads. Well done Keller! Looking forward to your records on VIT:)
I enjoy getting NanoGPT training speed records. I’m also interested in making my formulation of NanoGPT speedrunning an accessible benchmark on which other people find it easy to try new ideas. To that end, I have tried to keep the code of the current record short, and minimize…
Fixed a bug which caused all training losses to diverge for large gradient accumulation sizes. 1. First reported by @bnjmn_marie, GA is supposed to be mathematically equivalent to full batch training, but losses did not match. 2. We reproed the issue, and further investigation…
Harm's Law of Smol Models (HLSM) tells us how much we need to scale up the data size (k_D) as we scale down the model size (k_N), if we wish to preserve the loss of a Chinchilla-optimal model. harmdevries.com/post/model-siz…
Afraid of sum-of-squares (SOS) relaxations? Read this new blog post for a smooth ride in the Fourier domain. francisbach.com/sums-of-square…
United States الاتجاهات
- 1. Black Friday 347K posts
- 2. #SkylineSweeps N/A
- 3. #FanCashDropPromotion 1,040 posts
- 4. #releafcannabis N/A
- 5. Good Friday 57.6K posts
- 6. mainz biomed n.v. N/A
- 7. WHO DEY 13.2K posts
- 8. #AVenezuelaNoLaTocaNadie 2,002 posts
- 9. #FridayVibes 3,950 posts
- 10. CONGRATULATIONS JIN 54.9K posts
- 11. Clark Lea 1,038 posts
- 12. Black Cats 2,964 posts
- 13. $SMX 8,581 posts
- 14. CONGRATULATIONS J-HOPE 45.9K posts
- 15. Mr. President 22K posts
- 16. Andriy Yermak 9,274 posts
- 17. GreetEat Corp. N/A
- 18. Cyber Monday 5,905 posts
- 19. Victory Friday N/A
- 20. TODAY ONLY 65.6K posts
Something went wrong.
Something went wrong.