🚀 Introducing our new tech report: Muon is Scalable for LLM Training We found that Muon optimizer can be scaled up using the follow techniques: • Adding weight decay • Carefully adjusting the per-parameter update scale ✨ Highlights: • ~2x computational efficiency vs AdamW…
🚀 Introducing DeepSeek-V3! Biggest leap forward yet: ⚡ 60 tokens/second (3x faster than V2!) 💪 Enhanced capabilities 🛠 API compatibility intact 🌍 Fully open-source models & papers 🐋 1/n
Highly recommend this user-friendly project if you start with LM pretraining and want to build your own model/optimizer. The repo is easy to understand, easy to edit and easy to implement new ideas with minimum workloads. Well done Keller! Looking forward to your records on VIT:)
I enjoy getting NanoGPT training speed records. I’m also interested in making my formulation of NanoGPT speedrunning an accessible benchmark on which other people find it easy to try new ideas. To that end, I have tried to keep the code of the current record short, and minimize…
Fixed a bug which caused all training losses to diverge for large gradient accumulation sizes. 1. First reported by @bnjmn_marie, GA is supposed to be mathematically equivalent to full batch training, but losses did not match. 2. We reproed the issue, and further investigation…
Harm's Law of Smol Models (HLSM) tells us how much we need to scale up the data size (k_D) as we scale down the model size (k_N), if we wish to preserve the loss of a Chinchilla-optimal model. harmdevries.com/post/model-siz…
Afraid of sum-of-squares (SOS) relaxations? Read this new blog post for a smooth ride in the Fourier domain. francisbach.com/sums-of-square…
United States 트렌드
- 1. Romero 19.3K posts
- 2. #twitchrecap 11.4K posts
- 3. Penn State 17.8K posts
- 4. #TADCFriend 1,216 posts
- 5. Slay 19.3K posts
- 6. Fulham 42K posts
- 7. #GivingTuesday 30.6K posts
- 8. Lewandowski 25.5K posts
- 9. Pat Kraft 1,847 posts
- 10. Larry 58.8K posts
- 11. Zion 8,992 posts
- 12. Pedri 47.1K posts
- 13. Adam Thielen 3,351 posts
- 14. Rashford 9,230 posts
- 15. Olmo 21.2K posts
- 16. Bentancur 2,612 posts
- 17. Paul Dano N/A
- 18. Newcastle 30.8K posts
- 19. Trump Accounts 26.4K posts
- 20. #AppleMusicReplay 11.9K posts
Something went wrong.
Something went wrong.