🚀 Introducing our new tech report: Muon is Scalable for LLM Training We found that Muon optimizer can be scaled up using the follow techniques: • Adding weight decay • Carefully adjusting the per-parameter update scale ✨ Highlights: • ~2x computational efficiency vs AdamW…
🚀 Introducing DeepSeek-V3! Biggest leap forward yet: ⚡ 60 tokens/second (3x faster than V2!) 💪 Enhanced capabilities 🛠 API compatibility intact 🌍 Fully open-source models & papers 🐋 1/n
Highly recommend this user-friendly project if you start with LM pretraining and want to build your own model/optimizer. The repo is easy to understand, easy to edit and easy to implement new ideas with minimum workloads. Well done Keller! Looking forward to your records on VIT:)
I enjoy getting NanoGPT training speed records. I’m also interested in making my formulation of NanoGPT speedrunning an accessible benchmark on which other people find it easy to try new ideas. To that end, I have tried to keep the code of the current record short, and minimize…
Fixed a bug which caused all training losses to diverge for large gradient accumulation sizes. 1. First reported by @bnjmn_marie, GA is supposed to be mathematically equivalent to full batch training, but losses did not match. 2. We reproed the issue, and further investigation…
Harm's Law of Smol Models (HLSM) tells us how much we need to scale up the data size (k_D) as we scale down the model size (k_N), if we wish to preserve the loss of a Chinchilla-optimal model. harmdevries.com/post/model-siz…
Afraid of sum-of-squares (SOS) relaxations? Read this new blog post for a smooth ride in the Fourier domain. francisbach.com/sums-of-square…
United States 趨勢
- 1. Lamar 26K posts
- 2. Thanksgiving 1.8M posts
- 3. Ravens 33.3K posts
- 4. Chiefs 96.2K posts
- 5. Tanner Hudson N/A
- 6. Zac Taylor 2,053 posts
- 7. Cam Boozer 1,099 posts
- 8. Mahomes 30.3K posts
- 9. Joe Burrow 10.3K posts
- 10. Pickens 30.5K posts
- 11. Sarah Beckstrom 159K posts
- 12. Derrick Henry 3,061 posts
- 13. Tinsley 1,177 posts
- 14. #AEWCollision 8,431 posts
- 15. Duke 18.5K posts
- 16. Caleb Foster N/A
- 17. Jason Garrett N/A
- 18. Kyle Hamilton N/A
- 19. #CINvsBAL 1,388 posts
- 20. Chase Brown 2,090 posts
Something went wrong.
Something went wrong.