Training LLMs end to end is hard. Very excited to share our new blog (book?) that cover the full pipeline: pre-training, post-training and infra. 200+ pages of what worked, what didn’t, and how to make it run reliably huggingface.co/spaces/Hugging…
Please steal my AI research ideas. This is a list of research questions and concrete experiments I would love to see done, but don't have bandwidth to get to. If you are looking to break into AI research (e.g. as an undergraduate, or a software engineer in industry), these are…
New blog post: The bug that taught me more about PyTorch than years of using it started with a simple training loss plateau... ended up digging through optimizer states, memory layouts, kernel dispatch, and finally understanding how PyTorch works!
Our latest post explores on-policy distillation, a training approach that unites the error-correcting relevance of RL with the reward density of SFT. When training it for math reasoning and as an internal chat assistant, we find that on-policy distillation can outperform other…
one of the often overlooked factors behind deepseek’s success lies in how the team built their distributed infrastructure from the ground up (despite severe gpu constraints!). their custom communication library hfreduce replaced nccl and delivered substantially higher bandwidth…
You know how if you spend the whole day sitting on the couch watching TV, you get kind of restless yet somehow also too tired to get off your butt? Like you're tired *of* doing nothing, yet you're also tired *from* doing nothing? You know what I'm talking about, the state of…
Some patterns on how iterators in Rust play well with the borrow checker - Iterator-friendly APIs, Borrow splitting and reborrowing .. Iterators prefer borrows Iterators often yield references to avoid allocation: Keep using an iterator after borrowing it Use `by_ref()` to…
I have fine-tuned over 100 different LLMs/VLMs for various use cases over the last 1–2 years, and here is my framework whenever I pick a new project or problem statement: 1. Benchmark/Evals Any problem you are solving for should have an evaluation set that you can easily…
What should I name this plugin?
can this be an extension in vscode, it just freezes your vscode randomly and plays phonk
Last night I taught nanochat d32 how to count 'r' in strawberry (or similar variations). I thought this would be a good/fun example of how to add capabilities to nanochat and I wrote up a full guide here: github.com/karpathy/nanoc… This is done via a new synthetic task…
full workflow with training & converting to hf tokenizers: github.com/thepowerfuldee…
Of ~200 books I've read, the few that stayed with me over time and I find myself often thinking back to or referring to, in ~random order: All short stories by Ted Chiang, especially Exhalation, Division By Zero, Understand, The Story of Your Life, Liking What You See, The…
There are two somewhat related myths about neural networks in many intro ML courses that, I think, mislead more than they help. 1) The statement, "neural networks are powerful," is often followed by the citation to universal approximation theorem. 2) Neural networks are often…
Finally had a chance to listen through this pod with Sutton, which was interesting and amusing. As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea…
i have been scaling law- and manifold hypothesis-pilled youtube.com/watch?v=5eqRuV…
youtube.com
YouTube
AI can't cross this line and we don't know why.
nanochat now has a primordial identity and can talk a bit about itself and its capabilities (e.g. it knows it's nanochat d32 that cost $800, that it was built by me, that it can't speak languages other than English too well and why, etc.). This kind of customization is all done…
I fixed it :) deployed live now. This was done by doing a round of synthetic data generation to collect a 1000 multi-turn conversations (given a bunch of information including the readme of the nanochat project), and then mixing that into midtraining and SFT. fun!
In Rust, how the borrow-checker shapes your API inputs and outputs .. The general design principle is to choose signatures that minimize ownership churn while keeping call sites clean and safe. Accepting input • Borrow when you only read: `fn parse(src: &str)` • Borrow…
NVIDIA over USB4 on MacBook is ready to try! * ADT-UT3G dock + any 30/40/50 series GPU * Disable SIP * Install driver `extra/usbgpu/tbgpu` * Install NVK compiler `brew install tinymesa` * Test with: `DEBUG=2 NV_NAK=1 NV=1 python3 test/test_tiny.py TestTiny.test_plus`
Sharding. Database sharding is one of the common techniques to scale a database horizontally. You split the db into small parts called shards and distribute them across machines. Shards are typically in the few hundreds or even thousands (for extremely large databases). Usually…
United States 트렌드
- 1. Happy Halloween 208K posts
- 2. Dolphins 40.6K posts
- 3. YouTube TV 47.7K posts
- 4. #SinisterMinds 6,290 posts
- 5. Ryan Rollins 12.2K posts
- 6. Mike McDaniel 5,089 posts
- 7. Ravens 55.7K posts
- 8. Lamar 52.3K posts
- 9. Talbot 1,034 posts
- 10. Mary Ann N/A
- 11. YTTV N/A
- 12. Fiala 1,713 posts
- 13. Hulu 18.8K posts
- 14. #RUNSEOKJIN_epTOUR_ENCORE 82.8K posts
- 15. #DBX4 1,804 posts
- 16. #RHOC 3,411 posts
- 17. #Jin_TOUR_ENCORE 79.2K posts
- 18. Achane 4,793 posts
- 19. Henry 57K posts
- 20. UTSA 3,689 posts
Something went wrong.
Something went wrong.