Ben Evans
@justsomecsnerd
PhD student studying Computer Science at NYU in the CILVR Lab
You might like
When I train an RL agent to play soccer
The "Supervised Policy Learning for Real Robots" tutorial at #RSS2024 is now public! supervised-robot-learning.github.io We're trying to democratize the "folk knowledge" of training real robot policies, and cover the fundamentals of imitation, recent algorithms, & practical considerations.
diff History was accepted to ICML and the camera-ready version is now available on arXiv! arxiv.org/abs/2312.07540 We argue that LM agents should be tuned and prompted using "text delta" summaries of recent interaction -- a text analogue of optical flow See you in Vienna! 🇦🇹
Open-ended decision-making is 𝙝𝙖𝙧𝙙, even for SOTA LLMs. Tuning on demos and providing models with history can help, but prompts quickly become verbose. Excited to share our new work: 𝙙𝙞𝙛𝙛 𝙝𝙞𝙨𝙩𝙤𝙧𝙮. Instead of scaling up models or data, tune on *text deltas* 🧵 1/n
Building robot intelligence requires high-quality robot data. But far too many tools to collect data are closed and custom-built. Open Teach is fully open-sourced and is: - Calibration-free - Supports multiple arms, hands, & mobile manipulators - Costs just $500
LLMs swept the world by predicting discrete tokens. But what’s the right tool to model continuous, multi-modal, and high dim behaviors? Meet Vector Quantized Behavior Transformer (VQ-BeT), beating or matching diffusion based models in speed, quality, and diversity. 🧵
Deep SSMs are great in modeling audio / language, but they really excel in modeling continuous, raw sensor data. Introducing Hierarchical SSMs (HiSS), a new framework that improves upon Transformers, RNNs, S4 & Mamba for a range of sensors (touch, accelerometers & IMUs). 🧵👇
Open-ended decision-making is 𝙝𝙖𝙧𝙙, even for SOTA LLMs. Tuning on demos and providing models with history can help, but prompts quickly become verbose. Excited to share our new work: 𝙙𝙞𝙛𝙛 𝙝𝙞𝙨𝙩𝙤𝙧𝙮. Instead of scaling up models or data, tune on *text deltas* 🧵 1/n
A simple way to improve Language Agents: feed in sequences of text diffs with text generated in previous timesteps. ✅ SOTA in NetHack neural models. ✅ Requires 1800x lesser data than prior SOTA. ✅ Also works on simpler BabyAI environments. diffhistory.github.io
Wouldn’t it be nice if you could bring a robot home, give it a video of your room, and immediately start asking it to move objects around? Turns out, now you can! Introducing OK-Robot, a zero-shot language-specified pick & drop system that we built with exactly ZERO training! 🧵
Excited to release OK-Robot, an open-vocabulary mobile-manipulator for homes. Simply tell the robot what to pick and where to drop it in natural language, and it will do it. Like: Me: "OK Robot, move the Takis from the desk to the nightstand" Robot: ⬇️
Meet Dobb·E: a home robot system that needs just 5 minutes of human teaching to learn new tasks. Dobb·E has visited 10 homes, learned 100+ tasks, and we are just getting started! Dobb·E is fully open-sourced (including hardware, models, and software): dobb-e.com 🧵
Creating a "unified" policy for a robot is uniquely challenging for two reasons: (a) data is scarce; (b) a single robot can only act sequentially. In PolyTask, we learn a single policy to solve many tasks in a lifelong fashion given *just 1 demonstration* for each task. 🧵
📢#𝗥𝗼𝗯𝗼𝗛𝗶𝘃𝗲 - a unified robot learning framework ✅Designed for genralizn first robot-learning era ✅Diverse (500 envs, 8 domain) ✅Single flag for Sim<>Real ✅TeleOper Support ✅Multi-(Skill x Task) realworld dataset ✅pip install robohive tinyurl.com/robohive 🧵👇
Was great to be a part of this! Dexterous robots are becoming more proficient all the time.
We just released TAVI -- a robotics framework that combines touch and vision to solve challenging dexterous tasks in under 1 hour. The key? Use human demonstrations to initialize a policy, followed by tactile-based online learning with vision-based rewards. Details in🧵(1/7)
When did you last see a cool robot in your home? Unless you own a robot vacuum, the answer is likely never. Lerrel Pinto, one of this year's 35 Innovators Under 35, wants to change that. trib.al/RFMBe8Q
> Services used by only IST teams too slow > Realize it's cause pods in us-west-1 > Ask teams to migrate them to ap-south-1 > 6 months discussions > Resolution reached, Migration approved, Yay! > Mandates a new VPN for login though, np. > mfw it is now twice as slow cause the…
🎉 A majority of NYU researchers have signed a Union Authorization Card to form our union 🎉 We’re excited to join thousands of researchers across the country who have organized for a voice over our working conditions, including most recently at Sinai and Columbia here in NYC!
I-JEPA: Efficient method for Self-Supervised Learning of image features. No need for data augmentation, just masking. Joint embedding predictive architecture, not generative. And it's open source, of course. Blog: ai.facebook.com/blog/yann-lecu… Paper: arxiv.org/abs/2301.08243 Code &…
Today we're releasing our work on I-JEPA — self-supervised computer vision that learns to understand the world by predicting it. It's the first model based on a component of @ylecun's vision to make AI systems learn and reason like animals and humans. Details ⬇️
Tactile feedback is one of the most important modalities in manipulation, but has been underutilized in dexterous hands. T-Dex is a framework for learning dexterous policies from tactile play data, beating vision and torque-based methods by 1.7x. tactile-dexterity.github.io 🧵👇
While we are going gaga over large models and big data, there is still incredible value left to extract in small models and data, especially in robotics. All the skills shown below were each trained with <1 min of human data and <20 min of online RL fast-imitation.github.io 🧵👇
United States Trends
- 1. Thanksgiving 363K posts
- 2. Fani Willis 8,359 posts
- 3. Trumplican 2,386 posts
- 4. Golesh 1,514 posts
- 5. Stranger Things 154K posts
- 6. Hong Kong 72.6K posts
- 7. Elijah Moore N/A
- 8. #wednesdaymotivation 6,382 posts
- 9. Ruth 13.6K posts
- 10. #Wednesdayvibe 3,386 posts
- 11. Nuns 9,911 posts
- 12. #WednesdayWisdom 1,114 posts
- 13. Khabib 5,339 posts
- 14. Karoline Leavitt 27.4K posts
- 15. Ribs 11.3K posts
- 16. Mora 22.4K posts
- 17. #TejRan 4,029 posts
- 18. #BurnoutSyndromeSeriesEP1 269K posts
- 19. Guinea-Bissau 12K posts
- 20. Gretzky N/A
You might like
-
Jacky Liang
@jackyliang42 -
Oier Mees
@oier_mees -
Vikash Kumar
@Vikashplus -
Lerrel Pinto
@LerrelPinto -
Kay - Liyiming Ke
@xkelym -
Jyo Pari
@jyo_pari -
Shubham Tulsiani
@shubhtuls -
Aravind Rajeswaran
@aravindr93 -
Ulyana Piterbarg
@ulyanapiterbarg -
Shikhar Bahl
@shikharbahl -
Siddhant Haldar
@haldar_siddhant -
Suraj Nair
@SurajNair_1 -
David Brandfonbrener
@brandfonbrener -
Anthony GX-Chen
@AntChen_ -
Mahi Shafiullah 🏠🤖
@notmahi
Something went wrong.
Something went wrong.