This is the ImageNet moment for real robotics. But instead of classifying images, it's about robots acting in our world.
1/ 🚨How well does embodied intelligence perform in the real physical world? 👉 Try it now: [robochallenge.ai] We are thrilled to introduce [@RoboChallengeAI] co-initiated by [@Dexmal_AI] and [@huggingface] The first large scale real-robot-based benchmark platform of…
2/ 🤖The first benchmark hosted on the platform, Table 30, consists of 30 different tasks and 24250 episodes of training data in total. The results of the first 6 baseline VLA algorithms can now be viewed on our website. Website: [robochallenge.ai]
3/ 📡 Advantages - Reproducibility & Fairness : Carefully engineered testing protocol for stable results; - Openness: All test procedures are video-recorded and released; - Diverse tasks. See tech report for detail [robochallenge.ai/robochallenge_…]
4/ ❓Why This Matters? Robots are stepping into the real world, but there's still no common benchmark that's open, unified and reproducible. How can we measure progress—or even fairly compare methods—without a shared standard? We're fixing that.
5/ 💻Real-Robot Benchmark Select a fleet of robots for their popularity. • UR5. A single 6-dof UR5 arm with a Robotiq gripper • Franka Panda. A-7 dof Franka arm • Aloha-like bimanual machine • A 6-dof ARX-5 arm, mouted on a table
6/ 🛡Real-Robot Testing Online → A set of low-level api is formalized to provide the exact timestamp of observations and state of the action queue to enable fine-grained control. → No docker images or model checkpoints are needed to be exchanged.
7/ ⏱Large-Scale Tasklist Table30 → Select 30 tasks for our initial benchmark release to benchmark the VLAs, and common real-robot benchmark only consist of 3-5 tasks. → Demonstration data was collected for each of them (around 1000 episodes per task).
8/ Controlled Tester We decide to control the task preparation by matching visual inputs and call this method controlled tester. In this way, the initial state of the scene and objects is largely fixed across evaluation of different models, making tests scalable.
9/ Scientific Distribution of Tasks (1) by the difficulties encountered by a VLA solution (2) by the type of robot (3) by the intended location of the task scenario (4) by the property of the main target object. 👆It shows good diversity and coverage.
10/ 🤖Base Models and Results → We tested popular open source VLA algorithms. As shown in results, even the most SOTA base model fails to achieve an overall high success rate, So we argue that our benchmark is a “necessity test” in the pursuit of general robotics.
11/ Evaluation & Submission → We’re scaling real-robot evaluation to accelerate embodied AI. → Ready to test your algorithms in the real world? ⬇️ Here’s how：
12/ 🏠Join our Discord to ask questions and discuss: discord.com/invite/8pD8QWDv
13/ RoboChallenge is more than a benchmark — it’s a public good for the robotics community. GitHub:github.com/RoboChallenge/… 😍 Whether you’re a robotics veteran or just entering the field, we’re here to support you!
I hope you've found this thread helpful. Follow me @jackcoder0 for more. Like/Repost the quote below if you can:
This is the ImageNet moment for real robotics. But instead of classifying images, it's about robots acting in our world.
Think of it as ImageNet for embodied AI: instead of predicting classes, robots predict actions in messy, physics-bound environments. Big leap from simulation to real-world utility.
Interesting parallel. Embodied AI's growth could mirror computer vision's rapid progress.
The holiday shopping rush will be here before you know it. Join us on the 📅 20th Oct at 14:00 pm (London time) for the first webinar in our Road to Holiday series and learn how to get started with advertising on X.
United States Trends
- 1. $SWC 2,455 posts
- 2. #UFC321 68.2K posts
- 3. Mateer 2,989 posts
- 4. Almeida 50.4K posts
- 5. Ole Miss 10.9K posts
- 6. Volkov 43.3K posts
- 7. Umar 23.3K posts
- 8. Arbuckle 2,104 posts
- 9. Oklahoma 15.6K posts
- 10. UCLA 7,592 posts
- 11. Lane Kiffin 4,968 posts
- 12. Hugh Freeze 1,234 posts
- 13. Jackson Arnold N/A
- 14. Bautista 41.4K posts
- 15. Northwestern 5,458 posts
- 16. #Huskers 1,201 posts
- 17. #iufb 2,216 posts
- 18. Lamar 17.9K posts
- 19. Kerkez 7,388 posts
- 20. #Sooners 2,062 posts
Something went wrong.
Something went wrong.