shaokaiyeah's profile picture. Machine learning researcher @ Apple

Shaokai Ye

@shaokaiyeah

Machine learning researcher @ Apple

Insightful! I have been thinking about "thinking in video" too

Some further thoughts on the idea of "thinking with images": 1) zero-shot tool use is limited -- you can’t just call an object detector to do visual search. That’s why approaches like VisProg/ViperGPT/Visual-sketchpad will not generalize or scale well. 2) visual search needs to…

sainingxie's tweet image. Some further thoughts on the idea of "thinking with images":

1) zero-shot tool use is limited -- you can’t just call an object detector to do visual search. That’s why approaches like VisProg/ViperGPT/Visual-sketchpad will not generalize or scale well.

2) visual search needs to…


I am on the job market! Contact me if you are interested in building multi-modal agentic systems that understand behaviors!

✨ Introducing a new #SOTA action recognition large multimodal language model: #LLaVAction! Understanding human behavior requires recognizing actions—a challenging task given the complexity of behavior. Large multimodal language models (#MLLMs) offer a promising path forward,…

TrackingActions's tweet image. ✨ Introducing a new #SOTA action recognition large multimodal language model: #LLaVAction!
 
Understanding human behavior requires recognizing actions—a challenging task given the complexity of behavior. Large multimodal language models (#MLLMs) offer a promising path forward,…
TrackingActions's tweet image. ✨ Introducing a new #SOTA action recognition large multimodal language model: #LLaVAction!
 
Understanding human behavior requires recognizing actions—a challenging task given the complexity of behavior. Large multimodal language models (#MLLMs) offer a promising path forward,…


So sweet to see them standing together

✨️🎅Merry Christmas🎅✨️

NOVECT_JP's tweet image. ✨️🎅Merry Christmas🎅✨️


Shaokai Ye 님이 재게시함

Automating the Search for Artificial Life with Foundation Models Presents ASAL, the first use of vision-language models in Artificial Life, discovering diverse, novel simulations across substrates proj: pub.sakana.ai/asal/ abs: arxiv.org/abs/2412.17799


Shaokai Ye 님이 재게시함

🚀 DLC AI Residents call for 2025! This year, we are doing something a little different to support even more underrepresented individuals 🥰 We are having in-person workshops again 🙌, & we are taking the 2025 AI Residency on the Road! 🛤️ App open now! deeplabcutairesidency.org


Shaokai Ye 님이 재게시함

🚨adversarial robustness is becoming even more critical as AI systems are deployed in the real-world, but how can we detect outliers (adversarials) without having trained on them 👀?  In our new preprint, we introduce AROS💍: It leverages neural ODEs and Lyapunov stability…

TrackingActions's tweet image. 🚨adversarial robustness is becoming even more critical as AI systems are deployed in the real-world, but how can we detect outliers (adversarials) without having trained on them 👀? 

In our new preprint, we introduce AROS💍: It leverages neural ODEs and Lyapunov stability…
TrackingActions's tweet image. 🚨adversarial robustness is becoming even more critical as AI systems are deployed in the real-world, but how can we detect outliers (adversarials) without having trained on them 👀? 

In our new preprint, we introduce AROS💍: It leverages neural ODEs and Lyapunov stability…

Really cool!

✨🥰 check out our article - and cover 🤩- about Decoding the Brain in @CellCellPress cell.com/cell/fulltext/… We review the mathematics, current approaches, and muse about the future… #BCI #neuraldecoding #neuroAI Thanks to my awesome co-authors Adriana Perez Rotondo,…

TrackingActions's tweet image. ✨🥰 check out our article - and cover 🤩- about Decoding the Brain in @CellCellPress 

cell.com/cell/fulltext/…

We review the mathematics, current approaches, and muse about the future…

#BCI #neuraldecoding #neuroAI

Thanks to my awesome co-authors Adriana Perez Rotondo,…


Come check this really cool paper if you are at ECCV!

I'm thrilled to present our paper "Elucidating the Hierarchical Nature of Behavior with Masked Autoencoders" at #ECCV24! 🎉 We focus on self-supervised learning for action segmentation to uncover the hierarchical structure of behavior. Paper: ecva.net/papers/eccv_20… 👇🧵



Shaokai Ye 님이 재게시함

How does sensorimotor (S1/M1) cortex support adaptive motor control?  Come find out in our latest preprint, which spans the development of a full adult forelimb model + physics simulations, neural-modeling for control, complex 🐭behavior 🕹️, large-scale imaging, and of course…


Shaokai Ye 님이 재게시함

DLC Residency 2024 Recap! 🎓💜 🎉Check out the awesome video from DLC Community Manager, @TheVetFuturist, and the 2024 Residents! youtu.be/lyJ2NDKng3g?si…

DeepLabCut's tweet card. DeepLabCut AI Residency 2024! 🎓💜

youtube.com

YouTube

DeepLabCut AI Residency 2024! 🎓💜


Shaokai Ye 님이 재게시함

I rewatch this once a month for motivation

@levelsio님으로부터

Shaokai Ye 님이 재게시함

🎉 Happy to have contributed a bit to this with my PhD student @shaokaiyeah, where we use @DeepLabCut #SuperAnimal foundation model for pose to do fully end-to-end unsupervised behavioral analysis with #keypointMoSeq. Huge congrats to first author Caleb Weinreb, and Scott…

Keypoint-MoSeq from the @Datta_lab is an unsupervised behavior segmentation algorithm that extracts behavioral modules from keypoint tracking data acquired with diverse algorithms, as demonstrated on data from mice, rats and fruit flies. (1/2)

naturemethods's tweet image. Keypoint-MoSeq from the @Datta_lab is an unsupervised behavior segmentation algorithm that extracts behavioral modules from keypoint tracking data acquired with diverse algorithms, as demonstrated on data from mice, rats and fruit flies. (1/2)


the pace of progress blows my mind

AI music videos are so hot right now 🔥



United States 트렌드

Loading...

Something went wrong.


Something went wrong.