Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
How can we use language supervision to learn better visual representations for robotics?
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The secret is *balance* (3/12)
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
Why is the ability to shape this balance important? (5/12)
How do we know?
Because we build an evaluation suite of 5 diverse robotics problem domains! (6/12)
Evaluation: the ARC Grasping dataset (https://t.co/rRI4ya84DL) – CC @andyzengtweets @SongShuran. (7/12)
Modeling *multi-frame* contexts (easy with Voltron) is also high-impact!
Evaluation: Franka Kitchen & Adroit Manipulation domains from R3M – CC @aravindr93 @Vikashplus. (8/12)
Given a video & language intent, we can score – in real time – how well the behavior in the video captures the intent.
Transfers to *robot data* – no robots during pretraining! (9/12)
Models & Pretraining: https://t.co/NOB3cpATYG
Evaluation Suite: https://t.co/aOzQu95J8z
Use our models: `pip install voltron-robotics` (10/12)
Further thanks to @ToyotaResearch, @stanfordnlp, and the @StanfordAILab ! (11/12)
More from All
The best morning routine?
Starts the night before.
9 evening habits that make all the difference:
1. Write down tomorrow's 3:3:3 plan
• 3 hours on your most important project
• 3 shorter tasks
• 3 maintenance activities
Defining a "productive day" is crucial.
Or else you'll never be at peace (even with excellent output).
Learn more
2. End the workday with a shutdown ritual
Create a short shutdown ritual (hat-tip to Cal Newport). Close your laptop, plug in the charger, spend 2 minutes tidying your desk. Then say, "shutdown."
Separating your life and work is key.
3. Journal 1 beautiful life moment
Delicious tacos, presentation you crushed, a moment of inner peace. Write it down.
Gratitude programs a mindset of abundance.
4. Lay out clothes
Get exercise clothes ready for tomorrow. Upon waking up, jump rope for 2 mins. It will activate your mind + body.
Starts the night before.
9 evening habits that make all the difference:
1. Write down tomorrow's 3:3:3 plan
• 3 hours on your most important project
• 3 shorter tasks
• 3 maintenance activities
Defining a "productive day" is crucial.
Or else you'll never be at peace (even with excellent output).
Learn more
How to be 5x more productive.
— Ben Meer (@SystemSunday) August 1, 2022
A best-selling author\u2019s 3-3-3 Method:
2. End the workday with a shutdown ritual
Create a short shutdown ritual (hat-tip to Cal Newport). Close your laptop, plug in the charger, spend 2 minutes tidying your desk. Then say, "shutdown."
Separating your life and work is key.
3. Journal 1 beautiful life moment
Delicious tacos, presentation you crushed, a moment of inner peace. Write it down.
Gratitude programs a mindset of abundance.
4. Lay out clothes
Get exercise clothes ready for tomorrow. Upon waking up, jump rope for 2 mins. It will activate your mind + body.