Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
How can we use language supervision to learn better visual representations for robotics?
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The secret is *balance* (3/12)
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
Why is the ability to shape this balance important? (5/12)
How do we know?
Because we build an evaluation suite of 5 diverse robotics problem domains! (6/12)
Evaluation: the ARC Grasping dataset (https://t.co/rRI4ya84DL) – CC @andyzengtweets @SongShuran. (7/12)
Modeling *multi-frame* contexts (easy with Voltron) is also high-impact!
Evaluation: Franka Kitchen & Adroit Manipulation domains from R3M – CC @aravindr93 @Vikashplus. (8/12)
Given a video & language intent, we can score – in real time – how well the behavior in the video captures the intent.
Transfers to *robot data* – no robots during pretraining! (9/12)
Models & Pretraining: https://t.co/NOB3cpATYG
Evaluation Suite: https://t.co/aOzQu95J8z
Use our models: `pip install voltron-robotics` (10/12)
Further thanks to @ToyotaResearch, @stanfordnlp, and the @StanfordAILab ! (11/12)
More from All
You May Also Like
Ivor Cummins has been wrong (or lying) almost entirely throughout this pandemic and got paid handsomly for it.
He has been wrong (or lying) so often that it will be nearly impossible for me to track every grift, lie, deceit, manipulation he has pulled. I will use...
... other sources who have been trying to shine on light on this grifter (as I have tried to do, time and again:
Example #1: "Still not seeing Sweden signal versus Denmark really"... There it was (Images attached).
19 to 80 is an over 300% difference.
Tweet: https://t.co/36FnYnsRT9
Example #2 - "Yes, I'm comparing the Noridcs / No, you cannot compare the Nordics."
I wonder why...
Tweets: https://t.co/XLfoX4rpck / https://t.co/vjE1ctLU5x
Example #3 - "I'm only looking at what makes the data fit in my favour" a.k.a moving the goalposts.
Tweets: https://t.co/vcDpTu3qyj / https://t.co/CA3N6hC2Lq
He has been wrong (or lying) so often that it will be nearly impossible for me to track every grift, lie, deceit, manipulation he has pulled. I will use...
... other sources who have been trying to shine on light on this grifter (as I have tried to do, time and again:
Ivor Cummins BE (Chem) is a former R&D Manager at HP (sourcre: https://t.co/Wbf5scf7gn), turned Content Creator/Podcast Host/YouTube personality. (Call it what you will.)
— Steve (@braidedmanga) November 17, 2020
Example #1: "Still not seeing Sweden signal versus Denmark really"... There it was (Images attached).
19 to 80 is an over 300% difference.
Tweet: https://t.co/36FnYnsRT9
Example #2 - "Yes, I'm comparing the Noridcs / No, you cannot compare the Nordics."
I wonder why...
Tweets: https://t.co/XLfoX4rpck / https://t.co/vjE1ctLU5x
Example #3 - "I'm only looking at what makes the data fit in my favour" a.k.a moving the goalposts.
Tweets: https://t.co/vcDpTu3qyj / https://t.co/CA3N6hC2Lq