Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
How can we use language supervision to learn better visual representations for robotics?
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The secret is *balance* (3/12)
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
Why is the ability to shape this balance important? (5/12)
How do we know?
Because we build an evaluation suite of 5 diverse robotics problem domains! (6/12)
Evaluation: the ARC Grasping dataset (https://t.co/rRI4ya84DL) – CC @andyzengtweets @SongShuran. (7/12)
Modeling *multi-frame* contexts (easy with Voltron) is also high-impact!
Evaluation: Franka Kitchen & Adroit Manipulation domains from R3M – CC @aravindr93 @Vikashplus. (8/12)
Given a video & language intent, we can score – in real time – how well the behavior in the video captures the intent.
Transfers to *robot data* – no robots during pretraining! (9/12)
Models & Pretraining: https://t.co/NOB3cpATYG
Evaluation Suite: https://t.co/aOzQu95J8z
Use our models: `pip install voltron-robotics` (10/12)
Further thanks to @ToyotaResearch, @stanfordnlp, and the @StanfordAILab ! (11/12)
More from All
You May Also Like
I’m torn on how to approach the idea of luck. I’m the first to admit that I am one of the luckiest people on the planet. To be born into a prosperous American family in 1960 with smart parents is to start life on third base. The odds against my very existence are astronomical.
I’ve always felt that the luckiest people I know had a talent for recognizing circumstances, not of their own making, that were conducive to a favorable outcome and their ability to quickly take advantage of them.
In other words, dumb luck was just that, it required no awareness on the person’s part, whereas “smart” luck involved awareness followed by action before the circumstances changed.
So, was I “lucky” to be born when I was—nothing I had any control over—and that I came of age just as huge databases and computers were advancing to the point where I could use those tools to write “What Works on Wall Street?” Absolutely.
Was I lucky to start my stock market investments near the peak of interest rates which allowed me to spend the majority of my adult life in a falling rate environment? Yup.
Ironies of Luck https://t.co/5BPWGbAxFi
— Morgan Housel (@morganhousel) March 14, 2018
"Luck is the flip side of risk. They are mirrored cousins, driven by the same thing: You are one person in a 7 billion player game, and the accidental impact of other people\u2019s actions can be more consequential than your own."
I’ve always felt that the luckiest people I know had a talent for recognizing circumstances, not of their own making, that were conducive to a favorable outcome and their ability to quickly take advantage of them.
In other words, dumb luck was just that, it required no awareness on the person’s part, whereas “smart” luck involved awareness followed by action before the circumstances changed.
So, was I “lucky” to be born when I was—nothing I had any control over—and that I came of age just as huge databases and computers were advancing to the point where I could use those tools to write “What Works on Wall Street?” Absolutely.
Was I lucky to start my stock market investments near the peak of interest rates which allowed me to spend the majority of my adult life in a falling rate environment? Yup.
