How can we use language supervision to learn better visual representations for robotics?

Introducing Voltron: Language-Driven Representation Learning for Robotics!

Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z

🧵👇(1 / 12)

Videos of humans performing everyday tasks (Something-Something-v2, Ego4D) offer a rich and diverse resource for learning representations for robotic manipulation.

Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The Voltron framework offers a simple way to use language supervision to shape representation learning, building off of prior work in representations for robotics like MVP (https://t.co/Pb0mk9hb4i) and R3M (https://t.co/o2Fkc3fP0e).

The secret is *balance* (3/12)
Starting with a masked autoencoder over frames from these video clips, make a choice:

1) Condition on language and improve our ability to reconstruct the scene.

2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
By trading off *conditioning* and *generation* we show that we can learn 1) better representations than prior methods, and 2) explicitly shape the balance of low and high-level features captured.

Why is the ability to shape this balance important? (5/12)
Because robotics isn't a single thing! While prior work focuses on learning for control, there are so many problems we care about – problems that require different features!

How do we know?

Because we build an evaluation suite of 5 diverse robotics problem domains! (6/12)
Problems like grasp affordance prediction (per-pixel segmentation) tend to require more *low-level* spatial features; edges, object boundaries, textures.

Evaluation: the ARC Grasping dataset (https://t.co/rRI4ya84DL) – CC @andyzengtweets @SongShuran. (7/12)
Learning for control tasks benefit from representations that mix of low and high-level features.

Modeling *multi-frame* contexts (easy with Voltron) is also high-impact!

Evaluation: Franka Kitchen & Adroit Manipulation domains from R3M – CC @aravindr93 @Vikashplus. (8/12)
Really cool is how we can use the generative language model zero-shot, with no extra data.

Given a video & language intent, we can score – in real time – how well the behavior in the video captures the intent.

Transfers to *robot data* – no robots during pretraining! (9/12)
But don't take our word for it – try out our representations yourself... or evaluate your own!

Models & Pretraining: https://t.co/NOB3cpATYG
Evaluation Suite: https://t.co/aOzQu95J8z

Use our models: `pip install voltron-robotics` (10/12)
This project was a huge endeavor; one that would not have been possible without amazing collaborators and mentors – @SurajNair_1 @_anniechen_ @tkollar @chelseabfinn @DorsaSadigh and @percyliang.

Further thanks to @ToyotaResearch, @stanfordnlp, and the @StanfordAILab ! (11/12)
I'm really excited to see the impact of language on shaping representations for robotics... but this isn't the end. The hard parts of robotics remain hard.

Voltron is a building block – a tool. I can't wait to see how y'all use it. Thanks folks – and stay tuned 🤖🚀! (12/12)

More from All

@franciscodeasis https://t.co/OuQaBRFPu7
Unfortunately the "This work includes the identification of viral sequences in bat samples, and has resulted in the isolation of three bat SARS-related coronaviruses that are now used as reagents to test therapeutics and vaccines." were BEFORE the


chimeric infectious clone grants were there.https://t.co/DAArwFkz6v is in 2017, Rs4231.
https://t.co/UgXygDjYbW is in 2016, RsSHC014 and RsWIV16.
https://t.co/krO69CsJ94 is in 2013, RsWIV1. notice that this is before the beginning of the project

starting in 2016. Also remember that they told about only 3 isolates/live viruses. RsSHC014 is a live infectious clone that is just as alive as those other "Isolates".

P.D. somehow is able to use funds that he have yet recieved yet, and send results and sequences from late 2019 back in time into 2015,2013 and 2016!

https://t.co/4wC7k1Lh54 Ref 3: Why ALL your pangolin samples were PCR negative? to avoid deep sequencing and accidentally reveal Paguma Larvata and Oryctolagus Cuniculus?
कुंडली में 12 भाव होते हैं। कैसे ज्योतिष द्वारा रोग के आंकलन करते समय कुंडली के विभिन्न भावों से गणना करते हैं आज इस पर चर्चा करेंगे।
कुण्डली को कालपुरुष की संज्ञा देकर इसमें शरीर के अंगों को स्थापित कर उनसे रोग, रोगेश, रोग को बढ़ाने घटाने वाले ग्रह


रोग की स्थिति में उत्प्रेरक का कार्य करने वाले ग्रह, आयुर्वेदिक/ऐलोपैथी/होमियोपैथी में से कौन कारगर होगा इसका आँकलन, रक्त विकार, रक्त और आपरेशन की स्थिति, कौन सा आंतरिक या बाहरी अंग प्रभावित होगा इत्यादि गणना करने में कुंडली का प्रयोग किया जाता है।


मेडिकल ज्योतिष में आज के समय में Dr. K. S. Charak का नाम निर्विवाद रूप से प्रथम स्थान रखता है। उनकी लिखी कई पुस्तकें आज इस क्षेत्र में नए ज्योतिषों का मार्गदर्शन कर रही हैं।
प्रथम भाव -
इस भाव से हम व्यक्ति की रोगप्रतिरोधक क्षमता, सिर, मष्तिस्क का विचार करते हैं।


द्वितीय भाव-
दाहिना नेत्र, मुख, वाणी, नाक, गर्दन व गले के ऊपरी भाग का विचार होता है।
तृतीय भाव-
अस्थि, गला,कान, हाथ, कंधे व छाती के आंतरिक अंगों का शुरुआती भाग इत्यादि।

चतुर्थ भाव- छाती व इसके आंतरिक अंग, जातक की मानसिक स्थिति/प्रकृति, स्तन आदि की गणना की जाती है


पंचम भाव-
जातक की बुद्धि व उसकी तीव्रता,पीठ, पसलियां,पेट, हृदय की स्थिति आंकलन में प्रयोग होता है।

षष्ठ भाव-
रोग भाव कहा जाता है। कुंडली मे इसके तत्कालिक भाव स्वामी, कालपुरुष कुंडली के स्वामी, दृष्टि संबंध, रोगेश की स्थिति, रोगेश के नक्षत्र औऱ रोगेश व भाव की डिग्री इत्यादि।
The best morning routine?

Starts the night before.

9 evening habits that make all the difference:

1. Write down tomorrow's 3:3:3 plan

• 3 hours on your most important project
• 3 shorter tasks
• 3 maintenance activities

Defining a "productive day" is crucial.

Or else you'll never be at peace (even with excellent output).

Learn more


2. End the workday with a shutdown ritual

Create a short shutdown ritual (hat-tip to Cal Newport). Close your laptop, plug in the charger, spend 2 minutes tidying your desk. Then say, "shutdown."

Separating your life and work is key.

3. Journal 1 beautiful life moment

Delicious tacos, presentation you crushed, a moment of inner peace. Write it down.

Gratitude programs a mindset of abundance.

4. Lay out clothes

Get exercise clothes ready for tomorrow. Upon waking up, jump rope for 2 mins. It will activate your mind + body.

You May Also Like

Ivor Cummins has been wrong (or lying) almost entirely throughout this pandemic and got paid handsomly for it.

He has been wrong (or lying) so often that it will be nearly impossible for me to track every grift, lie, deceit, manipulation he has pulled. I will use...


... other sources who have been trying to shine on light on this grifter (as I have tried to do, time and again:


Example #1: "Still not seeing Sweden signal versus Denmark really"... There it was (Images attached).
19 to 80 is an over 300% difference.

Tweet: https://t.co/36FnYnsRT9


Example #2 - "Yes, I'm comparing the Noridcs / No, you cannot compare the Nordics."

I wonder why...

Tweets: https://t.co/XLfoX4rpck / https://t.co/vjE1ctLU5x


Example #3 - "I'm only looking at what makes the data fit in my favour" a.k.a moving the goalposts.

Tweets: https://t.co/vcDpTu3qyj / https://t.co/CA3N6hC2Lq
Oh my Goodness!!!

I might have a panic attack due to excitement!!

Read this thread to the end...I just had an epiphany and my mind is blown. Actually, more than blown. More like OBLITERATED! This is the thing! This is the thing that will blow the entire thing out of the water!


Has this man been concealing his true identity?

Is this man a supposed 'dead' Seal Team Six soldier?

Witness protection to be kept safe until the right moment when all will be revealed?!

Who ELSE is alive that may have faked their death/gone into witness protection?


Were "golden tickets" inside the envelopes??


Are these "golden tickets" going to lead to their ultimate undoing?

Review crumbs on the board re: 'gold'.


#SEALTeam6 Trump re-tweeted this.
1/12

RT-PCR corona (test) scam

Symptomatic people are tested for one and only one respiratory virus. This means that other acute respiratory infections are reclassified as


2/12

It is tested exquisitely with a hypersensitive non-specific RT-PCR test / Ct >35 (>30 is nonsense, >35 is madness), without considering Ct and clinical context. This means that more acute respiratory infections are reclassified as


3/12

The Drosten RT-PCR test is fabricated in a way that each country and laboratory perform it differently at too high Ct and that the high rate of false positives increases massively due to cross-reaction with other (corona) viruses in the "flu


4/12

Even asymptomatic, previously called healthy, people are tested (en masse) in this way, although there is no epidemiologically relevant asymptomatic transmission. This means that even healthy people are declared as COVID


5/12

Deaths within 28 days after a positive RT-PCR test from whatever cause are designated as deaths WITH COVID. This means that other causes of death are reclassified as