More from All
How can we use language supervision to learn better visual representations for robotics?
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Videos of humans performing everyday tasks (Something-Something-v2, Ego4D) offer a rich and diverse resource for learning representations for robotic manipulation.
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The Voltron framework offers a simple way to use language supervision to shape representation learning, building off of prior work in representations for robotics like MVP (https://t.co/Pb0mk9hb4i) and R3M (https://t.co/o2Fkc3fP0e).
The secret is *balance* (3/12)
Starting with a masked autoencoder over frames from these video clips, make a choice:
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
By trading off *conditioning* and *generation* we show that we can learn 1) better representations than prior methods, and 2) explicitly shape the balance of low and high-level features captured.
Why is the ability to shape this balance important? (5/12)
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Videos of humans performing everyday tasks (Something-Something-v2, Ego4D) offer a rich and diverse resource for learning representations for robotic manipulation.
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The Voltron framework offers a simple way to use language supervision to shape representation learning, building off of prior work in representations for robotics like MVP (https://t.co/Pb0mk9hb4i) and R3M (https://t.co/o2Fkc3fP0e).
The secret is *balance* (3/12)
Starting with a masked autoencoder over frames from these video clips, make a choice:
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
By trading off *conditioning* and *generation* we show that we can learn 1) better representations than prior methods, and 2) explicitly shape the balance of low and high-level features captured.
Why is the ability to shape this balance important? (5/12)
1. Mini Thread on Conflicts of Interest involving the authors of the Nature Toilet Paper:
https://t.co/VUYbsKGncx
Kristian G. Andersen
Andrew Rambaut
Ian Lipkin
Edward C. Holmes
Robert F. Garry
2. Thanks to @newboxer007 for forwarding the link to the research by an Australian in Taiwan (not on
3. K.Andersen didn't mention "competing interests"
Only Garry listed Zalgen Labs, which we will look at later.
In acknowledgements, Michael Farzan, Wellcome Trust, NIH, ERC & ARC are mentioned.
Author affiliations listed as usual.
Note the 328 Citations!
https://t.co/nmOeohM89Q
4. Kristian Andersen (1)
Andersen worked with USAMRIID & Fort Detrick scientists on research, with Robert Garry, Jens Kuhn & Sina Bavari among
5. Kristian Andersen (2)
Works at Scripps Research Institute, which WAS in serious financial trouble, haemorrhaging 20 million $ a year.
But just when the first virus cases were emerging, they received great news.
They issued a press release dated November 27, 2019:
https://t.co/VUYbsKGncx
Kristian G. Andersen
Andrew Rambaut
Ian Lipkin
Edward C. Holmes
Robert F. Garry
2. Thanks to @newboxer007 for forwarding the link to the research by an Australian in Taiwan (not on
3. K.Andersen didn't mention "competing interests"
Only Garry listed Zalgen Labs, which we will look at later.
In acknowledgements, Michael Farzan, Wellcome Trust, NIH, ERC & ARC are mentioned.
Author affiliations listed as usual.
Note the 328 Citations!
https://t.co/nmOeohM89Q
4. Kristian Andersen (1)
Andersen worked with USAMRIID & Fort Detrick scientists on research, with Robert Garry, Jens Kuhn & Sina Bavari among
Our Hans Kristian Andersen working with Jens H. Kuhn, Sina Bavari, Robert F. Garry, Stuart T. Nichol,Gustavo Palacios, Sheli R. Radoshitzky from USAMRIID and Fort Detrick to tell more fairy tales? Full emails listed for queries...https://t.co/kLRoQTxiGD pic.twitter.com/uHNuGraPP2
— Billy Bostickson \U0001f3f4\U0001f441&\U0001f441 \U0001f193 (@BillyBostickson) August 26, 2020
5. Kristian Andersen (2)
Works at Scripps Research Institute, which WAS in serious financial trouble, haemorrhaging 20 million $ a year.
But just when the first virus cases were emerging, they received great news.
They issued a press release dated November 27, 2019:
You May Also Like
I just finished Eric Adler's The Battle of the Classics, and wanted to say something about Joel Christiansen's review linked below. I am not sure what motivates the review (I speculate a bit below), but it gives a very misleading impression of the book. 1/x
The meat of the criticism is that the history Adler gives is insufficiently critical. Adler describes a few figures who had a great influence on how the modern US university was formed. It's certainly critical: it focuses on the social Darwinism of these figures. 2/x
Other insinuations and suggestions in the review seem wildly off the mark, distorted, or inappropriate-- for example, that the book is clickbaity (it is scholarly) or conservative (hardly) or connected to the events at the Capitol (give me a break). 3/x
The core question: in what sense is classics inherently racist? Classics is old. On Adler's account, it begins in ancient Rome and is revived in the Renaissance. Slavery (Christiansen's primary concern) is also very old. Let's say classics is an education for slaveowners. 4/x
It's worth remembering that literacy itself is elite throughout most of this history. Literacy is, then, also the education of slaveowners. We can honor oral and musical traditions without denying that literacy is, generally, good. 5/x
As someone\u2019s who\u2019s read the book, this review strikes me as tremendously unfair. It mostly faults Adler for not writing the book the reviewer wishes he had! https://t.co/pqpt5Ziivj
— Teresa M. Bejan (@tmbejan) January 12, 2021
The meat of the criticism is that the history Adler gives is insufficiently critical. Adler describes a few figures who had a great influence on how the modern US university was formed. It's certainly critical: it focuses on the social Darwinism of these figures. 2/x
Other insinuations and suggestions in the review seem wildly off the mark, distorted, or inappropriate-- for example, that the book is clickbaity (it is scholarly) or conservative (hardly) or connected to the events at the Capitol (give me a break). 3/x
The core question: in what sense is classics inherently racist? Classics is old. On Adler's account, it begins in ancient Rome and is revived in the Renaissance. Slavery (Christiansen's primary concern) is also very old. Let's say classics is an education for slaveowners. 4/x
It's worth remembering that literacy itself is elite throughout most of this history. Literacy is, then, also the education of slaveowners. We can honor oral and musical traditions without denying that literacy is, generally, good. 5/x
This is a pretty valiant attempt to defend the "Feminist Glaciology" article, which says conventional wisdom is wrong, and this is a solid piece of scholarship. I'll beg to differ, because I think Jeffery, here, is confusing scholarship with "saying things that seem right".
The article is, at heart, deeply weird, even essentialist. Here, for example, is the claim that proposing climate engineering is a "man" thing. Also a "man" thing: attempting to get distance from a topic, approaching it in a disinterested fashion.
Also a "man" thing—physical courage. (I guess, not quite: physical courage "co-constitutes" masculinist glaciology along with nationalism and colonialism.)
There's criticism of a New York Times article that talks about glaciology adventures, which makes a similar point.
At the heart of this chunk is the claim that glaciology excludes women because of a narrative of scientific objectivity and physical adventure. This is a strong claim! It's not enough to say, hey, sure, sounds good. Is it true?
Imagine for a moment the most obscurantist, jargon-filled, po-mo article the politically correct academy might produce. Pure SJW nonsense. Got it? Chances are you're imagining something like the infamous "Feminist Glaciology" article from a few years back.https://t.co/NRaWNREBvR pic.twitter.com/qtSFBYY80S
— Jeffrey Sachs (@JeffreyASachs) October 13, 2018
The article is, at heart, deeply weird, even essentialist. Here, for example, is the claim that proposing climate engineering is a "man" thing. Also a "man" thing: attempting to get distance from a topic, approaching it in a disinterested fashion.
Also a "man" thing—physical courage. (I guess, not quite: physical courage "co-constitutes" masculinist glaciology along with nationalism and colonialism.)
There's criticism of a New York Times article that talks about glaciology adventures, which makes a similar point.
At the heart of this chunk is the claim that glaciology excludes women because of a narrative of scientific objectivity and physical adventure. This is a strong claim! It's not enough to say, hey, sure, sounds good. Is it true?
1/x Fort Detrick History
Mr. Patrick, one of the chief scientists at the Army Biological Warfare Laboratories at Fort Detrick in Frederick, Md., held five classified US patents for the process of weaponizing anthrax.
2/x
Under Mr. Patrick’s direction, scientists at Fort Detrick developed a tularemia agent that, if disseminated by airplane, could cause casualties & sickness over 1000s mi². In a 10,000 mi² range, it had 90% casualty rate & 50% fatality rate
3/x His team explored Q fever, plague, & Venezuelan equine encephalitis, testing more than 20 anthrax strains to discern most lethal variety. Fort Detrick scientists used aerosol spray systems inside fountain pens, walking sticks, light bulbs, & even in 1953 Mercury exhaust pipes
4/x After retiring in 1986, Mr. Patrick remained one of the world’s foremost specialists on biological warfare & was a consultant to the CIA, FBI, & US military. He debriefed Soviet defector Ken Alibek, the deputy chief of the Soviet biowarfare program
https://t.co/sHqSaTSqtB
5/x Back in Time
In 1949 the Army created a small team of chemists at "Camp Detrick" called Special Operations Division. Its assignment was to find military uses for toxic bacteria. The coercive use of toxins was a new field, which fascinated Allen Dulles, later head of the CIA
Mr. Patrick, one of the chief scientists at the Army Biological Warfare Laboratories at Fort Detrick in Frederick, Md., held five classified US patents for the process of weaponizing anthrax.
2/x
Under Mr. Patrick’s direction, scientists at Fort Detrick developed a tularemia agent that, if disseminated by airplane, could cause casualties & sickness over 1000s mi². In a 10,000 mi² range, it had 90% casualty rate & 50% fatality rate
3/x His team explored Q fever, plague, & Venezuelan equine encephalitis, testing more than 20 anthrax strains to discern most lethal variety. Fort Detrick scientists used aerosol spray systems inside fountain pens, walking sticks, light bulbs, & even in 1953 Mercury exhaust pipes
4/x After retiring in 1986, Mr. Patrick remained one of the world’s foremost specialists on biological warfare & was a consultant to the CIA, FBI, & US military. He debriefed Soviet defector Ken Alibek, the deputy chief of the Soviet biowarfare program
https://t.co/sHqSaTSqtB
5/x Back in Time
In 1949 the Army created a small team of chemists at "Camp Detrick" called Special Operations Division. Its assignment was to find military uses for toxic bacteria. The coercive use of toxins was a new field, which fascinated Allen Dulles, later head of the CIA