Today work starts on my next book - KINGS & QUEENS. It will include all English monarchs since Alfred the Great in 886AD, of which there are 61 + Oliver Cromwell, who will also get a chapter. I've already started recruiting writers, but am open to suggestions.
More from All
How can we use language supervision to learn better visual representations for robotics?
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Videos of humans performing everyday tasks (Something-Something-v2, Ego4D) offer a rich and diverse resource for learning representations for robotic manipulation.
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The Voltron framework offers a simple way to use language supervision to shape representation learning, building off of prior work in representations for robotics like MVP (https://t.co/Pb0mk9hb4i) and R3M (https://t.co/o2Fkc3fP0e).
The secret is *balance* (3/12)
Starting with a masked autoencoder over frames from these video clips, make a choice:
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
By trading off *conditioning* and *generation* we show that we can learn 1) better representations than prior methods, and 2) explicitly shape the balance of low and high-level features captured.
Why is the ability to shape this balance important? (5/12)
Introducing Voltron: Language-Driven Representation Learning for Robotics!
Paper: https://t.co/gIsRPtSjKz
Models: https://t.co/NOB3cpATYG
Evaluation: https://t.co/aOzQu95J8z
🧵👇(1 / 12)
Videos of humans performing everyday tasks (Something-Something-v2, Ego4D) offer a rich and diverse resource for learning representations for robotic manipulation.
Yet, an underused part of these datasets are the rich, natural language annotations accompanying each video. (2/12)
The Voltron framework offers a simple way to use language supervision to shape representation learning, building off of prior work in representations for robotics like MVP (https://t.co/Pb0mk9hb4i) and R3M (https://t.co/o2Fkc3fP0e).
The secret is *balance* (3/12)
Starting with a masked autoencoder over frames from these video clips, make a choice:
1) Condition on language and improve our ability to reconstruct the scene.
2) Generate language given the visual representation and improve our ability to describe what's happening. (4/12)
By trading off *conditioning* and *generation* we show that we can learn 1) better representations than prior methods, and 2) explicitly shape the balance of low and high-level features captured.
Why is the ability to shape this balance important? (5/12)
You May Also Like
1/x Fort Detrick History
Mr. Patrick, one of the chief scientists at the Army Biological Warfare Laboratories at Fort Detrick in Frederick, Md., held five classified US patents for the process of weaponizing anthrax.
2/x
Under Mr. Patrick’s direction, scientists at Fort Detrick developed a tularemia agent that, if disseminated by airplane, could cause casualties & sickness over 1000s mi². In a 10,000 mi² range, it had 90% casualty rate & 50% fatality rate
3/x His team explored Q fever, plague, & Venezuelan equine encephalitis, testing more than 20 anthrax strains to discern most lethal variety. Fort Detrick scientists used aerosol spray systems inside fountain pens, walking sticks, light bulbs, & even in 1953 Mercury exhaust pipes
4/x After retiring in 1986, Mr. Patrick remained one of the world’s foremost specialists on biological warfare & was a consultant to the CIA, FBI, & US military. He debriefed Soviet defector Ken Alibek, the deputy chief of the Soviet biowarfare program
https://t.co/sHqSaTSqtB
5/x Back in Time
In 1949 the Army created a small team of chemists at "Camp Detrick" called Special Operations Division. Its assignment was to find military uses for toxic bacteria. The coercive use of toxins was a new field, which fascinated Allen Dulles, later head of the CIA
Mr. Patrick, one of the chief scientists at the Army Biological Warfare Laboratories at Fort Detrick in Frederick, Md., held five classified US patents for the process of weaponizing anthrax.
2/x
Under Mr. Patrick’s direction, scientists at Fort Detrick developed a tularemia agent that, if disseminated by airplane, could cause casualties & sickness over 1000s mi². In a 10,000 mi² range, it had 90% casualty rate & 50% fatality rate
3/x His team explored Q fever, plague, & Venezuelan equine encephalitis, testing more than 20 anthrax strains to discern most lethal variety. Fort Detrick scientists used aerosol spray systems inside fountain pens, walking sticks, light bulbs, & even in 1953 Mercury exhaust pipes
4/x After retiring in 1986, Mr. Patrick remained one of the world’s foremost specialists on biological warfare & was a consultant to the CIA, FBI, & US military. He debriefed Soviet defector Ken Alibek, the deputy chief of the Soviet biowarfare program
https://t.co/sHqSaTSqtB
5/x Back in Time
In 1949 the Army created a small team of chemists at "Camp Detrick" called Special Operations Division. Its assignment was to find military uses for toxic bacteria. The coercive use of toxins was a new field, which fascinated Allen Dulles, later head of the CIA
