Barlow Twins: a new super-simple self-supervised method to train joint-embedding architectures (aka Siamese nets) non

Basic idea: maximize the normalized correlation between a variable in the left branch and the same var in the right branch, while making the normalized cross-correlation between one var in the left branch and all other vars in the right branch as close to zero as possible.
2/N
In short: the loss tries to make the normalized cross-correlation between the embedding vectors coming out of the left branch and the right branch as close to the identity matrix as possible.
3/N
The 2 branches are always fed with differently-distorted version of the same image, and there is no need for dissimilar training pairs.

The objective makes the embedding vectors of the two branches as similar as possible, while maximizing their information content.
4/N
No contrastive samples, no huge batch size (optimal is 128), nor predictor, no moving-average weights, no vector quantization, nor cut gradients in one of the branches.
5/N
Competitive results on ImageNet with a linear classifier head.
Great results on semi-supervised ImageNet in the low labeled-data regime and on transfer tasks.
6/N
Results on ImageNet with linear classifier head
7/N
Results with 1% and 10% of ImagNet labeled images
8/N
Results on transfer tasks.
9/N
Arch is standard ResNet50 with 2048-D feature vec.
But contrary to others, the embedding size (projector output) is larger. The perf keeps going up as the embedding dim grows (we stopped at 16384).
Probably cause the feature vars are made independent, not just decorrelated.
10/N
Why Barlow? Horace Barlow was a pioneer of visual neuroscience who proposed the idea that the brain tries to minimize redundancy in representations.

By Jure Zbontar, Li Jing, Ishan Misra, yours truly, and Stéphane Deny.
All from FAIR.
To appear at ICML 2021
11/N
Don't you just hate slicing what would be a decent-size post into threaded thin tweets?
12/N
No, really. Don't you hate reading those long thread slices?
If you do, you could just read my Facebook post:
https://t.co/dQii7BEPQ5
13/N
N=13
Typo: optimal batch size is 1024, not 128.
14/13 (haha).

More from All

MASTER THREAD on Short Strangles.

Curated the best tweets from the best traders who are exceptional at managing strangles.

• Positional Strangles
• Intraday Strangles
• Position Sizing
• How to do Adjustments
• Plenty of Examples
• When to avoid
• Exit Criteria

How to sell Strangles in weekly expiry as explained by boss himself. @Mitesh_Engr

• When to sell
• How to do Adjustments
• Exit


Beautiful explanation on positional option selling by @Mitesh_Engr
Sir on how to sell low premium strangles yourself without paying anyone. This is a free mini course in


1st Live example of managing a strangle by Mitesh Sir. @Mitesh_Engr

• Sold Strangles 20% cap used
• Added 20% cap more when in profit
• Booked profitable leg and rolled up
• Kept rolling up profitable leg
• Booked loss in calls
• Sold only


2nd example by @Mitesh_Engr Sir on converting a directional trade into strangles. Option Sellers can use this for consistent profit.

• Identified a reversal and sold puts

• Puts decayed a lot

• When achieved 2% profit through puts then sold

You May Also Like