We're excited to announce WILDS v2.0, which adds unlabeled …

Unlabeled data can be a powerful source of leverage. It comes from a mixture of:
- source domains (same as the labeled training data)
- target domains (same as the labeled test data)
- extra domains with no labeled data.

We illustrate this for the GlobalWheat dataset:

We evaluated domain adaptation, self-training, & self-supervised methods on these datasets. Unfortunately, many methods did not do better than standard supervised training, despite using additional unlabeled data.

This table shows OOD test performance; higher numbers are better.

In contrast, prior work has shown these methods to be successful on standard domain adaptation tasks such as DomainNet, which we replicate below. This underscores the importance of developing and evaluating methods on a broad variety of distribution shifts.

We've added the unlabeled data loaders + method implementations to our Python package: https://t.co/S73kjDxMis. They're easy to use: check out the code snippet below!

We've also updated our leaderboards to accept submissions with and without unlabeled data.

We've uploaded the exact commands and hyperparameters used in our paper, as well as trained model checkpoints, to https://t.co/qI7yvTWGsT. This is thanks to @tonyh_lee, who oversaw all of the experimental infrastructure and made it fully reproducible on @CodaLabWS.

We're grateful to everyone who helped us with WILDS and the v2.0 update: https://t.co/1CAsr8JV99.

We'd also like to thank Jiang et al. for https://t.co/CSIYF8gcFT and Zhang et al. for https://t.co/Kla5i4C9Y9, which were very helpful references for our method implementations.

This was joint work with @shiorisagawa* @tonyh_lee* IrenaGao*, and @sangmichaelxie @kendrick_shen @ananyaku @weihua916 @michiyasunaga HenrikMarklund @sarameghanbeery @EtienneDavid @IanStavness @guowei_net @jure @kate_saenko_ @tatsu_hashimoto @svlevine @chelseabfinn @percyliang.

We'll be presenting this at the DistShift workshop at NeurIPS. Find us at our poster on Dec 13, 1-3pm Pacific Time: https://t.co/gid3wBSqb6

Read our paper for more details and analysis: https://t.co/m95JSY9LbJ

We're excited to announce WILDS v2.0, which adds unlabeled data to 8 datasets! This lets us benchmark methods for domain adaptation & representation learning. All labeled data & evaluations are unchanged.

(New) paper: https://t.co/9MaYUFluu7
Website: https://t.co/vA5KxsZf6c

🧵

More from All

You May Also Like

We're excited to announce WILDS v2.0, which adds unlabeled data to 8 datasets! This lets us benchmark methods for domain adaptation & representation learning. All labeled data & evaluations are unchanged. (New) paper: https://t.co/9MaYUFluu7Website: https://t.co/vA5KxsZf6c🧵

More from All

You May Also Like

We're excited to announce WILDS v2.0, which adds unlabeled data to 8 datasets! This lets us benchmark methods for domain adaptation & representation learning. All labeled data & evaluations are unchanged.

(New) paper: https://t.co/9MaYUFluu7
Website: https://t.co/vA5KxsZf6c

🧵