Buzz Chronicles
Follow
  • Home
  • Threads
    • Daily Charts
    • Most Popular
    • Most Recent
  • Authors
  • Categories
    • Life
    • Tech
    • Culture
    • Politics
    • Society
    • Fun
    • See All Categories
  • About

Ankitsrihbti Authors Ilyaeck

7 days 30 days All time Recent Popular
ilyaeck
ilyaeck
@ilyaeck
Attention may be all you *want*, but what you *need* is effective token mixing!
In which we replace Transformers' self-attention with FFT and it works nearly as well but faster/cheaper.
https://t.co/GiUvHkB3SK
By James Lee-Thorpe, Joshua Ainslie, @santiontanon and myself, sorta


Attention clearly works - but why? What's essential in it and what's secondary? What needs to be adaptive/learned and what can be precomputed?

The paper asks these questions, with some surprising insights.

These questions and insights echo other very recent findings like @ytay017's Pretrained CNNs for NLP
https://t.co/k0jOuYMxzz and MLP-Mixer for Vision from @neilhoulsby and co. (Like them, we also found combos of MLP to be promising).
ALL
  • Page 1 of 1
How does it work?
  • 💬 Reply to a thread with "@buzz_chronicles save" or "@buzz_chronicles save as category"
    🤖 Our bot will send you a link to your own folder on Buzz Chronicles. The thread will be saved in a form of an easy-to-read article
    📁 All your saved threads will be available at buzzchronicles.com/your_twitter_handle
Buzz Chronicles
  • Explore
  • Threads
  • Daily Charts
  • Authors
  • Categories
  • About
  • Terms of Service

Copyright © 2021 Buzz Chronicles - All right reserved