BC MACHINE LEARNING

Saved by @zmbnski

Roger Grosse
@RogerGrosse 10 months, 4 weeks ago 1286 views

Share See On Twitter

Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ

The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.

They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.

When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.

Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.

The time-to-target-validation is explained purely by optimization considerations. There's a regime where variance dominates, and you get linear speedups w/ batch size. Then there's a regime where curvature dominates and larger batches don't help. As theory would predict.

Incidentally, this paper must have been absurdly expensive, even by Google's standards. Doing careful empirical work on optimizers requires many, many runs of the algorithm. (I think surprising phenomena on ImageNet are often due to the difficulty of running proper experiments.)

More from Machine learning

Advertisement

Kieran Snyder
@KieranSnyder

1 There's a chasm between an NLP technology that works well in the research lab and something that works for applications that real people use. This was eye-opening when I started my career, and every time I talk to an NLP engineer at @textio, it continues to strike me even now.

2 Research conditions are theoretical and/or idealized. A huge problem for so-called NLP or AI startups with highly credentialed academic founders is that they bring limited knowledge of what it takes to build real products outside the lab.

3 A product is ultimately a thing that people pay for - not just cool technology or user experience. But I’m not even talking about knowledge gaps in go-to-market work. I'm talking purely technical gaps: how you go from science project to performant + delightful user experience.

4 Most commoditized NLP packages solve well-understood problems in standard ways that sacrifice either precision or performance. In a research lab, this is not usually a hard trade-off; in general, no one is using what you make, so performance is less important than precision.

5 In software, when you’re making something for real people to use, these tradeoffs are a big deal. Especially if you’re asking those people to pay for what you’ve made (can’t get away from that pesky GTM thinking). They expect quality, which includes precision AND performance.

John Burn-Murdoch
@jburnmurdoch

Really enjoyed digging into recent innovations in the football analytics industry.

>10 hours of interviews for this w/ a dozen or so of top firms in the game. Really grateful to everyone who gave up time & insights, even those that didnt make final cut 🙇‍♂️ https://t.co/9YOSrl8TdN

For avoidance of doubt, leading tracking analytics firms are now well beyond voronoi diagrams, using more granular measures to assess control and value of space.

This @JaviOnData & @LukeBornn paper from 2018 referenced in the piece demonstrates one method https://t.co/Hx8XTUMpJ5

Bit of this that I nerded out on the most is "ghosting" — technique used by @counterattack9 & co @stats_insights, among others.

Deep learning models predict how specific players — operating w/in specific setups — will move & execute actions. A paper here: https://t.co/9qrKvJ70EN

So many use-cases:
1/ Quickly & automatically spot situations where opponent's defence is abnormally vulnerable. Drill those to death in training.
2/ Swap target player B in for current player A, and simulate. How does target player strengthen/weaken team? In specific situations?

You May Also Like

Sam Rocha
@SamRochadotcom

THREAD

I get asked for writing tips a lot. Of all the ones I’ve given, here are a few of the ones that seem to hold true and remain somewhat universal or at least mostly unobjectionable:

#amwriting #writingtips

1. If you are a fast writer (i.e., if you compose first drafts very quickly) then you absolutely must become a slow editor. Much of what I call “writing” is really, for me, editing.

2. If you don’t read you cannot write a lot or well. Sure, maybe you can read like a maniac for a decade and then read less after that, but without some large volume of intake, there will never be a meaningful output.

2.1 I sometimes say that I feel like writing without reading is like trying to run a marathan on a cracker. And sadly I see many students struggling to write when they have simply never read enough to fuel that demand of the task at hand.

3. All writing is writing if one decides to treat it that way. If one takes some degree of intentional will when sending emails, posting on social media and writing notes and marginalia, then all of that can count as meaningful daily writing.

John Papa
@John_Papa

Choosing the Right JavaScript Framework

Thanks for joining me @pluralsight to talk about javascript frameworks. We had 150+ awesome questions, so I'll try to answer the ones I couldn't get to online right here in a tweet stream

Watch the video -->
https://t.co/G7zeiKhXme

Q. Do arrow functions really make dealing with JavaScript's "this" keywords easier?

A. Yes! When using arrow functions, the scope of "this" is effectively passed in from the outer scope. See here in this REPL

1/n

Q. Where does Polymer fit in? Is it considered a framework like the others?

A. @Polymer is an excellent way to create web components, which are ideal for creating sets of interactivity that can be shared in web apps.

2/n

Q. Have we gone framework crazy?

A. At one time, quite possibly. A few years back we literally had framework overload. I feel it really has solidified into a few very strong choices.

3/n

Q. [Is] there no more need for @polymer?

A. See my previous question on this ... but in general, I think there is a lot of room for Web Components with frameworks.

4/n

Advertisement

Advertisement

Delilah S. Dawson
@DelilahSDawson

Hello, writers! How about a thread on LITERARY AGENTS? How to get one, how to talk to one, how to keep one happy. Spoiler alert: I am 100% pro agent and can't imagine not having one on my side. 1/

Why would you want a literary agent?
* you want to be traditionally published
* you want someone experienced to help guide your career
* you want to learn how to edit like a pro
* you want to sell foreign and movie rights
* you want answers to your newbie questions 2/

Why you might *NOT* want a literary agent:
* you want to self publish
* you're not willing to compromise on your edits
* you don't think their expertise is worth 15% of your advance
I... can't think this way. Literary agents have been crucial to my career. 3/

So, how do you get a literary agent?
1. Have a finished, revised, edited, polished manuscript.
2. Write a query letter for your book
3. Send your query to agents who rep your genre and are open to submissions
4. Repeat steps 1-4 until you're offered representation. 4/

So, let's go through those four steps. First of all, you must have a finished, revised, edited, polished book, and it must be sellable. That is, you can't sell a 600k picture book or a 40k adult Fantasy, etc. You must read extensively in the genre you're writing. 5/

Andrew Chen
@andrewchen

👇Thread.

Published a new essay: The red flags and magic numbers that investors look for in your startup’s metrics – 80 slide deck included!

This was a deck that I created on my (longish) interview process with @a16z. It was a long path, starting with meeting folks at the firm 10 years ago. But the purpose of the deck was to explain how I would use my superpower in an investing context

Here's what I explain in the deck. As investors (whether angel or VC) we're often confronted with an up-and-to-the-right graph. Is it going to go up? Or down?

One solution to forecast these growth curves is the Growth Accounting Framework, where you add up New+Reactivated and subtract churned users. In each time period that gives you the difference in monthly actives.

The problem with this is that it's a lagging metric, not a leading one. We need to go one level deeper and look at the underlying loops that drive these numbers, to understand the quality.