I’ve got some interesting responses to this so I’ll like to demystify a commonly used but misleading saying!

If you study the Bible to extract the nature of God, you will most definitely find that God prioritizes “efficiency” and is NOT wasteful about resource allocation

THREAD

Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ

The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.

They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.

When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.

Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.

Observing the public conversation around FB, and the private ones happening among techies and ex-FBers, I think the mutual misunderstanding is worse than when I set out two years (and 500 pages) ago to (in a small way) bridge that gulf.

We're basically fucked.

The tech world has gotten so huge, self-reinforcing, and insulated from reality they can no longer even vaguely look at themselves (and their actions) as others do. They just live on a different planet than most people.

Conversely, the average tech consumer doesn't understand the technology that has slowly taken over their lives, and their designated emissaries to figure it out--politicians, pundits, regulators, journalists--understand it barely better than they do, and have their own agendas.

To say more than generalities for a moment, here's what I think is likely the core problem.

Techies take weird, improbable visions, and make them realities: some BS pitch deck to a VC, mixed with money and people, really does turn into some novel thing.

Most people work inside a legacy industry that's evolved that way over time (usually for good reasons), and they think about the future via some analogy with their present (which is a function of a long-ago past). The interruption that tech will introduce is often hard to grasp.

"How the Avengers would respond to the Capitol riot" (a thread)

CAPTAIN AMERICA - tells Falcon, "I fought Nazis once before. They're extremely dangerous." They then see the guy scaling the Capitol wall fall into the shrubs. Falcon: "You sure these are the same Nazis?"

IRON MAN - lands by the guy carrying the podium. Says, "Mind if I borrow this?" then tosses it to Hulk, who swings it like a bat at the mob. Guy: "Hey, that's MY podium!" Iron Man: "Actually it's the Speaker's. And technically, it's a lectern." Then punches him through a wall.

BLACK WIDOW - enters hotel room where Jack Dorsey is tied up. Rips duct tape off his mouth. Jack: "I'll do anything you want!" BW: "Disable POTUS's Twitter. NOW!" Jack fumbles with his phone. "Done. Anything else?" BW: "Yeah. Trim your beard for God's sake." *tosses him clippers*

HULK - grabs a selfie stick from one of the guys and beats him over the head with it. Deeming it too small, he grabs another guy's Confederate flag and takes out a whole swath of rioters. He then tosses the flag to Cap, who uses it as a javelin to impale a fleeing Ted Cruz.

HAWKEYE - hears Iron Man in his earpiece say, "Looks like Orange Julius is tweeting from another account." Hawkeye: "On it." He locks in on Trump through the White House window and fires an arrow, knocking the phone out of Trump's hands. Hawkeye: "Damn that felt good."