#NavinFluorine making steady progress @caniravkaria

Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ

The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.

They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.

When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.

Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.

I present to you all, my (possibly) superthread On how and why the Unknown God from Genshin Impact, and The Herrscher of Void/God Kiana/Sirin from Honkai Impact!

First off! We’ll start with the obvious things that constitute a possible connection.

They share the same Japanese Voice actress! Rie Kugimiya.
Usually this is a stretch reason, but considering everything else, it’s not unlikely that it’s the same character.

Secondly, their Appearances.
The both have very long white hair with identical bang style, and a very noticeable small piece of hair on top that curls upwards. That’s a defining part Of all of Kiana’s hairstyles, and for it to be shared with this god says something.

Another notable thing is their eyes. They both have Glowing golden eyes with very similar appearances. They contain a ring, and a 4-point star pupil. Resemblant of a scope if you will. That’s The Herrscher of Void’s defining characteristic

If you’d even like to take a look at their outfits, there’s resemblances including the black fur collar, the designs on the torso and chest, and the use of something similar to coat tails/skirt. The Herrscher can change their outfit and they aren’t stuck with one. Sirin does this

Some people really benefit from hearing advice that everyone knows, for the same reason we keep schools open despite every subject in them having been taught before.

In that spirit, here's some quick Things Many People Find Too Obvious To Have Told You Already.

Your idea is not valuable, at all. All value is in the execution. You think you are an exception; you are not. You should not insist on an NDA to talk about it; nobody serious will engage in contract review over an idea, and this will mark you as clueless.

Technologists tend to severely underestimate the difficulty and expense of creating software, especially at companies which do not have fully staffed industry leading engineering teams ("because software is so easy there, amirite guys?")

Charge more. Charge more still. Go on.

The press is a lossy and biased compression of events in the actual world, and is singularly consumed with its own rituals, status games, and incentives. The news necessarily fails to capture almost everything which happened yesterday. What it says is important usually isn't.

Companies find it incredibly hard to reliably staff positions with hard-working generalists who operate autonomously and have high risk tolerances. This is not the modal employee, including at places which are justifiably proud of the skill/diligence/etc of their employees.

1/ there is this sense when you are young that your accomplishments need to be a list of things that seem impressive to others. A list of several items you did.

This isn't actually right, so here is another suggestion.

2/ I remember being 26 and writing about reading 52 books a year. I wrote blog posts about it. They got copied. It became "a thing." Now it's in Twitter bios. It looks impressive but it's insanely useless and I shouldn't have done it.

3/ what I should have known at that time is that only young idiots like myself, with no accomplishments, find list of tiny achievements impressive. Anyone who has actually done anything of substance doesn't gaf

4/ what is actually difficult, and worthwhile, instead is to do ONE single thing for a very, very long time. It's much harder and much rarer and results in outlier outcomes much more often.

Of course you can find this out too late if you are chasing the dragon of Ted talks etc

5/ if I had only worked on a startup for a year, I would've gotten nowhere, the same way that if you lift for 3 months, it achieves nothing. Everything good in life comes from perseverance, but at the beginning, you're just like "I need to be somebody!!!"