So the cryptocurrency industry has basically two products, one which is relatively benign and doesn't have product market fit, and one which is malignant and does. The industry has a weird superposition of understanding this fact and (strategically?) not understanding it.

Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ

The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.

They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.

When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.

Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.

A thread on HN about bad code in legacy projects both makes me think how little we've learned as a discipline over the years and, honestly, how little credit we give ourselves for some pretty major

Fun going down this list and thinking: "Hmm, plausible at a well-run modern software shop", "Hmm, possible, but requires implausible tradeoffs", "Literally disallowed by languages", and "If you were to attempt doing that our test suite wouldn't let you merge."

I think we as an industry celebrate (not quite the right word) failure too much and don't celebrate success nearly enough. There is no DailyWTF for competent execution, word of which generally stays pretty local to the source while incompetence passes into legend.

Alrighty let me try to thread the needle on being the change I want to see in the world while not giving away anything that will get me in trouble:

Ruby has wonderful developer ergonomics. Typed languages are easier for machines to guarantee the correctness of. We built a type checker for Ruby (and I believe it is slated for OSS release sometime).

c.f.

(A THREAD)

18 Things I Would Do At 18:

I was a dumb 18-year-old. I thought I had the world figured out. I didn't.

I stayed up till the wee hours. Drank every liquor invented.

Partied non-stop. Woke up after noon.

Surfed the web aimlessly.

Read BS self-help.

Repeat.

If I could do it all over again, here's 18 things I would do...

PHASE ONE: GET MY MIND RIGHT

ACTION #1: Break the chains - I wish I made the decision earlier. I didn't. I was scared. Anxious. Nervous. I knew once I made the decision, there was no going back.

Don't be like me.

Make the decision at 18. The decision to pave your own path in life. To go against the herd. To choose your own adventure. To commit to whatever sets your heart on fire.

ACTION #2: Cut off every negative person in your life.

Scroll through the contacts in your phone...

Notice how the names make you feel.

Delete any and all names that fill you with doubt and negativity.

ACTION #3: Condition your mind to embrace the producer mindset instead of the consumer mindset.

Consume 20% of the time. Produce 80% of the time.

Case closed.

ACTION #4: Set an intention to make $10k a month and write it out every MORNING and NIGHT as if you already achieved it (in the past tense).

Visualize yourself already having the money as you write.

Cheesy?

Yes.

Does it work?

Yes.

PHASE TWO: EDUCATION