Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ

The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.
They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.
When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.
Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.
The time-to-target-validation is explained purely by optimization considerations. There's a regime where variance dominates, and you get linear speedups w/ batch size. Then there's a regime where curvature dominates and larger batches don't help. As theory would predict.
Incidentally, this paper must have been absurdly expensive, even by Google's standards. Doing careful empirical work on optimizers requires many, many runs of the algorithm. (I think surprising phenomena on ImageNet are often due to the difficulty of running proper experiments.)

More from Machine learning

10 PYTHON 🐍 libraries for machine learning.

Retweets are appreciated.
[ Thread ]


1. NumPy (Numerical Python)

- The most powerful feature of NumPy is the n-dimensional array.

- It contains basic linear algebra functions, Fourier transforms, and tools for integration with other low-level languages.

Ref:
https://t.co/XY13ILXwSN


2. SciPy (Scientific Python)

- SciPy is built on NumPy.

- It is one of the most useful libraries for a variety of high-level science and engineering modules like discrete Fourier transform, Linear Algebra, Optimization, and Sparse matrices.

Ref: https://t.co/ALTFqM2VUo


3. Matplotlib

- Matplotlib is a comprehensive library for creating static, animated, and interactive visualizations in Python.

- You can also use Latex commands to add math to your plot.

- Matplotlib makes hard things possible.

Ref: https://t.co/zodOo2WzGx


4. Pandas

- Pandas is for structured data operations and manipulations.

- It is extensively used for data munging and preparation.

- Pandas were added relatively recently to Python and have been instrumental in boosting Python’s usage.

Ref: https://t.co/IFzikVHht4

You May Also Like

The UN just voted to condemn Israel 9 times, and the rest of the world 0.

View the resolutions and voting results here:

The resolution titled "The occupied Syrian Golan," which condemns Israel for "repressive measures" against Syrian citizens in the Golan Heights, was adopted by a vote of 151 - 2 - 14.

Israel and the U.S. voted 'No'
https://t.co/HoO7oz0dwr


The resolution titled "Israeli practices affecting the human rights of the Palestinian people..." was adopted by a vote of 153 - 6 - 9.

Australia, Canada, Israel, Marshall Islands, Micronesia, and the U.S. voted 'No' https://t.co/1Ntpi7Vqab


The resolution titled "Israeli settlements in the Occupied Palestinian Territory, including East Jerusalem, and the occupied Syrian Golan" was adopted by a vote of 153 – 5 – 10.

Canada, Israel, Marshall Islands, Micronesia, and the U.S. voted 'No'
https://t.co/REumYgyRuF


The resolution titled "Applicability of the Geneva Convention... to the
Occupied Palestinian Territory..." was adopted by a vote of 154 - 5 - 8.

Canada, Israel, Marshall Islands, Micronesia, and the U.S. voted 'No'
https://t.co/xDAeS9K1kW
॥ॐ॥
अस्य श्री गायत्री ध्यान श्लोक:
(gAyatri dhyAna shlOka)
• This shloka to meditate personified form of वेदमाता गायत्री was given by Bhagwaan Brahma to Sage yAgnavalkya (याज्ञवल्क्य).

• 14th shloka of गायत्री कवचम् which is taken from वशिष्ठ संहिता, goes as follows..


• मुक्ता-विद्रुम-हेम-नील धवलच्छायैर्मुखस्त्रीक्षणै:।
muktA vidruma hEma nIla dhavalachhAyaiH mukhaistrlkShaNaiH.

• युक्तामिन्दुकला-निबद्धमुकुटां तत्वार्थवर्णात्मिकाम्॥
yuktAmindukalA nibaddha makutAm tatvArtha varNAtmikam.

• गायत्रीं वरदाभयाङ्कुश कशां शुभ्रं कपालं गदाम्।
gAyatrIm vardAbhayANkusha kashAm shubhram kapAlam gadAm.

• शंखं चक्रमथारविन्दयुगलं हस्तैर्वहन्ती भजै॥
shankham chakramathArvinda yugalam hastairvahantIm bhajE.

This shloka describes the form of वेदमाता गायत्री.

• It says, "She has five faces which shine with the colours of a Pearl 'मुक्ता', Coral 'विद्रुम', Gold 'हेम्', Sapphire 'नील्', & a Diamond 'धवलम्'.

• These five faces are symbolic of the five primordial elements called पञ्चमहाभूत:' which makes up the entire existence.

• These are the elements of SPACE, FIRE, WIND, EARTH & WATER.

• All these five faces shine with three eyes 'त्रिक्षणै:'.