Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ

The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.

They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.

When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.

Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.

1 There's a chasm between an NLP technology that works well in the research lab and something that works for applications that real people use. This was eye-opening when I started my career, and every time I talk to an NLP engineer at @textio, it continues to strike me even now.

2 Research conditions are theoretical and/or idealized. A huge problem for so-called NLP or AI startups with highly credentialed academic founders is that they bring limited knowledge of what it takes to build real products outside the lab.

3 A product is ultimately a thing that people pay for - not just cool technology or user experience. But I’m not even talking about knowledge gaps in go-to-market work. I'm talking purely technical gaps: how you go from science project to performant + delightful user experience.

4 Most commoditized NLP packages solve well-understood problems in standard ways that sacrifice either precision or performance. In a research lab, this is not usually a hard trade-off; in general, no one is using what you make, so performance is less important than precision.

5 In software, when you’re making something for real people to use, these tradeoffs are a big deal. Especially if you’re asking those people to pay for what you’ve made (can’t get away from that pesky GTM thinking). They expect quality, which includes precision AND performance.

20 machine learning questions that will make you think.

(Cool questions. Not the regular, introductory stuff that you find everywhere.)

🧵👇

1. Why is it important to introduce non-linearities in a neural network?

2. What are the differences between a multi-class classification problem and a multi-label classification problem?

3. Why does the use of Dropout work as a regularizer?

👇

4. Why you shouldn't use a softmax output activation function in a multi-label classification problem when using a one-hot-encoded target?

5. Does the use of Dropout in your model slow down or speed up the training process? Why?

👇

6. In a Linear or Logistic Regression problem, do all Gradient Descent algorithms lead to the same model, provided you let them run long enough?

7. Explain the difference between Batch Gradient Descent, Stochastic Gradient Descent, and Mini-batch Gradient Descent.

👇

8. What are the advantages of Convolution Neural Networks (CNN) over a fully connected network for image classification?

9. What are the advantages of Recurrent Neural Networks (RNN) over a fully connected network when working with text data?

👇

Are you planning to learn to Python for machine learning this year?

Here's everything you need to get started.
🧵👇

In this thread we'll look at all the concepts in Python you need to know for machine learning along with free resources to help you out.

All of this is based on my experience of successfully teaching 300+ students how to code using Python.

(2 / 19

You can use many languages for machine learning, why Python?

Because of 2 reasons:
- Comparatively easier to learn than other languages
- Has the biggest and most mature community

This makes Python a no-brainer to learn for machine learning as a beginner.

(3 / 19)

These are the absolute basics which you must know about:

- Basic terminal commands
- Basic arithmetic (+,-,/,*)
- Accepting user input
- For & While loops
- Exception handling
- If-Else statements
- Functions, modules & Imports

(4 / 19)

Then comes the more tougher concepts which you must know about:

- Object oriented programming in Python:Classes, Objects, Methods
- PIP (Pypi)
- List slicing
- String formatting
- Dictionaries & Tuples
- Managing environments
- Dunder methods like __init__

(5 / 19)

The 8-step quick-start guide to learn Machine Learning.

🧵👇

1⃣ Start with Python 🐍

Yes, you can do other languages, but Python is by far the most straightforward option.

👇

2⃣ Get familiar with numpy, pandas, and matplotlib

These three libraries are probably the most common Python libraries you'll have to use every day.

(Even if you don't end up doing machine learning, these libraries are awesome and useful.)

👇

3⃣ Start using notebooks

Look into Jupyter or Google Colab.

Notebooks are essential for data scientists and machine learning practitioners. Most of the code you'll read and write will be in notebooks.

👇

4⃣ Find a problem (already solved)

In my opinion, the best way to start is by working through a problem —especially when you can learn from its solution.

Start with something simple. I usually recommend "Titanic" from Kaggle.

👇

How to learn Python for machine learning. 🐍

The beginner's guide.
(based on my real life experience of mentoring 300+ students)

🧵👇

In this thread we'll look at all the concepts in Python you need to know for machine learning along with all the free resources that you'll need.

All of this is based on my experience of successfully teaching 300+ students how to code using Python.

(2 / 19)

You can use many languages for machine learning, why Python?

Because of 2 reasons:
- Comparatively easier to learn than other languages
- Has the biggest and most mature community

This makes Python a no-brainer to learn for machine learning as a beginner.

(3 / 19)

These are the absolute basics which you must know about:

- Basic terminal commands
- Basic arithmetic (+,-,/,*)
- Accepting user input
- For & While loops
- Exception handling
- If-Else statements
- Functions, modules & Imports

(4 / 19)

Then comes the more tougher concepts which you must know about:

- Object oriented programming in Python:Classes, Objects, Methods
- PIP (Pypi)
- List slicing
- String formatting
- Dictionaries & Tuples
- Managing environments
- Dunder methods like __init__

(5 / 19)

These are the tools you will need for machine learning in Python.

🧵👇

Anaconda

When you work in python, you'll be working with several frameworks and many of them work only on specific versions of python.

(2 / 13)

Now imagine downloading a new version of python and then installing it for every framework you want to work with 😬.

Meet Anaconda which allows you to run several versions of python. It comes pre-installed with several data science and machine learning frameworks.

(3 / 13)

Pip-env is also a way of maintaining several versions of Python and comes pre installed with Python.
You can use pip env or Anaconda, whichever works for you.

(4 / 13)

Jupyter Notebooks

Jupyter notebooks is an IDE just like VS code or Sublime. The special thing about jupyter is that you can parts of code in mini code editors called cells. This is great for prototyping and testing code.

(5 / 13)

Can you get a job in data science and machine learning without a college degree?

🧵👇

Short Answer: Yes.

Long Answer, keep reading 👇

(Advice from industry experts who I talked to.)

Companies are looking for people who add value.
In order to add value, you'll need skills. Simple as that.

Get the skills to provide value and you'll get the job.

In the age of the internet where everything is pretty much free, why do college degrees matter?

A college degree makes it easier to get the skills to get a job in machine learning or data science.

College degrees also help you make many useful connections and provides many opportunities in the form of internships and whatnot.

College degrees have their own place

Here's how I got started with my first machine learning project and you can too.
Let's take a look.

(This thread will take from zero to being a hero in machine learning, trust me)

(1 / 22)
🧵👇

Getting started with your first machine learning project might actually much easier than it seems, if I can do it, certainly anyone can.

I did not use:
- Any Math
- An expensive computer
- Complex programming concepts

(2 / 22)

Here's what I did use:

- A free GPU on Google Colab
- Python
- TensorFlow
- Numpy
- Pandas
- Kaggle
- Scikit-Learn
- Google
- StackOverFlow

(3 / 22)

This project was actually a Kaggle challenge based on the MNIST dataset which is a collection images of 70,000 hand written digits.

You can find the dataset here👇
🔗//kaggle.com/c/digit-recognizer

(4 / 22)

Before we go over the code of this project, it is highly reccomended that you complete this free course on YouTube👇

Machine Learning foundations course
🔗//youtu.be/_Z9TRANg4c0

Code👇
🔗//colab.research.google.com/github/PrasoonPratham/Kaggle/blob/main/MNIST.ipynb

(5 / 22)

Here's everything you need to know about the math for machine learning.

(+ free resources )
🧵👇

Before diving into the math, I suggest first having solid programming skills.

For example👇

(2 / 18)

In Python, these are the concepts which you must know:

- Object oriented programming in Python : Classes, Objects, Methods
- List slicing
- String formatting
- Dictionaries & Tuples
- Basic terminal commands
- Exception handling

(3 / 18)

If you want to learn those concepts for python, these courses are freecodecamp could be of help to you.

🔗Basics:youtube∙com/watch?v=rfscVS0vtbw
🔗Intermediate :youtube∙com/watch?v=HGOBQPFzWKo

(4 / 18)

👉You need to have really strong fundamentals in programming, because machine learning involves a lot of it.

It is 100% compulsory.

(5 / 18)

Are you planning to learning machine learning this year?

These are the Python frameworks you need to learn as an absolute beginner.

🧵👇

It can be overwhelming seeing all the machine learning frameworks at once and not even knowing what they do.

I made this thread so that your machine learning journey becomes just a little bit easier.

(2 / 11)

This thread will only focus on Python frameworks.

Why?
Because Python is the most common path into machine learning.

( If you have experience with any other language then learning Python will not be difficult at all. )

(3 / 11)

Here's the entire list:

- Pandas
- Numpy
- Matplotlib
- TensorFlow
- PyTorch
- Keras
- Scikit Learn

Let's see what they do.

(4 / 11)

You'll have to interact with csv files or databases on a regular basis to access data in machine learning, pandas helps you do just that.

The cool thing about Pandas is that it actually doesn't change the original data...

(5 / 11

11 key concepts of Machine Learning.

— Supervised Learning Edition —

🧵👇

😜

Before starting, remember that, if you follow me, one of your enemies will be immediately destroyed (and you'll get to read more of these threads, of course.)

And if you don't follow me, well, you just hurt my feelings.

😜

1. Labels

(Also referred to as "y")

The label is the piece of information that we are predicting.

For example:

- the animal that's shown in a picture
- the price of a house
- whether a message is spam or not

👇

2. Features

(Also referred to as "x")

These are the input variables to our problem. We use these features to predict the "label."

For example:

- pixels of a picture
- number of bedrooms of a house
- square footage of a house

👇

3. Samples

(This is also known as "examples.")

A sample is a particular instance of data (features or "x.") It could be "labeled" or "unlabeled."

👇

Which libraries do you really need to get started with Machine Learning and why?

🧵👇

First and foremost, make sure that you have nailed the fundamental concepts of Python because machine learning requires a lot of programming!

(2 / 19)

If you know these topics, then you are good to go with machine learning in Python

- Object-oriented programming in Python:Classes,Objects,Methods
- Lists & List functions
- List comprehension
- List slicing
- String formatting
- List,Dictionaries & Tuples

(3 / 19)

Now, let's understand what popular python machine learning libraries do.

We will talk about👇
- TensorFlow (+ Keras)
- PyTorch
- Pandas
- Numpy
- Matplotlib
- SciKit Learn
- Seaborn

(4 / 19)

Now let's cover the libraries that you have to learn (at least the basics) for machine learning

1. Pandas

Pandas is a python library that allows you to store and read data from spreadsheets ( .csv, .xlsv files ) in structures called Dataframes.

(5 / 19)

Here's what you'll hear when you first get into machine learning.

- Neural networks
- Loss
- Weights
- Biases
- Epochs
- Neurons
- Optimizers
...

It can get very confusing really fast!

Here are some of the terms you should know about.
(I wish I had this before)

🧵👇

These terms won't mean anything unless you know what Machine learning is all about.

> Machine learning is the process of making a program which allows a computer to learn from data.

The data could be anything, images, audio or even text.

In machine learning we use something called a neural network, this is essentially an imitation of the human brain.

> Neural Networks are a digital imitation of the neurons you see in the human brain.

In these neural networks, data flows through them and each neuron (the circle) has a numerical value which will change.

> The value of a neuron gets changes to something which is close to what we want each time the data passes through the neural network.

Think of the neurons as dials on a lock, you have to tune every dial to open the lock.

It is almost impossible for a human to tune thousands of dials like these, but a computer certainly can.

A simple guide on how to train your first neural network in under 10 minutes to kick-start your machine learning journey.

(+ no setup required and free extra resources)
🧵👇

For this exercise, you'll need :
- A computer/phone
- An internet connection
- Very basic Python knowledge
- Willingness to learn

What you do not need:
- Complex math
- An expensive computer
- A PhD

Math will come later on in your machine learning journey :)

(2 / 15)

Here's our problem,we are given data in which we are given the number of flats in a house and its corresponding price. Like a house with one flat is worth 10000, and 20000 for a house with 2 flats.

(3 / 15)

We can clearly tell that the price of the house increases by 10000 per extra flat however our computer does not know this and we won't it tell it about this, it'll have to figure things out on its own.

(4 / 15)

Here's the code👇

(5 / 15)

🤔 Python decorators

What are they? How do you use them?

🧵 Let's find out 👇

1️⃣ Decorator is a function or a class that wraps another function or class modifying its behavior.

So how does that work?

The first thing to know is that everything in Python is an object - functions too

2️⃣ That means they can be passed to another function as an argument or returned from a function

Functions that take other functions as an argument are called higher-order functions

3️⃣ In Python, you can define a function inside other function - such functions are called inner functions

4️⃣ To create a decorator you just need to apply all of that together

log_enter_leave is a decorator.

my_function is the function.

To alter my_function's behavior we reassign it applying log_enter_leave decorator.

For a long time, I didn't understand how to use Virtual Environments in Python 🐍.

If this is just, let's end it here and now: 🧵👇

[2] Virtual Environments let you deal with the dependencies that your code has with external Python libraries.

It avoids having conflicts when your projects depend on different versions of the same library.

👇

[3] Let's imagine that you are building your first Python project and you install the "requests" library:

pip install requests

You get version 2.24.0 installed in your system.

👇

[4] A month later, you decide to work on your second project. It also needs the "requests" library.

But the latest version is not 2.24.0 anymore.

Now version 3 is available, and that's the one you want to use!

👇

[5] You could upgrade your entire system to version 3, but then you'll be potentially breaking the first project you built that depends on 2.24.0!

Can you imagine this happening on a server with many more applications running?

👇