Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ

The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.

They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.

When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.

Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.

1 There's a chasm between an NLP technology that works well in the research lab and something that works for applications that real people use. This was eye-opening when I started my career, and every time I talk to an NLP engineer at @textio, it continues to strike me even now.

2 Research conditions are theoretical and/or idealized. A huge problem for so-called NLP or AI startups with highly credentialed academic founders is that they bring limited knowledge of what it takes to build real products outside the lab.

3 A product is ultimately a thing that people pay for - not just cool technology or user experience. But I’m not even talking about knowledge gaps in go-to-market work. I'm talking purely technical gaps: how you go from science project to performant + delightful user experience.

4 Most commoditized NLP packages solve well-understood problems in standard ways that sacrifice either precision or performance. In a research lab, this is not usually a hard trade-off; in general, no one is using what you make, so performance is less important than precision.

5 In software, when you’re making something for real people to use, these tradeoffs are a big deal. Especially if you’re asking those people to pay for what you’ve made (can’t get away from that pesky GTM thinking). They expect quality, which includes precision AND performance.

Are you planning to learn to Python for machine learning this year?

Here's everything you need to get started.
🧵👇

In this thread we'll look at all the concepts in Python you need to know for machine learning along with free resources to help you out.

All of this is based on my experience of successfully teaching 300+ students how to code using Python.

(2 / 19

You can use many languages for machine learning, why Python?

Because of 2 reasons:
- Comparatively easier to learn than other languages
- Has the biggest and most mature community

This makes Python a no-brainer to learn for machine learning as a beginner.

(3 / 19)

These are the absolute basics which you must know about:

- Basic terminal commands
- Basic arithmetic (+,-,/,*)
- Accepting user input
- For & While loops
- Exception handling
- If-Else statements
- Functions, modules & Imports

(4 / 19)

Then comes the more tougher concepts which you must know about:

- Object oriented programming in Python:Classes, Objects, Methods
- PIP (Pypi)
- List slicing
- String formatting
- Dictionaries & Tuples
- Managing environments
- Dunder methods like __init__

(5 / 19)

Are you planning to learn to Python for machine learning this year?

Here's everything you need to get started.
🧵👇

In this thread we'll look at all the concepts in Python you need to know for machine learning along with free resources to help you out.

All of this is based on my experience of successfully teaching 300+ students how to code using Python.

(2 / 19

You can use many languages for machine learning, why Python?

Because of 2 reasons:
- Comparatively easier to learn than other languages
- Has the biggest and most mature community

This makes Python a no-brainer to learn for machine learning as a beginner.

(3 / 19)

These are the absolute basics which you must know about:

- Basic terminal commands
- Basic arithmetic (+,-,/,*)
- Accepting user input
- For & While loops
- Exception handling
- If-Else statements
- Functions, modules & Imports

(4 / 19)

Then comes the more tougher concepts which you must know about:

- Object oriented programming in Python:Classes, Objects, Methods
- PIP (Pypi)
- List slicing
- String formatting
- Dictionaries & Tuples
- Managing environments
- Dunder methods like __init__

(5 / 19)

The 8-step quick-start guide to learn Machine Learning.

🧵👇

1⃣ Start with Python 🐍

Yes, you can do other languages, but Python is by far the most straightforward option.

👇

2⃣ Get familiar with numpy, pandas, and matplotlib

These three libraries are probably the most common Python libraries you'll have to use every day.

(Even if you don't end up doing machine learning, these libraries are awesome and useful.)

👇

3⃣ Start using notebooks

Look into Jupyter or Google Colab.

Notebooks are essential for data scientists and machine learning practitioners. Most of the code you'll read and write will be in notebooks.

👇

4⃣ Find a problem (already solved)

In my opinion, the best way to start is by working through a problem —especially when you can learn from its solution.

Start with something simple. I usually recommend "Titanic" from Kaggle.

👇

The 8-step quick-start guide to learn Machine Learning.

🧵👇

1⃣ Start with Python 🐍

Yes, you can do other languages, but Python is by far the most straightforward option.

👇

2⃣ Get familiar with numpy, pandas, and matplotlib

These three libraries are probably the most common Python libraries you'll have to use every day.

(Even if you don't end up doing machine learning, these libraries are awesome and useful.)

👇

3⃣ Start using notebooks

Look into Jupyter or Google Colab.

Notebooks are essential for data scientists and machine learning practitioners. Most of the code you'll read and write will be in notebooks.

👇

4⃣ Find a problem (already solved)

In my opinion, the best way to start is by working through a problem —especially when you can learn from its solution.

Start with something simple. I usually recommend "Titanic" from Kaggle.

👇

20 machine learning questions that will make you think.

(Cool questions. Not the regular, introductory stuff that you find everywhere.)

🧵👇

1. Why is it important to introduce non-linearities in a neural network?

2. What are the differences between a multi-class classification problem and a multi-label classification problem?

3. Why does the use of Dropout work as a regularizer?

👇

4. Why you shouldn't use a softmax output activation function in a multi-label classification problem when using a one-hot-encoded target?

5. Does the use of Dropout in your model slow down or speed up the training process? Why?

👇

6. In a Linear or Logistic Regression problem, do all Gradient Descent algorithms lead to the same model, provided you let them run long enough?

7. Explain the difference between Batch Gradient Descent, Stochastic Gradient Descent, and Mini-batch Gradient Descent.

👇

8. What are the advantages of Convolution Neural Networks (CNN) over a fully connected network for image classification?

9. What are the advantages of Recurrent Neural Networks (RNN) over a fully connected network when working with text data?

👇

How to learn Python for machine learning. 🐍

The beginner's guide.
(based on my real life experience of mentoring 300+ students)

🧵👇

In this thread we'll look at all the concepts in Python you need to know for machine learning along with all the free resources that you'll need.

All of this is based on my experience of successfully teaching 300+ students how to code using Python.

(2 / 19)

You can use many languages for machine learning, why Python?

Because of 2 reasons:
- Comparatively easier to learn than other languages
- Has the biggest and most mature community

This makes Python a no-brainer to learn for machine learning as a beginner.

(3 / 19)

These are the absolute basics which you must know about:

- Basic terminal commands
- Basic arithmetic (+,-,/,*)
- Accepting user input
- For & While loops
- Exception handling
- If-Else statements
- Functions, modules & Imports

(4 / 19)

Then comes the more tougher concepts which you must know about:

- Object oriented programming in Python:Classes, Objects, Methods
- PIP (Pypi)
- List slicing
- String formatting
- Dictionaries & Tuples
- Managing environments
- Dunder methods like __init__

(5 / 19)

These are the tools you will need for machine learning in Python.

🧵👇

Anaconda

When you work in python, you'll be working with several frameworks and many of them work only on specific versions of python.

(2 / 13)

Now imagine downloading a new version of python and then installing it for every framework you want to work with 😬.

Meet Anaconda which allows you to run several versions of python. It comes pre-installed with several data science and machine learning frameworks.

(3 / 13)

Pip-env is also a way of maintaining several versions of Python and comes pre installed with Python.
You can use pip env or Anaconda, whichever works for you.

(4 / 13)

Jupyter Notebooks

Jupyter notebooks is an IDE just like VS code or Sublime. The special thing about jupyter is that you can parts of code in mini code editors called cells. This is great for prototyping and testing code.

(5 / 13)

Here's everything you need to know about the math for machine learning.

(+ free resources )
🧵👇

Before diving into the math, I suggest first having solid programming skills.

For example👇

(2 / 18)

In Python, these are the concepts which you must know:

- Object oriented programming in Python : Classes, Objects, Methods
- List slicing
- String formatting
- Dictionaries & Tuples
- Basic terminal commands
- Exception handling

(3 / 18)

If you want to learn those concepts for python, these courses are freecodecamp could be of help to you.

🔗Basics:youtube∙com/watch?v=rfscVS0vtbw
🔗Intermediate :youtube∙com/watch?v=HGOBQPFzWKo

(4 / 18)

👉You need to have really strong fundamentals in programming, because machine learning involves a lot of it.

It is 100% compulsory.

(5 / 18)

11 key concepts of Machine Learning.

— Supervised Learning Edition —

🧵👇

😜

Before starting, remember that, if you follow me, one of your enemies will be immediately destroyed (and you'll get to read more of these threads, of course.)

And if you don't follow me, well, you just hurt my feelings.

😜

1. Labels

(Also referred to as "y")

The label is the piece of information that we are predicting.

For example:

- the animal that's shown in a picture
- the price of a house
- whether a message is spam or not

👇

2. Features

(Also referred to as "x")

These are the input variables to our problem. We use these features to predict the "label."

For example:

- pixels of a picture
- number of bedrooms of a house
- square footage of a house

👇

3. Samples

(This is also known as "examples.")

A sample is a particular instance of data (features or "x.") It could be "labeled" or "unlabeled."

👇

Here's how I got started with my first machine learning project and you can too.
Let's take a look.

(This thread will take from zero to being a hero in machine learning, trust me)

(1 / 22)
🧵👇

Getting started with your first machine learning project might actually much easier than it seems, if I can do it, certainly anyone can.

I did not use:
- Any Math
- An expensive computer
- Complex programming concepts

(2 / 22)

Here's what I did use:

- A free GPU on Google Colab
- Python
- TensorFlow
- Numpy
- Pandas
- Kaggle
- Scikit-Learn
- Google
- StackOverFlow

(3 / 22)

This project was actually a Kaggle challenge based on the MNIST dataset which is a collection images of 70,000 hand written digits.

You can find the dataset here👇
🔗//kaggle.com/c/digit-recognizer

(4 / 22)

Before we go over the code of this project, it is highly reccomended that you complete this free course on YouTube👇

Machine Learning foundations course
🔗//youtu.be/_Z9TRANg4c0

Code👇
🔗//colab.research.google.com/github/PrasoonPratham/Kaggle/blob/main/MNIST.ipynb

(5 / 22)

Okay, doing my first baby steps with r2frida (which combines the power of @radareorg and @fridadotre).

Gonna share my progress in this thread (live, so keep calm).

The goal: Runtime inspection of data sent out by TikTok !!before!! it gets encrypted

1/many

First of all, we do not start from zero. I got some prior knowledge from past reversing attempts and want to share some important facts.

TikTok's (log data) encryption is accomplished by a native library. The Android Java code just serves as proxy function to the native function

The decompiled code for the respective native JNI function of an older TikTok version looks something like this, but in this example I use the most current TT version (no statical analysis done, yet)

In case you never reversed native libraries which were build to interface with Android Java layer via JNI, I highly suggest the entry level introduction on the topic by @maddiestone

Before we start, I want to pinpoint some important aspects (which are also covered by Maddie's videos).

1) Unlike raw C-functions, JNI functions like the one showcased above, receive pointers to complex Java objects .

F.e. a function receiving a String on the Java layer...

1/ It's probably not the first thing you think of, but when we started .NET (COM+) in the late 90s, C# didn't exist yet. We were working on it at the same time as the CLR and the framework. So, you might wonder, what language was being used to generate IL and write the BCL?

2/ The answer is a language that we called SMC that Peter Kukol wrote the compiler for. Peter is a flat out amazing engineer and wrote the core parts of the compiler in just a few days. This unblocked the framework team, allowed vetting the runtime and interpreter, etc.

3/ SMC was a trimmed down C++ variant and the compiler was written in itself (i.e. SMC). It didn't support things like destructors, multiple inheritance, virtual base classes, etc. But, overall it enabled progress that would have otherwise been stalled.

4/ It did introduce a problem for the C# language and compiler team though. Every day more and more code was written in SMC, a language that we did not intend to ship externally. Designing a new language is hard (well, at least, a general purpose one that becomes popular).

5/ So there was a constant balancing act between laying enough of the foundation of the language and the C# compiler to replace SMC and making sure that the design lived up to the goals and ideals the team was striving for. If we waited too long to adopt C#,

A simple guide on how to train your first neural network in under 10 minutes to kick-start your machine learning journey.

(+ no setup required and free extra resources)
🧵👇

For this exercise, you'll need :
- A computer/phone
- An internet connection
- Very basic Python knowledge
- Willingness to learn

What you do not need:
- Complex math
- An expensive computer
- A PhD

Math will come later on in your machine learning journey :)

(2 / 15)

Here's our problem,we are given data in which we are given the number of flats in a house and its corresponding price. Like a house with one flat is worth 10000, and 20000 for a house with 2 flats.

(3 / 15)

We can clearly tell that the price of the house increases by 10000 per extra flat however our computer does not know this and we won't it tell it about this, it'll have to figure things out on its own.

(4 / 15)

Here's the code👇

(5 / 15)

Machine Learning for the Web developer in 2021.

The beginner's guide.​

🧵👇

I started machine learning as a web developer, if I can do it then anyone can.

This carefully curated thread will give you key insights into my journey and how you can make this transition, seamlessly.

(2 / 14)

"Machine learning is not what you think it is"

One of the main reasons why people find it difficult to get started with machine learning is because of the lack of information, and rightfully so.

(3 / 14)

Machine learning as a concept has existed since the 1950s, but has only become popular in recent years because of the exponential rise of advancements in computer hardware.

(4 / 14)

In short, it because of the sudden rise of this technology ,which was previously unknown to the general public, that there is a lot of misinformation around it.

(5 / 14)