Important paper from Google on large batch optimization. They do impressively careful experiments measuring # iterations needed to achieve target validation error at various batch sizes. The main "surprise" is the lack of surprises. [thread]

https://t.co/7QIx5CFdfJ

The paper is a good example of lots of elements of good experimental design. They validate their metric by showing lots of variants give consistent results. They tune hyperparamters separately for each condition, check that optimum isn't at the endpoints, and measure sensitivity.

They have separate experiments where the hold fixed # iterations and # epochs, which (as they explain) measure very different things. They avoid confounds, such as batch norm's artificial dependence between batch size and regularization strength.

When the experiments are done carefully enough, the results are remarkably consistent between different datasets and architectures. Qualitatively, MNIST behaves just like ImageNet.

Importantly, they don't find any evidence for a "sharp/flat optima" effect whereby better optimization leads to worse final results. They have a good discussion of experimental artifacts/confounds in past papers where such effects were reported.

1 There's a chasm between an NLP technology that works well in the research lab and something that works for applications that real people use. This was eye-opening when I started my career, and every time I talk to an NLP engineer at @textio, it continues to strike me even now.

2 Research conditions are theoretical and/or idealized. A huge problem for so-called NLP or AI startups with highly credentialed academic founders is that they bring limited knowledge of what it takes to build real products outside the lab.

3 A product is ultimately a thing that people pay for - not just cool technology or user experience. But I’m not even talking about knowledge gaps in go-to-market work. I'm talking purely technical gaps: how you go from science project to performant + delightful user experience.

4 Most commoditized NLP packages solve well-understood problems in standard ways that sacrifice either precision or performance. In a research lab, this is not usually a hard trade-off; in general, no one is using what you make, so performance is less important than precision.

5 In software, when you’re making something for real people to use, these tradeoffs are a big deal. Especially if you’re asking those people to pay for what you’ve made (can’t get away from that pesky GTM thinking). They expect quality, which includes precision AND performance.

Are you planning to learn to Python for machine learning this year?

Here's everything you need to get started.
🧵👇

In this thread we'll look at all the concepts in Python you need to know for machine learning along with free resources to help you out.

All of this is based on my experience of successfully teaching 300+ students how to code using Python.

(2 / 19

You can use many languages for machine learning, why Python?

Because of 2 reasons:
- Comparatively easier to learn than other languages
- Has the biggest and most mature community

This makes Python a no-brainer to learn for machine learning as a beginner.

(3 / 19)

These are the absolute basics which you must know about:

- Basic terminal commands
- Basic arithmetic (+,-,/,*)
- Accepting user input
- For & While loops
- Exception handling
- If-Else statements
- Functions, modules & Imports

(4 / 19)

Then comes the more tougher concepts which you must know about:

- Object oriented programming in Python:Classes, Objects, Methods
- PIP (Pypi)
- List slicing
- String formatting
- Dictionaries & Tuples
- Managing environments
- Dunder methods like __init__

(5 / 19)

The 8-step quick-start guide to learn Machine Learning.

🧵👇

1⃣ Start with Python 🐍

Yes, you can do other languages, but Python is by far the most straightforward option.

👇

2⃣ Get familiar with numpy, pandas, and matplotlib

These three libraries are probably the most common Python libraries you'll have to use every day.

(Even if you don't end up doing machine learning, these libraries are awesome and useful.)

👇

3⃣ Start using notebooks

Look into Jupyter or Google Colab.

Notebooks are essential for data scientists and machine learning practitioners. Most of the code you'll read and write will be in notebooks.

👇

4⃣ Find a problem (already solved)

In my opinion, the best way to start is by working through a problem —especially when you can learn from its solution.

Start with something simple. I usually recommend "Titanic" from Kaggle.

👇

20 machine learning questions that will make you think.

(Cool questions. Not the regular, introductory stuff that you find everywhere.)

🧵👇

1. Why is it important to introduce non-linearities in a neural network?

2. What are the differences between a multi-class classification problem and a multi-label classification problem?

3. Why does the use of Dropout work as a regularizer?

👇

4. Why you shouldn't use a softmax output activation function in a multi-label classification problem when using a one-hot-encoded target?

5. Does the use of Dropout in your model slow down or speed up the training process? Why?

👇

6. In a Linear or Logistic Regression problem, do all Gradient Descent algorithms lead to the same model, provided you let them run long enough?

7. Explain the difference between Batch Gradient Descent, Stochastic Gradient Descent, and Mini-batch Gradient Descent.

👇

8. What are the advantages of Convolution Neural Networks (CNN) over a fully connected network for image classification?

9. What are the advantages of Recurrent Neural Networks (RNN) over a fully connected network when working with text data?

👇

How to learn Python for machine learning. 🐍

The beginner's guide.
(based on my real life experience of mentoring 300+ students)

🧵👇

In this thread we'll look at all the concepts in Python you need to know for machine learning along with all the free resources that you'll need.

All of this is based on my experience of successfully teaching 300+ students how to code using Python.

(2 / 19)

You can use many languages for machine learning, why Python?

Because of 2 reasons:
- Comparatively easier to learn than other languages
- Has the biggest and most mature community

This makes Python a no-brainer to learn for machine learning as a beginner.

(3 / 19)

These are the absolute basics which you must know about:

- Basic terminal commands
- Basic arithmetic (+,-,/,*)
- Accepting user input
- For & While loops
- Exception handling
- If-Else statements
- Functions, modules & Imports

(4 / 19)

Then comes the more tougher concepts which you must know about:

- Object oriented programming in Python:Classes, Objects, Methods
- PIP (Pypi)
- List slicing
- String formatting
- Dictionaries & Tuples
- Managing environments
- Dunder methods like __init__

(5 / 19)

Can you get a job in data science and machine learning without a college degree?

🧵👇

Short Answer: Yes.

Long Answer, keep reading 👇

(Advice from industry experts who I talked to.)

Companies are looking for people who add value.
In order to add value, you'll need skills. Simple as that.

Get the skills to provide value and you'll get the job.

In the age of the internet where everything is pretty much free, why do college degrees matter?

A college degree makes it easier to get the skills to get a job in machine learning or data science.

College degrees also help you make many useful connections and provides many opportunities in the form of internships and whatnot.

College degrees have their own place

Are you planning to learning machine learning this year?

These are the Python frameworks you need to learn as an absolute beginner.

🧵👇

It can be overwhelming seeing all the machine learning frameworks at once and not even knowing what they do.

I made this thread so that your machine learning journey becomes just a little bit easier.

(2 / 11)

This thread will only focus on Python frameworks.

Why?
Because Python is the most common path into machine learning.

( If you have experience with any other language then learning Python will not be difficult at all. )

(3 / 11)

Here's the entire list:

- Pandas
- Numpy
- Matplotlib
- TensorFlow
- PyTorch
- Keras
- Scikit Learn

Let's see what they do.

(4 / 11)

You'll have to interact with csv files or databases on a regular basis to access data in machine learning, pandas helps you do just that.

The cool thing about Pandas is that it actually doesn't change the original data...

(5 / 11

These are the tools you will need for machine learning in Python.

🧵👇

Anaconda

When you work in python, you'll be working with several frameworks and many of them work only on specific versions of python.

(2 / 13)

Now imagine downloading a new version of python and then installing it for every framework you want to work with 😬.

Meet Anaconda which allows you to run several versions of python. It comes pre-installed with several data science and machine learning frameworks.

(3 / 13)

Pip-env is also a way of maintaining several versions of Python and comes pre installed with Python.
You can use pip env or Anaconda, whichever works for you.

(4 / 13)

Jupyter Notebooks

Jupyter notebooks is an IDE just like VS code or Sublime. The special thing about jupyter is that you can parts of code in mini code editors called cells. This is great for prototyping and testing code.

(5 / 13)

1/🧵 Good #DataScience advice that breaks pretty much every rule you learned in class... a thread. (+full blog post linked)

English version: https://t.co/dG4l6vPFBT
Spanish version: https://t.co/gFAjPQ5clS

#AI #MachineLearning #Statistics #RStats

2/🧵 Allow your approach to be sloppy at first and burn some of your initial time, energy, and data on informing a good direction later. That's right, you're supposed to start sloppily ON PURPOSE.

3/🧵 Have a phase where the only result you’re after is *an idea of how to design your ultimate approach better.*

4/🧵 In other words, start with a pilot phase where the objective isn't finding answers, it's finding a good approach to finding answers.

5/🧵 That means you're encouraged (ENCOURAGED!) to start with everything your stats classes told you not to do:

11 key concepts of Machine Learning.

— Supervised Learning Edition —

🧵👇

😜

Before starting, remember that, if you follow me, one of your enemies will be immediately destroyed (and you'll get to read more of these threads, of course.)

And if you don't follow me, well, you just hurt my feelings.

😜

1. Labels

(Also referred to as "y")

The label is the piece of information that we are predicting.

For example:

- the animal that's shown in a picture
- the price of a house
- whether a message is spam or not

👇

2. Features

(Also referred to as "x")

These are the input variables to our problem. We use these features to predict the "label."

For example:

- pixels of a picture
- number of bedrooms of a house
- square footage of a house

👇

3. Samples

(This is also known as "examples.")

A sample is a particular instance of data (features or "x.") It could be "labeled" or "unlabeled."

👇

🤔 Python decorators

What are they? How do you use them?

🧵 Let's find out 👇

1️⃣ Decorator is a function or a class that wraps another function or class modifying its behavior.

So how does that work?

The first thing to know is that everything in Python is an object - functions too

2️⃣ That means they can be passed to another function as an argument or returned from a function

Functions that take other functions as an argument are called higher-order functions

3️⃣ In Python, you can define a function inside other function - such functions are called inner functions

4️⃣ To create a decorator you just need to apply all of that together

log_enter_leave is a decorator.

my_function is the function.

To alter my_function's behavior we reassign it applying log_enter_leave decorator.

Which libraries do you really need to get started with Machine Learning and why?

🧵👇

First and foremost, make sure that you have nailed the fundamental concepts of Python because machine learning requires a lot of programming!

(2 / 19)

If you know these topics, then you are good to go with machine learning in Python

- Object-oriented programming in Python:Classes,Objects,Methods
- Lists & List functions
- List comprehension
- List slicing
- String formatting
- List,Dictionaries & Tuples

(3 / 19)

Now, let's understand what popular python machine learning libraries do.

We will talk about👇
- TensorFlow (+ Keras)
- PyTorch
- Pandas
- Numpy
- Matplotlib
- SciKit Learn
- Seaborn

(4 / 19)

Now let's cover the libraries that you have to learn (at least the basics) for machine learning

1. Pandas

Pandas is a python library that allows you to store and read data from spreadsheets ( .csv, .xlsv files ) in structures called Dataframes.

(5 / 19)

Here's everything you need to know about the math for machine learning.

(+ free resources )
🧵👇

Before diving into the math, I suggest first having solid programming skills.

For example👇

(2 / 18)

In Python, these are the concepts which you must know:

- Object oriented programming in Python : Classes, Objects, Methods
- List slicing
- String formatting
- Dictionaries & Tuples
- Basic terminal commands
- Exception handling

(3 / 18)

If you want to learn those concepts for python, these courses are freecodecamp could be of help to you.

🔗Basics:youtube∙com/watch?v=rfscVS0vtbw
🔗Intermediate :youtube∙com/watch?v=HGOBQPFzWKo

(4 / 18)

👉You need to have really strong fundamentals in programming, because machine learning involves a lot of it.

It is 100% compulsory.

(5 / 18)

The "polarization is the problem" narrative to me is bad because it implies the status quo was a stable and acceptable set of trade-offs. It's the equivalent of a pollyanna "why can't we all get along" but the we in question is white supremacy and its targets.

like i understand that conflict mediation requires mutual concessions (i did my MA in conflict studies) and while it's important to meet people's deep needs across spectrums of difference, certain things are just not up for negotiation.

Also while peacebuilding work is good. It's not just "seeing each other as human" it's literally about transforming structural issues. (hence Galtung's distinction between positive and negative peace wherin negative peace is like a ceasefire and positive is a transformed society.

And I think that to whatever extent that kind of peacebuilding work is applicable, it's not between fascists and people who think fascists are bad. Fascists are not invited. They are only invited if they seize territories and can't be crushed. Then secret negotiations.

The US is always like "we don't negotiate with terrorists" but literally every president negotiates with who they consider to be terrorists. It's all just down low. But that's just because counter-insurgency failed.

Here's what you'll hear when you first get into machine learning.

- Neural networks
- Loss
- Weights
- Biases
- Epochs
- Neurons
- Optimizers
...

It can get very confusing really fast!

Here are some of the terms you should know about.
(I wish I had this before)

🧵👇

These terms won't mean anything unless you know what Machine learning is all about.

> Machine learning is the process of making a program which allows a computer to learn from data.

The data could be anything, images, audio or even text.

In machine learning we use something called a neural network, this is essentially an imitation of the human brain.

> Neural Networks are a digital imitation of the neurons you see in the human brain.

In these neural networks, data flows through them and each neuron (the circle) has a numerical value which will change.

> The value of a neuron gets changes to something which is close to what we want each time the data passes through the neural network.

Think of the neurons as dials on a lock, you have to tune every dial to open the lock.

It is almost impossible for a human to tune thousands of dials like these, but a computer certainly can.

Okay, doing my first baby steps with r2frida (which combines the power of @radareorg and @fridadotre).

Gonna share my progress in this thread (live, so keep calm).

The goal: Runtime inspection of data sent out by TikTok !!before!! it gets encrypted

1/many

First of all, we do not start from zero. I got some prior knowledge from past reversing attempts and want to share some important facts.

TikTok's (log data) encryption is accomplished by a native library. The Android Java code just serves as proxy function to the native function

The decompiled code for the respective native JNI function of an older TikTok version looks something like this, but in this example I use the most current TT version (no statical analysis done, yet)

In case you never reversed native libraries which were build to interface with Android Java layer via JNI, I highly suggest the entry level introduction on the topic by @maddiestone

Before we start, I want to pinpoint some important aspects (which are also covered by Maddie's videos).

1) Unlike raw C-functions, JNI functions like the one showcased above, receive pointers to complex Java objects .

F.e. a function receiving a String on the Java layer...