We need to represent the real world as accurately as possible. If some situations are underrepresented we are introducing Sampling Bias.
Sampling Bias is nasty because we'll have high test accuracy, but our model will perform badly when deployed.
👇
Machine Learning in the Real World 🧠 🤖
ML for real-world applications is much more than designing fancy networks and fine-tuning parameters.
In fact, you will spend most of your time curating a good dataset.
Let's go through the process together 👇
#RepostFriday
We need to represent the real world as accurately as possible. If some situations are underrepresented we are introducing Sampling Bias.
Sampling Bias is nasty because we'll have high test accuracy, but our model will perform badly when deployed.
👇
Let's build a model to recognize traffic lights for a self-driving car. We need to collect data for different:
▪️ Lighting conditions
▪️ Weather conditions
▪️ Distances and viewpoints
▪️ Strange variants
And if we sample only 🚦 we won't detect 🚥 🤷♂️
👇
Now we need to clean all corrupted and irrelevant samples. We need to remove:
▪️ Overexposed or underexposed images
▪️ Images in irrelevant situations
▪️ Faulty images
Leaving them in the dataset will hurt our model's performance!
👇
Most ML models like their data nicely normalized and properly scaled. Bad normalization can also lead to worse performance (I have a nice story for another time...)
▪️ Crop and resize all images
▪️ Normalize all values (usually 0 mean and 1 std. dev.)
👇
Manual labeling is expensive. Try to be clever and automate as much as possible:
▪️ Generate labels from the input data
▪️ Use slow, but accurate algorithms offline
▪️ Pre-label data during collection
▪️ Develop good labeling tools
▪️ Use synthetic data?
👇
You will always have errors in the labels - humans make mistakes. Review and iterate!
▪️ Spot checks to find systematic problems
▪️ Improve labeling guidelines and tools
▪️ Review test results and fix labels
▪️ Label samples multiple times
👇
A recent study by MIT found that 10 of the most popular public datasets had 3.4% label errors on average (ImageNet had 5.8%).
This even lead authors to choose the wrong (and more complex) model as their best one!
https://t.co/dfZPz6xnU0
👇
Dealing with imbalanced data can be tricky...
Let's classify the color of the 🚦 - we can get 97% just by learning to recognize 🟢 and 🔴, just because 🟡 is severely underrepresented.
I have a separate thread on this topic:
https://t.co/R8z3AeDD2b
👇
Dealing with imbalanced datasets \U0001f401 \u2696\ufe0f \U0001f418
— haltakov.eth \U0001f30d \U0001f1fa\U0001f1e6 (@haltakov) February 10, 2021
Real world datasets are often imbalanced - some of the classes appear much more often in your data than others.
The problem? You ML model will likely learn to only predict the dominant classes.
What can you do about it? \U0001f914
Thread \U0001f447
This is the part that is usually covered by ML courses. Now is the time to try out different features, network architectures, fine-tune parameters etc.
But we are not done yet... 👇
In most real-world applications the bottleneck is not the model itself, but the data. After having a first model, we need to review where it has problems and go back to:
▪️ Collecting and labeling more data
▪️ Correcting labels
▪️ Balancing the data
👇
Deploying the model in production poses some additional constraints:
▪️ Speed
▪️ Cost
▪️ Stability
▪️ Privacy
▪️ Hardware availability and integration
We have to find a good trade-off between these factors and accuracy.
Now we are done, right? No...👇
The performance of the model will start degrading over time because the world keeps changing:
▪️ Concept drift - the real-world distribution changes
▪️ Data drift - the properties of the data change
We need to detect this, retrain, and deploy again.
Example 👇
We now have a trained model to recognize 🚦, but people keep inventing new variants - see what some creative people in Munich came up with 😄
We need a way to detect that we have a problem, collect data, label, and retrain our model.
👇
This is how a typical ML pipeline for real-world applications looks like. Please remember this:
▪️ Curating a good dataset is the most important thing
▪️ Dataset curation is an iterative process
▪️ Monitoring is critical to ensure good performance over time
If you are interested in seeing more, follow me @haltakov
.
More from haltakov.eth 🌍 🇺🇦
Machine Learning Paper Reviews 🔎📜
Check out this thread for short reviews of some interesting Machine Learning and Computer Vision papers. I explain the basic ideas and main takeaways of each paper in a Twitter thread.
👇 I'm adding new reviews all the time! 👇
AlexNet - the paper that started the deep learning revolution in Computer Vision!
DenseNet - reducing the size and complexity of CNNs by adding dense connections between layers.
Playing for data - generating synthetic GT from a video game (GTA V) and using it to improving semantic segmentation models.
Transformers for image recognition - a new paper with the potential to replace convolutions with a transformer.
Check out this thread for short reviews of some interesting Machine Learning and Computer Vision papers. I explain the basic ideas and main takeaways of each paper in a Twitter thread.
👇 I'm adding new reviews all the time! 👇
AlexNet - the paper that started the deep learning revolution in Computer Vision!
It's finally time for some paper review! \U0001f4dc\U0001f50d\U0001f9d0
— Vladimir Haltakov (@haltakov) September 28, 2020
I promised the other day to start posting threads with summaries of papers that had a big impact on the field of ML and CV.
Here is the first one - the AlexNet paper! pic.twitter.com/QNLPIMZSIa
DenseNet - reducing the size and complexity of CNNs by adding dense connections between layers.
ML paper review time - DenseNet! \U0001f578\ufe0f
— Vladimir Haltakov (@haltakov) October 15, 2020
This paper won the Best Paper Award at the 2017 Conference on Computer Vision and Pattern Recognition (CVPR) - the best conference for computer vision problems.
It introduces a new CNN architecture where the layers are densely connected. pic.twitter.com/DuHytaoXia
Playing for data - generating synthetic GT from a video game (GTA V) and using it to improving semantic segmentation models.
Time for another ML paper review - generating synthetic ground truth data from video games! \U0001f3ae
— Vladimir Haltakov (@haltakov) October 5, 2020
I love this paper, because it pushes the boundaries of creating realistic synthetic ground truth data and shows that you can use it for training and improve your model.
Details \U0001f447 pic.twitter.com/fBgORYG8Lz
Transformers for image recognition - a new paper with the potential to replace convolutions with a transformer.
Another paper review, but a little different this time... \U0001f937\u200d\u2642\ufe0f
— Vladimir Haltakov (@haltakov) October 5, 2020
The paper is not published yet, but is submitted for review at ICLR 2021. It is getting a lot of attention from the CV/ML community, though, and many speculate that it is the end of CNNs... \U0001f447https://t.co/bh6wUxYfxu pic.twitter.com/dZGBYB8A5U
Let's talk about a common problem in ML - imbalanced data ⚖️
Imagine we want to detect all pixels belonging to a traffic light from a self-driving car's camera. We train a model with 99.88% performance. Pretty cool, right?
Actually, this model is useless ❌
Let me explain 👇
The problem is the data is severely imbalanced - the ratio between traffic light pixels and background pixels is 800:1.
If we don't take any measures, our model will learn to classify each pixel as background giving us 99.88% accuracy. But it's useless!
What can we do? 👇
Let me tell you about 3 ways of dealing with imbalanced data:
▪️ Choose the right evaluation metric
▪️ Undersampling your dataset
▪️ Oversampling your dataset
▪️ Adapting the loss
Let's dive in 👇
1️⃣ Evaluation metrics
Looking at the overall accuracy is a very bad idea when dealing with imbalanced data. There are other measures that are much better suited:
▪️ Precision
▪️ Recall
▪️ F1 score
I wrote a whole thread on
2️⃣ Undersampling
The idea is to throw away samples of the overrepresented classes.
One way to do this is to randomly throw away samples. However, ideally, we want to make sure we are only throwing away samples that look similar.
Here is a strategy to achieve that 👇
Imagine we want to detect all pixels belonging to a traffic light from a self-driving car's camera. We train a model with 99.88% performance. Pretty cool, right?
Actually, this model is useless ❌
Let me explain 👇
The problem is the data is severely imbalanced - the ratio between traffic light pixels and background pixels is 800:1.
If we don't take any measures, our model will learn to classify each pixel as background giving us 99.88% accuracy. But it's useless!
What can we do? 👇
Let me tell you about 3 ways of dealing with imbalanced data:
▪️ Choose the right evaluation metric
▪️ Undersampling your dataset
▪️ Oversampling your dataset
▪️ Adapting the loss
Let's dive in 👇
1️⃣ Evaluation metrics
Looking at the overall accuracy is a very bad idea when dealing with imbalanced data. There are other measures that are much better suited:
▪️ Precision
▪️ Recall
▪️ F1 score
I wrote a whole thread on
How to evaluate your ML model? \U0001f4cf
— Vladimir Haltakov (@haltakov) August 31, 2021
Your accuracy is 97%, so this is pretty good, right? Right? No! \u274c
Just looking at the model accuracy is not enough. Let me tell you about some other metrics:
\u25aa\ufe0f Recall
\u25aa\ufe0f Precision
\u25aa\ufe0f F1 score
\u25aa\ufe0f Confusion matrix
Let's start \U0001f447
2️⃣ Undersampling
The idea is to throw away samples of the overrepresented classes.
One way to do this is to randomly throw away samples. However, ideally, we want to make sure we are only throwing away samples that look similar.
Here is a strategy to achieve that 👇
More from All
You May Also Like
Department List of UCAS-China PROFESSORs for ANSO, CSC and UCAS (fully or partial) Scholarship Acceptance
1) UCAS School of physical sciences Professor
https://t.co/9X8OheIvRw
2) UCAS School of mathematical sciences Professor
3) UCAS School of nuclear sciences and technology
https://t.co/nQH8JnewcJ
4) UCAS School of astronomy and space sciences
https://t.co/7Ikc6CuKHZ
5) UCAS School of engineering
6) Geotechnical Engineering Teaching and Research Office
https://t.co/jBCJW7UKlQ
7) Multi-scale Mechanics Teaching and Research Section
https://t.co/eqfQnX1LEQ
😎 Microgravity Science Teaching and Research
9) High temperature gas dynamics teaching and research section
https://t.co/tVIdKgTPl3
10) Department of Biomechanics and Medical Engineering
https://t.co/ubW4xhZY2R
11) Ocean Engineering Teaching and Research
12) Department of Dynamics and Advanced Manufacturing
https://t.co/42BKXEugGv
13) Refrigeration and Cryogenic Engineering Teaching and Research Office
https://t.co/pZdUXFTvw3
14) Power Machinery and Engineering Teaching and Research
1) UCAS School of physical sciences Professor
https://t.co/9X8OheIvRw
2) UCAS School of mathematical sciences Professor
3) UCAS School of nuclear sciences and technology
https://t.co/nQH8JnewcJ
4) UCAS School of astronomy and space sciences
https://t.co/7Ikc6CuKHZ
5) UCAS School of engineering
6) Geotechnical Engineering Teaching and Research Office
https://t.co/jBCJW7UKlQ
7) Multi-scale Mechanics Teaching and Research Section
https://t.co/eqfQnX1LEQ
😎 Microgravity Science Teaching and Research
9) High temperature gas dynamics teaching and research section
https://t.co/tVIdKgTPl3
10) Department of Biomechanics and Medical Engineering
https://t.co/ubW4xhZY2R
11) Ocean Engineering Teaching and Research
12) Department of Dynamics and Advanced Manufacturing
https://t.co/42BKXEugGv
13) Refrigeration and Cryogenic Engineering Teaching and Research Office
https://t.co/pZdUXFTvw3
14) Power Machinery and Engineering Teaching and Research
