A topic that comes up in every interview.

Bias, variance, and their relationship with machine learning algorithms. One of the most basic concepts that you have to know by heart.

Here is a simple summary that you will easily remember.

Every machine learning algorithm deals with 3 types of errors:

1. Bias error
2. Variance error
3. Irreducible error

There's nothing we can do about #3.

Let's focus on the other two.

↓ 1/5
"Bias" refers to the assumptions the model makes to simplify the process of finding answers.

The more assumptions it makes, the more biased the model is.
"Variance" refers to how much the answers given by the model will change if we use different training data.

If the answers stay the same regardless of the data, the model has low variance.
Often, linear models are high-bias, and nonlinear models are low-bias.

Example low-bias algorithms:
• Decision Trees
• SVN
• kNN

Example high-bias algorithms:
• Linear Regression
• Logistic Regression
Often, linear models are low-variance, and nonlinear models are high-variance.

Example low-variance algorithms:
• Linear Regression
• Logistic Regression

Example high-variance algorithms:
• Decision Trees
• SVN
• kNN
Sometimes, you can change how these algorithms work to get a different tradeoff between their bias and variance.

Example:

• By increasing the value of "k" in kNN, we can increase the algorithm's bias.

• By pruning a Decision Tree, we can reduce its variance.
It doesn't matter what you do; the tradeoff is always there:

• Increasing bias decreases variance.
• Increasing variance decreases bias.

To work around this:

• Choose the appropriate algorithm
• Configure it correctly
• Work with the underlying dataset
If you want low-bias and low-variance machine learning content, follow me @svpino.

I come here to write about machine learning, and I promise you'll enjoy it.

More from Santiago

More from All

You May Also Like

A brief analysis and comparison of the CSS for Twitter's PWA vs Twitter's legacy desktop website. The difference is dramatic and I'll touch on some reasons why.

Legacy site *downloads* ~630 KB CSS per theme and writing direction.

6,769 rules
9,252 selectors
16.7k declarations
3,370 unique declarations
44 media queries
36 unique colors
50 unique background colors
46 unique font sizes
39 unique z-indices

https://t.co/qyl4Bt1i5x


PWA *incrementally generates* ~30 KB CSS that handles all themes and writing directions.

735 rules
740 selectors
757 declarations
730 unique declarations
0 media queries
11 unique colors
32 unique background colors
15 unique font sizes
7 unique z-indices

https://t.co/w7oNG5KUkJ


The legacy site's CSS is what happens when hundreds of people directly write CSS over many years. Specificity wars, redundancy, a house of cards that can't be fixed. The result is extremely inefficient and error-prone styling that punishes users and developers.

The PWA's CSS is generated on-demand by a JS framework that manages styles and outputs "atomic CSS". The framework can enforce strict constraints and perform optimisations, which is why the CSS is so much smaller and safer. Style conflicts and unbounded CSS growth are avoided.