Think accuracy should be your primary metric? You’re sorely mistaken.
Most of the best metrics instead focus on how far your were from the correct answer. Think RMSE and MAE.
Others point to how well calibrated your model is, like F1.
The worst taught skill in machine learning is model validation.
If you can’t validate your models well, you have no idea if they will actually work.
Here are 3 steps I’d take if I was relearning model validation from scratch 🧵
Think accuracy should be your primary metric? You’re sorely mistaken.
Most of the best metrics instead focus on how far your were from the correct answer. Think RMSE and MAE.
Others point to how well calibrated your model is, like F1.
Before diving in too deep, make sure you understand the basics.
You can’t become an expert in validation in the classroom, but knowing what is out there (simple k-fold, stratified, grouped, roll forward, etc.) is crucial.
Every day, or multiple times a week, pick an old Kaggle competition.
Read every solution that is posted and skip to their validation schemes.
There are nuances to every dataset, and this is the best way to see how pros navigate them.
Get a dataset and create a random test set.
Then, build some simple models and switch validation strategies in and out and see how well your models generalize for each scheme.
This will cement the importance of validation.
You will only improve at validation if you apply it to a ton of datasets.
If you stop after step 2, your skills will not be good enough. Full stop.
Never rest on your laurels. There is always something new to learn, and some new trick you can use.
I also discussed them on a podcast with @bhutanisanyam1 here: https://t.co/AiGAe1zBH3
Follow me @marktenenholtz so that you don’t miss it!
More from All
Margatha Natarajar murthi - Uthirakosamangai temple near Ramanathapuram,TN
#ArudraDarisanam
Unique Natarajar made of emerlad is abt 6 feet tall.
It is always covered with sandal paste.Only on Thriuvadhirai Star in month Margazhi-Nataraja can be worshipped without sandal paste.
After removing the sandal paste,day long rituals & various abhishekam will be https://t.co/e1Ye8DrNWb day Maragatha Nataraja sannandhi will be closed after anointing the murthi with fresh sandal paste.Maragatha Natarajar is covered with sandal paste throughout the year
as Emerald has scientific property of its molecules getting disturbed when exposed to light/water/sound.This is an ancient Shiva temple considered to be 3000 years old -believed to be where Bhagwan Shiva gave Veda gyaana to Parvati Devi.This temple has some stunning sculptures.
#ArudraDarisanam
Unique Natarajar made of emerlad is abt 6 feet tall.
It is always covered with sandal paste.Only on Thriuvadhirai Star in month Margazhi-Nataraja can be worshipped without sandal paste.
After removing the sandal paste,day long rituals & various abhishekam will be https://t.co/e1Ye8DrNWb day Maragatha Nataraja sannandhi will be closed after anointing the murthi with fresh sandal paste.Maragatha Natarajar is covered with sandal paste throughout the year
as Emerald has scientific property of its molecules getting disturbed when exposed to light/water/sound.This is an ancient Shiva temple considered to be 3000 years old -believed to be where Bhagwan Shiva gave Veda gyaana to Parvati Devi.This temple has some stunning sculptures.
You May Also Like
1/Politics thread time.
To me, the most important aspect of the 2018 midterms wasn't even about partisan control, but about democracy and voting rights. That's the real battle.
2/The good news: It's now an issue that everyone's talking about, and that everyone cares about.
3/More good news: Florida's proposition to give felons voting rights won. But it didn't just win - it won with substantial support from Republican voters.
That suggests there is still SOME grassroots support for democracy that transcends
4/Yet more good news: Michigan made it easier to vote. Again, by plebiscite, showing broad support for voting rights as an
5/OK, now the bad news.
We seem to have accepted electoral dysfunction in Florida as a permanent thing. The 2000 election has never really
To me, the most important aspect of the 2018 midterms wasn't even about partisan control, but about democracy and voting rights. That's the real battle.
2/The good news: It's now an issue that everyone's talking about, and that everyone cares about.
3/More good news: Florida's proposition to give felons voting rights won. But it didn't just win - it won with substantial support from Republican voters.
That suggests there is still SOME grassroots support for democracy that transcends
4/Yet more good news: Michigan made it easier to vote. Again, by plebiscite, showing broad support for voting rights as an
5/OK, now the bad news.
We seem to have accepted electoral dysfunction in Florida as a permanent thing. The 2000 election has never really
Bad ballot design led to a lot of undervotes for Bill Nelson in Broward Co., possibly even enough to cost him his Senate seat. They do appear to be real undervotes, though, instead of tabulation errors. He doesn't really seem to have a path to victory. https://t.co/utUhY2KTaR
— Nate Silver (@NateSilver538) November 16, 2018
