BC DATA SCIENCE
Saved by @zmbnski

Tired of word clouds? Want to do better sentiment analysis? Not sure how to look at the words underneath your measures?

Our long overdue paper on generalized word shift graphs is finally here!
https://t.co/lIBXvbMJWX
https://t.co/vSL1REYT8V

So what are they?

1/n

 If we have two texts, there are many ways we can compare them. Weighted averages are a particularly useful measure because they're flexible and interpretable

Proportions, Shannon entropy, the KLD, the JSD, and dictionary methods can all be written as weighted averages

2/n
But weighted avgs are also slippery. When we try to compress complex phenomena like happiness, surprise, divergence, or diversity into a single number, it can be unclear what we're measuring

If the measure goes up, what does that mean? Why did it do that? Can we trust it?

3/n
Very often, that's the end of the line and we're left with an uneasy feeling in the pit of our stomach that our weighted avg is actually picking up a data artifact or some other unintended peculiarity

Word shift graphs help us address those concerns

4/n
First, word shifts look under the hood of weighted averages to see what's going on

All weighted averages are a sum of contributions from individual words. We can pull out those words, and rank which ones contribute the most to the difference between two texts

5/n
But we can go further

Consider dictionary sentiment analysis. We don't just know the scores of words. We also have an understanding of which words are *more* or *less* positive

We know that there's a point on the sentiment scale that distinguishes positive from negative

6/n
The first thing word shifts do is make this *reference value* explicit

For sentiment analysis, this gives us 4 qualitatively different ways a word can contribute:
1. + word is used more
2. - word is used less
3. + word is used less
4. - word is used more

7/n
We can encode the 4 types of word contribution through 4 types of bars

This is how we construct basic word shift graphs. They give us details of both *what* words contribute and *how* they do so

And we've made it easy for you to make your own:
https://t.co/vSL1REYT8V

8/n
But we can go even further! We generalize the word shift framework so that you can use it for more than just single dictionary sentiment analysis

You can word shifts with multiple dictionaries, entropy-based measures, and any metric that can be written as a weighted avg

9/n
Generalized word shifts account for how words can change scores across texts, allowing us to use context-dependent sentiment dictionaries, or measures like entropy

This ends up giving us 8 qualitatively different ways a word can contribute to the diff between two texts

10/n
There's a lot to unpack here! So to give you practice reading word shift graphs and show you how to use them in practice, we present 5 case studies covering presidential speeches, Moby Dick, U.S. urban parks, 280 character tweets, and labor diversity in the Great Recession

11/n
Word shift graphs are an invaluable tool for unpacking weighted averages and looking at how words affect our measures

We hope this paper can be the ultimate field guide for those are interested in using word shift graphs to help validate their own text-as-data analyses

12/n
All of the word shifts that we mention in the paper are implemented in the Shifterator package
https://t.co/vSL1REYT8V

We have new documentation that includes a comprehensive cookbook for using word shift graphs in Python
https://t.co/p9FdEJYlKN

Please reach out w/ Qs

13/n
I originally presented a draft of this paper at the Text as Data conference in Seattle almost 2 years ago

The Shifterator code has come a long way since then, and I've put a lot of time into it. I hope that people find it useful for their own work
https://t.co/vSL1REYT8V

14/n
Finally, this paper wouldn't have been possible without the hard work of many @compstorylab members past and present

I had a great time collaborating with @mrfrank5790 @lewis_math @andyreagan @ChrisDanforth @peterdodds and Aaron Schwartz on this Story Lab piece!

15/n
In summary: always look at the words!

And please reach out if you have any questions or comments about using word shift graphs!

16/16

More from Data science

Research Engineering at the Turing
Research Engineering a...
@turinghut23
✨✨ BIG NEWS: We are hiring!! ✨✨
Amazing Research Software Engineer / Research Data Scientist positions within the @turinghut23 group at the @turinginst, at Standard (permanent) and Junior levels 🤩

👇 Here below a thread on who we are and what we

We are a highly diverse and interdisciplinary group of around 30 research software engineers and data scientists 😎💻 👉 https://t.co/KcSVMb89yx #RSEng

We value expertise across many domains - members of our group have backgrounds in psychology, mathematics, digital humanities, biology, astrophysics and many other areas 🧬📖🧪📈🗺️⚕️🪐
https://t.co/zjoQDGxKHq
/ @DavidBeavan @LivingwMachines

In our everyday job we turn cutting edge research into professionally usable software tools. Check out @evelgab's #LambdaDays 👩‍💻 presentation for some examples:

We create software packages to analyse data in a readable, reliable and reproducible fashion and contribute to the #opensource community, as @drsarahlgibson highlights in her contributions to @mybinderteam and @turingway: https://t.co/pRqXtFpYXq #ResearchSoftwareHour
DATA SCIENCE
Data Professor
Data Professor
@thedataprof
Cheat sheet that summarizes #DataScience in 10 pages
(Links in the comments below 👇)


2/ Link to the cheatsheet by Maverick
CODING RESOURCE , DATA SCIENCE
Advertisement
Goku Mohandas
Goku Mohandas
@GokuMohandas
🔥 Putting ML in Production! We're going to publicly develop @madewithml's first ML service. Here is the broad curriculum:

- 📦 Product
- 🔢 Data
- 🤖 Modeling
- 📝 Scripting
- 🛠 API
- 🚀 Production

More details (lessons, task, etc.) here: https://t.co/xmMm9XGK9j

Thread 👇


Questions that this thread will answer:

- What is it?
- Who is this course for?
- What is the format?
- What makes this course unique?
- Why constrain to open source tools?
- What are my qualifications?
- Why is this free?
- What are the

What is it?

Putting ML in Production: a guide and code-driven case study on MLOps. We will be developing and deploying Made With ML's first ML service, from Product → ML → Production, with open source tools.

This ML service will act as a foundation for all future ML features and subsequent iterations. The first feature is tagifai - multilabel classification of tags for a project. We'll discuss the need and utility of this feature in the first lesson.

Who is this course for?

- ML developers looking to become end-to-end ML developers.
- Software engineers looking to learn how to responsibly deploy and monitor ML systems.
- Product managers who want to have a comprehensive understanding of the different stages of ML dev.
DATA SCIENCE
elvis
elvis
@omarsar0
I have always emphasized on the importance of mathematics in machine learning.

Here is a compilation of resources (books, videos & papers) to get you going.

(Note: It's not an exhaustive list but I have carefully curated it based on my experience and observations)

📘 Mathematics for Machine Learning

by Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong

https://t.co/zSpp67kJSg

Note: this is probably the place you want to start. Start slowly and work on some examples. Pay close attention to the notation and get comfortable with it.


📘 Pattern Recognition and Machine Learning

by Christopher Bishop

Note: Prior to the book above, this is the book that I used to recommend to get familiar with math-related concepts used in machine learning. A very solid book in my view and it's heavily referenced in academia.


📘 The Elements of Statistical Learning

by Jerome H. Friedman, Robert Tibshirani, and Trevor Hastie

Mote: machine learning deals with data and in turn uncertainty which is what statistics teach. Get comfortable with topics like estimators, statistical significance,...


📘 Probability Theory: The Logic of Science

by E. T. Jaynes

Note: In machine learning, we are interested in building probabilistic models and thus you will come across concepts from probability theory like conditional probability and different probability distributions.
DATA SCIENCE
Maria Khalusova
Maria Khalusova
@mariaKhalusova
To my JVM friends looking to explore Machine Learning techniques - you don’t necessarily have to learn Python to do that. There are libraries you can use from the comfort of your JVM environment. 🧵👇

https://t.co/EwwOzgfDca : Deep Learning framework in Java that supports the whole cycle: from data loading and preprocessing to building and tuning a variety deep learning networks.

https://t.co/J4qMzPAZ6u Framework for defining machine learning models, including feature generation and transformations, as directed acyclic graphs (DAGs).

https://t.co/9IgKkSxPCq a machine learning library in Java that provides multi-class classification, regression, clustering, anomaly detection and multi-label classification.

https://t.co/EAqn2YngIE : TensorFlow Java API (experimental)
DATA SCIENCE

You May Also Like

Karol296
Karol296
@Karol296
Wracając do sedna dzisiejszego zamieszania z artykułem autosportu / motorsportu. Kubica mówi w nim, że podejmie swoją decyzję w ciągu kilku dni. Rozważa opcję z Williamsem lub posadę kierowcy symulatora w Ferrari. Więc za kilka dni dowiemy się co go czeka za rok. #F1PL

Autosport / motorsport zmienił nagłówek swoje tekstu, bo pierwotnie sugerował on że Kubica jest numerem jeden na liście Williamsa na tytuł mówiący tylko tym, że nadal jest na liście kandydatów. Zrobili to po interwencji zespołu. Same wypowiedzi Kubicy z wywiadu się nie zmieniły.

Trudno powiedzieć czy autor się pośpieszył i wyciągnął inne wnioski niż powinien, a może wie coś więcej, ale nie mógł tego otwarcie napisać. Słowa Kubicy brzmią jednak jednoznacznie: decyzję podejmie w ciągu kilku dni.
SPORT
Carol Forden
Carol Forden
@CarolForden
1. The major problem most startups face is marketing and the ability to carve out a niche, find and retain paying customers.

Even with a stellar developer team and a stellar product, your startup may not grow or survive without a great marketing and sales teams.

2. Start growing your audience before you’ve a product or an idea.

You should start promoting yourself and building your audience before you have a finished product or even before you have a great idea that you want to build.

3. Don’t wait to start building your audience after you’ve launched a product.

Most first-time developers actually ignore marketing.

They’re oblivious to the challenge of attracting people to take a look at something they’ve created.

4. At least until they see their first project crash and burn within hours of the launch.

There are so many times that I have seen developers spend countless hours building something, releasing it with no fanfare...

5. and only then trying to figure out how to promote it by asking marketing questions on Indie Hackers, Quora or Hacker News. Don’t make that mistake.
STARTUPS
Advertisement
Thinking Patriot
Thinking Patriot
@ThinkinPatriot
1⃣
At the moment #POTUS is sitting on a stack of #Trump cards, he's just waiting to unleash a royal flush!
He has court cases that will go to #SCOTUS and thanks to the Texas case, he's now aware how to file them properly under article 3 not 2, so SCOTUS will be forced to listen.

2⃣
#Trump now has the #DNI report. #Barr stepped down and can now be a witness, he did his job. #Durham is special counsel and can prosecute, in any state.
He’s letting civil, criminal and federal courts fail to handle the situation properly so he can use military tribunals.

3⃣
#Trump has ALL the data from the #NSA, the #Kraken supercomputer, the #Alice #supercomputer, and likely many more computers, unknown to us.
He has the dueling electors from 7 state legislatures. He has VP #Pence, as the final arbiter of which ballots to accept.

4⃣
#Trump has the #InsurrectionAct, the #NDAA, the national emergency, the 14th amendment, the 2018 executive order, the 2017 very first EO, the #Patriot Act, the #FISA warrants, the Declassification of everything, people swearing affidavits by the 1000s.

5⃣
#Trump has all the statistical data being analyzed along with the videos, emails, phone calls and bank transfer statements showing the coordination of the #coup d'etat.
He has #RICO and he has the #CrimesAgainstHumanity videos. #Wikileaks just dropped extensive information
TRUMP , FOR LATER READ
Steve Stewart-Williams
Steve Stewart-Williams
@SteveStuWill
THREAD: 12 Things Everyone Should Know About IQ

1. IQ is one of the most heritable psychological traits – that is, individual differences in IQ are strongly associated with individual differences in genes (at least in fairly typical modern environments). https://t.co/3XxzW9bxLE


2. The heritability of IQ *increases* from childhood to adulthood. Meanwhile, the effect of the shared environment largely fades away. In other words, when it comes to IQ, nature becomes more important as we get older, nurture less. https://t.co/UqtS1lpw3n


3. IQ scores have been increasing for the last century or so, a phenomenon known as the Flynn effect. https://t.co/sCZvCst3hw (N ≈ 4 million)

(Note that the Flynn effect shows that IQ isn't 100% genetic; it doesn't show that it's 100% environmental.)


4. IQ predicts many important real world outcomes.

For example, though far from perfect, IQ is the single-best predictor of job performance we have – much better than Emotional Intelligence, the Big Five, Grit, etc. https://t.co/rKUgKDAAVx https://t.co/DWbVI8QSU3


5. Higher IQ is associated with a lower risk of death from most causes, including cardiovascular disease, respiratory disease, most forms of cancer, homicide, suicide, and accident. https://t.co/PJjGNyeQRA (N = 728,160)
LIFE
Greg Olear
Greg Olear
@gregolear
Trump is a Russian asset, a mob money launderer installed by his mob whoremasters to sow chaos and weaken the US & our allies.

EVERY SINGLE THING HE’S DONE since taking office is to achieve those purposes—that and to enrich himself personally.

[THREAD]


1/ His entire family is part of this. Melania is a fucking crook. Ivanka is a fucking crook. Junior and Gums are fucking crooks. They will all die in prison, except for Kushner, who, if he really *did* trade US intel for money, should be executed for espionage.

2/ Why do you think Trump installs uniquely corrupt & awful people to Cabinet positions? Pruitt, Zinke, Betsy, Rex/Pompeo, Wilbur "Melting Nazi Face" Ross. They are not there to serve us. They are not there to drain the swamp. They are there to BURN THE REPUBLIC DOWN.

3/ Trump steals. He takes taxpayer money—your money and mine—and lines his pockets with it. He exploits his office to make himself richer. This is all he cares about. Why are Republican citizens, you “fiscal conservatives,” not more angry about this? Riddle me THAT, #MAGA.

4/ Matt Whitaker, the human dildo and “acting AG,” was illegally installed to throw a monkey wrench in the Mueller investigation. That’s why they won’t release his financials. He’s fucking bought and sold. He’s going to prison for obstruction.
POLITICS