Tired of word clouds? Want to do better sentiment analysis? Not sure how to look at the words underneath your measures?

Our long overdue paper on generalized word shift graphs is finally here!

So what are they?


If we have two texts, there are many ways we can compare them. Weighted averages are a particularly useful measure because they're flexible and interpretable

Proportions, Shannon entropy, the KLD, the JSD, and dictionary methods can all be written as weighted averages

But weighted avgs are also slippery. When we try to compress complex phenomena like happiness, surprise, divergence, or diversity into a single number, it can be unclear what we're measuring

If the measure goes up, what does that mean? Why did it do that? Can we trust it?

Very often, that's the end of the line and we're left with an uneasy feeling in the pit of our stomach that our weighted avg is actually picking up a data artifact or some other unintended peculiarity

Word shift graphs help us address those concerns

First, word shifts look under the hood of weighted averages to see what's going on

All weighted averages are a sum of contributions from individual words. We can pull out those words, and rank which ones contribute the most to the difference between two texts

But we can go further

Consider dictionary sentiment analysis. We don't just know the scores of words. We also have an understanding of which words are *more* or *less* positive

We know that there's a point on the sentiment scale that distinguishes positive from negative

The first thing word shifts do is make this *reference value* explicit

For sentiment analysis, this gives us 4 qualitatively different ways a word can contribute:
1. + word is used more
2. - word is used less
3. + word is used less
4. - word is used more

We can encode the 4 types of word contribution through 4 types of bars

This is how we construct basic word shift graphs. They give us details of both *what* words contribute and *how* they do so

And we've made it easy for you to make your own:

But we can go even further! We generalize the word shift framework so that you can use it for more than just single dictionary sentiment analysis

You can word shifts with multiple dictionaries, entropy-based measures, and any metric that can be written as a weighted avg

Generalized word shifts account for how words can change scores across texts, allowing us to use context-dependent sentiment dictionaries, or measures like entropy

This ends up giving us 8 qualitatively different ways a word can contribute to the diff between two texts

There's a lot to unpack here! So to give you practice reading word shift graphs and show you how to use them in practice, we present 5 case studies covering presidential speeches, Moby Dick, U.S. urban parks, 280 character tweets, and labor diversity in the Great Recession

Word shift graphs are an invaluable tool for unpacking weighted averages and looking at how words affect our measures

We hope this paper can be the ultimate field guide for those are interested in using word shift graphs to help validate their own text-as-data analyses

All of the word shifts that we mention in the paper are implemented in the Shifterator package

We have new documentation that includes a comprehensive cookbook for using word shift graphs in Python

Please reach out w/ Qs

I originally presented a draft of this paper at the Text as Data conference in Seattle almost 2 years ago

The Shifterator code has come a long way since then, and I've put a lot of time into it. I hope that people find it useful for their own work

Finally, this paper wouldn't have been possible without the hard work of many @compstorylab members past and present

I had a great time collaborating with @mrfrank5790 @lewis_math @andyreagan @ChrisDanforth @peterdodds and Aaron Schwartz on this Story Lab piece!

In summary: always look at the words!

And please reach out if you have any questions or comments about using word shift graphs!


More from Data science

Wellll... A few weeks back I started working on a tutorial for our lab's Code Club on how to make shitty graphs. It was too dispiriting and I balked. A twitter workshop with figures and code:

Here's the code to generate the data frame. You can get the "raw" data from https://t.co/jcTE5t0uBT

Obligatory stacked bar chart that hides any sense of variation in the data

Obligatory stacked bar chart that shows all the things and yet shows absolutely nothing at the same time

STACKED Donut plot. Who doesn't want a donut? Who wouldn't want a stack of them!?! This took forever to render and looked worse than it should because coord_polar doesn't do scales="free_x".

You May Also Like