Do you need social media data for your machine learning project?

- Twitter data?
- Reddit data?
- Facebook data?

Where to get it?

Reddit: Pushshift

Pushshift is a big-data storage and analytics project.

Most people know it for its copy of reddit comments and submissions.

https://t.co/ne0XqKIt9A
Reddit: Pushshift API

The https://t.co/HWZNWEvrxY Reddit API was designed and created by the /r/datasets mod team to help provide enhanced functionality and search capabilities for searching Reddit comments and submissions.

https://t.co/FkUE7R2jlb
Reddit: Pushshift file download

Note: The latest data for manual download is from April 2020

https://t.co/jBM4U71dnm
Reddit: PMAW: Pushshift Multithread API Wrapper

PMAW is an ultra minimalist wrapper for the Pushshift API which uses multithreading to retrieve Reddit comments and submissions.

If you pull data via Pushshift use PMAW, highly recommended!

https://t.co/xSlaX3Di6T
Reddit: Redditsearch

Frontend which uses Pushshift for detail searches on subreddits or domain

https://t.co/8C37LM7aTx
Twitter: Stream as download

The Internet Archive is a digital library of Internet sites and other cultural artifacts in digital form.

Note: The last archived data is from January 2021

https://t.co/hywWgFtbjq
Twitter: Tweepy Twitter for Python!

An easy-to-use Python library for accessing the Twitter API.

Note: The downside is the API limitations of Twitter, so you need a lot of time.

https://t.co/dhc7x1lZ8U
Twitter: Script

Most twitter scraper are banned by Twitter or no longer work so here is a simple and unlimited twitter scraper with python and without authentication

Note: Headless mode no longer work and it uses Selenium to access Twitter

https://t.co/feZsbOFmJR
Facebook: Scrape Facebook public pages without an API key.

$ pip install facebook-scraper

https://t.co/Rh4s93P1YD
Facebook: Large Page-Page Network data

Nodes represent official Facebook pages while the links are mutual likes between sites.

Node features are extracted from the site descriptions that the page owners created to summarize the purpose of the site.

https://t.co/tLSvGA95Az
Octoparse: Easy Web Scraping for Anyone

Everything you need to automate your web scraping.

Note: It's a paid service.

https://t.co/f0bBLpkxSZ
Spread the open source love!

If you know an amazing project drop me message @philipvollet
we need this edit function. my inner zen isn't balanced every time i spot a typo

More from All

You May Also Like

**Thread on Bravery of Sikhs**
(I am forced to do this due to continuous hounding of Sikh Extremists since yesterday)

Rani Jindan Kaur, wife of Maharaja Ranjit Singh had illegitimate relations with Lal Singh (PM of Ranjit Singh). Along with Lal Singh, she attacked Jammu, burnt - https://t.co/EfjAq59AyI


Hindu villages of Jasrota, caused rebellion in Jammu, attacked Kishtwar.

Ancestors of Raja Ranjit Singh, The Sansi Tribe used to give daughters as concubines to Jahangir.


The Ludhiana Political Agency (Later NW Fronties Prov) was formed by less than 4000 British soldiers who advanced from Delhi and reached Ludhiana, receiving submissions of all sikh chiefs along the way. The submission of the troops of Raja of Lahore (Ranjit Singh) at Ambala.

Dabistan a contemporary book on Sikh History tells us that Guru Hargobind broke Naina devi Idol Same source describes Guru Hargobind serving a eunuch
YarKhan. (ref was proudly shared by a sikh on twitter)
Gobind Singh followed Bahadur Shah to Deccan to fight for him.


In Zafarnama, Guru Gobind Singh states that the reason he was in conflict with the Hill Rajas was that while they were worshiping idols, while he was an idol-breaker.

And idiot Hindus place him along Maharana, Prithviraj and Shivaji as saviours of Dharma.
THREAD: 12 Things Everyone Should Know About IQ

1. IQ is one of the most heritable psychological traits – that is, individual differences in IQ are strongly associated with individual differences in genes (at least in fairly typical modern environments). https://t.co/3XxzW9bxLE


2. The heritability of IQ *increases* from childhood to adulthood. Meanwhile, the effect of the shared environment largely fades away. In other words, when it comes to IQ, nature becomes more important as we get older, nurture less.
https://t.co/UqtS1lpw3n


3. IQ scores have been increasing for the last century or so, a phenomenon known as the Flynn effect. https://t.co/sCZvCst3hw (N ≈ 4 million)

(Note that the Flynn effect shows that IQ isn't 100% genetic; it doesn't show that it's 100% environmental.)


4. IQ predicts many important real world outcomes.

For example, though far from perfect, IQ is the single-best predictor of job performance we have – much better than Emotional Intelligence, the Big Five, Grit, etc. https://t.co/rKUgKDAAVx https://t.co/DWbVI8QSU3


5. Higher IQ is associated with a lower risk of death from most causes, including cardiovascular disease, respiratory disease, most forms of cancer, homicide, suicide, and accident. https://t.co/PJjGNyeQRA (N = 728,160)