How to scrape Twitter

David Barton
David Barton

So why use web scraping on Twitter? What kind of information can we get from the notoriously fast-moving website and how easy is it to get that information?

Twitter has become one of the most popular online discussion platforms in the world

Twitter started off as a simple ‘microblogging’ system for users to share short posts called tweets. That straightforward idea of expressing your thoughts in just 140 characters (and now 280 characters) has made Twitter one of the most active discussion platforms on the internet. People engage and argue, both companies and individuals market their brands, and politicians even use it as a way to reach their voters.

Twitter has more than 340 million users and more than 500 million tweets are posted every day. As Twitter itself boasts: Twitter is what’s happening and what people are talking about right now.

Chart from https://www.statista.com/

As you might imagine, that means that there’s a lot of useful data just sitting around on Twitter, waiting to be used for other purposes.

A single tweet can tell you information about:

  • the demographics of people who liked or retweeted the tweet
  • total clicks on a profile
  • how many people saw the tweet

And that’s just some of the vast amount of data ready to be extracted from Twitter.

For a Twitter user or marketer, access to data about how others engage with their tweets can be vital for developing a brand. For companies, gathering data across Twitter can provide them gain a competitive advantage. Academic researchers and journalists can make use of the data to understand how people interact and identify trends before they rise to the surface. Once you have the data, what you do with it is up to you.

What about the Twitter API?

The Twitter API is really great for developers. It gives you a lot of access to the platform underlying Twitter. You can use it to compose tweets, read profiles, access data about your followers, and get information on four main Twitter data points: Tweets, Entities, Places, and Users.

But we believe that web scraping can allow you to do more with Twitter than the API allows. Apify’s Twitter Scraper creates an unofficial Twitter API and has the following advantages over the official API:

  • you do not need to have an account
  • our scraper is not rate-limited
  • you don’t need a registered app and API key

Since scraping basically automates tasks that could manually be done by a human, it is legal. But it’s still important to watch out that the data you accumulate while scraping is not protected by copyright or personal data regulations. Read through our legality blog post to learn more about the laws that concern web scraping.

Just want to scrape Twitter data the easy way?

Before we get to the tutorial, maybe you'd like to start off by scraping some very specific Twitter data? Apify Store also offers a few specialized Twitter scrapers to carry out smaller scraping tasks. You only need to insert a keyword or a URL and start your run to extract your results, including Twitter usernames, profile photos, follower count, tweets, images, and more.

The full range of Twitter scrapers include:

Remember that these smaller scrapers offer a narrower range of input settings and limited results. If you're planning large-scale scraping and need more detailed results, we recommend you use Twitter Scraper, so read on for our full guide to scraping Twitter.

How to scrape data from Twitter — Apify’s step-by-step guide

The Apify platform has lots of ready-made scraping tools. The one we’re going to use is the Twitter Scraper.

1. First, you need to go to the actor's page, and click the Try for free button. You will be redirected to Apify Console, which is your workspace to run tasks for your scrapers. If you already have an Apify account and are logged in, go to Step 3.

Twitter Scraper's page on Apify Store

2. You can log in or sign up with your email address, or with a Google or GitHub account.

Log in or sign up with Google, GitHub, or your email

3. So let’s go to Twitter and find something to scrape. How about the Apify profile? 😁 Just insert the URL or Twitter handle into the actor's input fields: @apify or https://twitter.com/apify.

In the scraper's input fields, insert the Twitter handle...
...or the Twitter page's URL

But you can also change the following parameters:

  • Fill in the username you want to scrape.
  • Limit the number of max tweets to make everything go faster.
  • Select the types of tweets you’re interested in
  • Enter your credentials if you want to scrape a lot of information.

4. Once you are all set, click the Start button. Notice that your task will change its status to Running, so wait for the scraper's run to finish. It will be just a minute before you see the status switch to Succeeded.

Your run has succeeded

5. As soon as you see that the run has Succeeded, you can check the results in the Dataset tab. In fact, you can even check the Dataset tab before the scraper has finished its run if you’re curious to see how it’s doing. The Dataset tab contains your scraped data in many formats, including HTML table, JSON, CSV, Excel, XML, and RSS feed. 😁

Find your results in the Dataset tab

6. Preview the data by clicking the Preview button or viewing it in a new tab if the dataset is too large. You can choose to download it onto your computer for further use as spreadsheets or in other apps and your projects.

Preview your results before downloading them

And that’s it — you can learn lots more about how to scrape Twitter by studying the readme documentation over on the Twitter Scraper.

So go ahead and sign up for your free Apify account to start scraping Twitter with our Twitter Scraper!

Did you know?

  • 82% of B2B content marketers used Twitter for organic content marketing in the last 12 months.
  • 27% of B2B content marketers used paid Twitter ads in the last 12 months, indicating that Twitter might be an untapped market space.

Stats from https://blog.hootsuite.com/twitter-statistics/



Great! Next, complete checkout for full access to Apify
Welcome back! You've successfully signed in
You've successfully subscribed to Apify
Success! Your account is fully activated, you now have access to all content
Success! Your billing info has been updated
Your billing was not updated