Simple Twitter API Operations with Python

Streamline Twitter data collection and sentiment analysis with Python using Tweepy for real-time insights.

Ceyhun Enki Aksan
Ceyhun Enki Aksan Entrepreneur, Maker

I’ve been working on the Twitter API for some time. At this stage, I’ve set several thresholds, incorporating machine learning, in a way that allows me to apply what I’ve learned through practice. My first threshold is to capture trending topics, followed by processing these topic-related content using a simple sentiment analysis (emotion analysis) process, and then passing it through various operations before publishing.

I reviewed the available solutions on Twitter’s Tools and Libraries page1. There, a wide range of options are available, developed in various programming languages such as JavaScript, Go, Python, Lua, Julia, R, and Ruby, offering diversity depending on the context. In the following section, I will proceed with the tweepy library, specifically for the Python programming language2.

Tweepy

Tweepy is one of the popular Python libraries developed as a solution for the Twitter API2. It can be easily installed using the pip install tweepy command, enabling quick access to functionalities such as listing tweets from specific topics or user-generated content (displaying the most relevant 20 tweets for the timeline, up to 100 per request for a user ID, (re)tweeting), accessing user and user activity data, and even automating tweet posting3 4 5 6 7 8.

The maximum number of requests allowed for both Twitter API v2 and Standard v1.1 is time-bound, depending on a specific time period or duration. The general request limit is 15 minutes. For a user-based endpoint, 900 requests per 15 minutes are permitted, while for application-based endpoints, the limit is set at 300 requests9.

Tweepy offers the following features in alignment with the Twitter API:

  • OAuth
  • API class
  • Models
  • Cursors
  • Streams

In the following example flow, since Tweepy v3 does not support Twitter API v2, I will proceed using Standard v1.1. Therefore, I will use the tweepy.API() class and access the content of the home timeline10 11.

import tweepy
import pandas as pd

consumer_key = '...'
consumer_secret = '...'
access_token = '...'
access_token_secret = '...'

# OAuth
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

# API class
api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)

# API class > Methods > User Timelines
public_tweets = api.home_timeline()
df = pd.DataFrame([tweet.text for tweet in public_tweets])

The following methods are available within the API class12:

  • User timelines
  • Tweets
  • Users
  • Followers
  • User Account
  • Likes
  • Blocking users
  • Searches
  • Trends
  • Streaming

Thus, let’s proceed with an example workflow and retrieve tweets containing the hashtag we specified within a specific date range.

First, let’s set up our virtual environment (venv) and separate the workflow from other Python environments13. I named the project twBot; you can modify it as you prefer. However, ensure you maintain this change in the subsequent steps.

python3 -m venv twBot
cd twBot && source bin/activate && which pip

After executing these commands, the pip path within the project directory should be activated. For example: /Users/user/Desktop/flask/bin/pip

You can view the relevant packages and their versions below.

certifi==2021.5.30
charset-normalizer==2.0.3
DateTime==4.3
idna==3.2
oauthlib==3.1.1
PySocks==1.7.1
pytz==2021.1
requests==2.26.0
requests-oauthlib==1.3.0
six==1.16.0
tweepy==3.10.0
urllib3==1.26.6
zope.interface==5.4.0

You can save this list to a file named requirements.txt in the project directory and then install it using pip from within the directory.

bin/pip install -r requirements.txt

After installing the newly added packages, you can update the requirements.txt file content using the command pip freeze > requirements.txt.

Yes, the Python code snippet we’ll be using is as follows14:

Do not forget to update the credentials within this code snippet.

tw = getTweets(
    consumer_key='...',
    consumer_secret='...',
    access_token='...',
    access_token_secret='...'
)

If you save the relevant code snippet as a file (you can also use it in Colab), you can run it regularly using bin/python and have the tweets saved as files.

bin/python ./getTweets.py

The query parameter within the following code snippet specifies which topics’ tweets we will collect. The -filter:retweets option excludes tweets that are retweets. Additional filters such as has:images -is:retweet and from:ceaksan from:twitterapi has:links can also be added to filter by tweet content and/or username15. The query is based on the item level. However, with a small modification, you can also access tweets based on the page level16.

tw.writeToFile(filename='twitter.csv',
               query='#Tokyo2020 -filter:retweets',
               count=100,
               lang='tr',
               items=100)

As a continuation of the above code snippet, let’s perform Model operations. First, let’s list the mentions, and then favorite the mention and follow the relevant user.

tweets = tw.api.mentions_timeline()
for tweet in tweets:
  print(tweet.text)
  tweet.favorite()
  tweet.user.follow()

Finally, besides listing a certain number of tweets, let’s start tracking tweets that match the stream in real time17 18.

class TWStreamListener(tweepy.StreamListener):
    def on_status(self, status):
        print(status.text)

TWListener = TWStreamListener()
twStream = tweepy.Stream(auth = tw.api.auth, listener=TWListener)
twStream.filter(track=['Tokyo2020', 'Turkey'])

PyPA

Footnotes

  1. Twitter API Tools and libraries. Twitter Developer Platform
  2. Tweepy Documentation. Tweepy 2
  3. Tutorials. Twitter Developer Platform
  4. API Reference. Tweepy
  5. Data dictionary
  6. Miguel Garcia. How to Make a Twitter Bot in Python With Tweepy
  7. Getting historical Tweets using the full-archive search endpoint. Twitter Developer Platform
  8. Explore a user’s Tweets. Twitter Developer Platform
  9. Rate limits. Twitter Developer Platform
  10. Get Tweet timelines. Twitter Developer Platform
  11. Tweepy 3.10.0, AttributeError: module ‘tweepy’ has no attribute ‘Client’. stackoverflow
  12. Twitter API v1.1 Reference
  13. [Installing packages using pip and virtual environments.
  14. ceaksan/GetTweets.py. GitHub
  15. Listen for important events. Twitter Developer Platform
  16. Items or Pages. Cursor Tutorial. Tweepy
  17. Streaming With Tweepy
  18. Stream Tweets in real-time. Twitter Developer Platform