All projects start with inspiration and pain. My inspiration came from a stint doing #100DaysOfCode. The pain, on the other hand, came from having no consistent centralised feed of new information for optometrists.
#100DaysOfCode in its name describes exactly what you have to do: a commitment to code for one hundred days. Members engaged in this deliberate practice everyday would compose what has been learnt or done code-wise and post this onto Twitter.
For those who do not know or use Twitter, this is a social media platform where users create short posts or tweets that are limited to 280 characters. Users can leave comments on these tweets as well as favourite tweets and even republish or retweet posts worthy of redistribution. Tweets can include hashtags (#) inside these messages, which can be used to categorise posts to a particular topic (e.g. #optometry).
Back to this coding business, I would include the hashtag, #100DaysOfCode, to show my church-going commitment to improving my coding skills.
Usually, my posts would get lost in the void of the internet, unread and unloved. However, with a favourite and a retweet, an altruistic bot, aptly named #100DaysOfCode, would provide a breath of life for my code-bragging before it’s, again, lost to the vastness of the internet.
And here lies the inspiration for this project a bot that can favourite and retweet. And to combine this with my pain point: favourite and retweet optometry related posts. This bot that I wish to create will act as that ‘stream’ of optometry related information that I always wanted.
Now, we have a goal.
But goals are nothing without execution, so here is a rough overview of what we will be doing to see our creation come alive:
- Creating a Twitter account and apply for access to the API
- Use Python and a particular package called
tweepyto create a script to authenticate and access the Twitter API
- Create a script to favourite and retweet a post based on certain hashtags
- Use Docker to create an image of the bot script along with its dependent packages
- Use Amazon EC2 to host this bot and keep it live
Some Preparation and Environment
Before we really start to dive into this project, let’s quickly set up our environment. I use Linux but working with any other platform will be similar.
This is the file structure we will eventually want to reach:
TwitterOptomBot/ | |--- bots/ | |--- config.py | |--- fav_and_retweet.py | |--- .env |--- venv/ | |--- ... |--- requirements.txt |--- Dockerfile |
Let’s create a directory to store all our files
mkdir ~/TwitterOptomBot mkdir ./bots cd -
This should return us to the
We can create a virtual environment,
venv. A virtual environment is important because this will contain all the python packages needed for this project. Creating a virtual environment ensures the packages will be independent from other projects that we work on in the future.
python3 -m venv ./venv
We can ensure this environment in activated (and this will be done every time you revisit this project). We do this by typing the following, making sure we are in the correct directory.
After this, let’s install our packages that we will be using today.
pip3 install python-dotenv pip3 install tweepy pip3 freeze > requirements.txt
This will install
python-dotenv, which is handy for creating environment variables and storing API tokens (more on this later), and
tweepy, which is what we need to access the Twitter API.
The dependencies are saved to
requirements.txt, which helps later on for installing these packages on other machines.
Twitter Account and Access
First, we create an account, @OptomBot. This will be separate to any personal accounts.
Following this, we need to apply for access. This will supply us with API tokens and keys needed to access our bot with Python.
We apply for a developer account.
We select what our reason is and go for ‘Making a bot’. This will lead to another form which need to be filled out. There are variety of questions asked to really test the legitimacy of what we will be using the developer access for.
Afterwards we are provided with five tokens that we need to keep safe. We can do this by creating a
.env file in side the
TwitterOptomBot/bots directory. I use Vim (click here to see my post about Vim), so this is how I create the file:
This opens a
.env file, and this space is where we can save our token and secret keys. We can use this format,
VARIABLE=*token*, like so:
APIkey=<API Key> APISecretKey=<API Secret Key> BearerToken=<Bearer Token> AccessToken=<Access Token> AccessTokenSecret=<Access Token Secret>
We check that everything works out by creating a small Python script.
This creates a script called
config.py. We will use the following code:
# /TwitterOptomBot/bots/config.py # this script is used to authenticate the python bot with @OptomBot # credentials are saved to a .env file and used by the script as environment variables import tweepy import os from dotenv import load_dotenv def create_api(): load_dotenv() API_KEY = os.getenv('APIKey') API_SECRET_KEY = os.getenv('APISecretKey') ACCESS_TOKEN = os.getenv('AccessToken') ACCESS_TOKEN_SECRET = os.getenv('AccessTokenSecret') auth = tweepy.OAuthHandler(API_KEY, API_SECRET_KEY) auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET) api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True) try: api.verify_credentials() except Exception as e: logger.error('Error creating API', exc_info=True) raise e print('Authentication works!') return api create_api()
If everything does to plain, the terminal will print
Authentication works!. If there are problems, then there will be a screen full of errors.
You can see
load_dotenv(), which loads the variables in
.env, which are the API tokens and secret keys as environment variables for the code to use.
Creating the bot
All this preparation has lead up to this point: to create the bot. The first part, we have already done. With the code we used to check for authentication, we only need to rewrite some parts.
# /TwitterOptomBot/bots/config.py # this script is used to authenticate the python bot with @OptomBot # credentials are saved to a .env file and used by the script as environment variables import tweepy import os import logging from dotenv import load_dotenv logger = logging.getLogger() def create_api(): load_dotenv() API_KEY = os.getenv('APIKey') API_SECRET_KEY = os.getenv('APISecretKey') ACCESS_TOKEN = os.getenv('AccessToken') ACCESS_TOKEN_SECRET = os.getenv('AccessTokenSecret') auth = tweepy.OAuthHandler(API_KEY, API_SECRET_KEY) auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET) api = tweepy.API(auth, wait_on_rate_limit=True, wait_on_rate_limit_notify=True) try: api.verify_credentials() except Exception as e: logger.error('Error creating API', exc_info=True) raise e logging.info('API created') return api
There will be some differences. First, we are using the
logging to note what the script is doing without printing statements. In addition, the function
create_api() is not being called in this script. We plan to use other scripts to call this function, instead.
The next stage is creating a script that performs the action of favouriting and retweeting posts based on Optometry.
To keep things simple, we will use the hashtags: #Optometry and #Ophthalmology. It is fair to think that posts with at least one of these hashtags will have contain relevant information for optometrists
#!/usr/bin/env python3 # TwitterOptomBot/bots/fav_and_retweet.py # the aim of this script is to fav and retweet # posts with the hashtag #Optometry and #Opthalmology import tweepy import logging from config import create_api import json logging.basicConfig(level=logging.INFO) logger = logging.getLogger() class FavRetweetListener(tweepy.StreamListener): def __init__(self, api): self.api = api self.me = api.me() def on_status(self, tweet): logger.info('Processing tweet id %s' % tweet.id) if tweet.in_reply_to_status_id is not None or tweet.user.id == self.me.id: # ignores replies or author posts return if not tweet.favorited: # mark it as liked, as not been done try: tweet.favorite() except Exception as e: logger.error('Error on fav', exc_info=True) if not tweet.retweeted: # make sure we don't get endless retweeting try: tweet.retweet() # print('Tweet to retweet: %s' % tweet.text) except Exception as e: logger.error('Error on fav and retweeted', exc_info=True) def on_error(self, status): logger.error(status) def main(keywords): api = create_api() tweets_listener = FavRetweetListener(api) stream = tweepy.Stream(api.auth, tweets_listener) stream.filter(track=keywords, languages=['en']) if __name__ == '__main__': main(['#Optometry', '#Ophthalmology'])
We use the
StreamListener functionality to ‘listen’ in on the twitter posts.
We have the line
api = create_api(), which accesses our function we made before in the earlier code block to authenticate the bot.
We create a class,
FavRetweetListener which inherits from
Stream() method to essentially ‘listen’ to Twitter posts live. We apply a filter to look out for posts that contain the hashtags, #Optometry and #Ophthalmology.
With the posts that qualify through the filter we further ensure that these are not retweets (this prevents retweeting retweets); they have not been favourited before, and that the author of the tweets is not the bot itself.
except clauses in order to keep the script running and to return errors via a log that can be reviewed.
In order to run the script we simply use the command:
We have to also ensure that we are in the
/bots/ directory and also have the appropriate environment activated.
Packaging our script with Docker
We have written a script for the bot that successful runs on our machine. But that’s where the problem lies: it runs only on our machine. If you were to turn off your computer, the script that runs the bot will also shut down.
We need to move our script to a machine that is constantly running. Before we do this, we need to package our script with all its dependencies so it runs on any machine.
Docker is the key to this.
What is Docker?
Docker makes it easier to deploy applications as their own container. They include all their software and dependencies meaning we can build our application on one system and push it to another system no matter if it were running Linux, Windows or MacOS.
There is excellent documentation on how to install Docker on to your machine, here (this example is for Ubuntu).
From there, we create a file called
Dockerfile, which is a set of instructions to package our our code.
# TwitterOptomBot/Dockerfile FROM python:3.7-alpine COPY bots/config.py /bots/ COPY bots/fav_and_retweet.py /bots/ COPY bots/.env /bots/ COPY requirements.txt /tmp RUN pip3 install -r /tmp/requirements.txt WORKDIR /bots CMD ["python3", "fav_and_retweet.py"]
Here we create an image for our code. The image will contain the Python 3.7 intepreter. Along with this, we will be copying files needed to run the bot,
We also copy
requirements.txt, which lists all our packages used for this bot, and we will use
pip3 to install these required packages.
We use the following line to build the docker image and compress it:
docker build . -t fav-retweet-bot docker image save fav-retweet-bot:latest -o fav-retweet-bot.tar gzip fav-retweet-bot.tar
Deploying the bot using Amazon AWS EC2
Now we have the bot packaged as a Docker image and compressed into a format that makes it easily mobile.
We can use a service like Amazon AWS EC2.
For instructions on how to set up an get comfortable with the EC2 instance, check out this tutorial series. This series is a 30 day course. However, you are not required to do the whole course to complete this project. At least knowing how to set up the EC2 instance and access it using SSH along with creating a key pair for security when logging in is the minimum.
When we SSH into our EC2 instance, we need to install docker on to this machine. We do this:
sudo apt-get update sudo apt install docker.io sudo adduser ubuntu docker exit
The next stage is uploading the docker image onto the EC2 instance. We can do this on our machine by using the following (with the appropriate fields replaced:
scp -i "<location of keypair.pem>" fav-retweet-bot.tar.gz \\ [email protected]<what-your-ip-is>:/tmp
We then head back into our EC2 instance, decompress the file and load the image:
gunzip /tmp/fav-retweet-bot.tar.gz docker image load -i /tmp/fav-retweet-bot.tar
We can now deploy and run the docker image using the following line on our EC2 instance:
docker run -d --restart always fav-retweet-bot
-d tag ensures the bot runs in the background, meaning if the SSH session is closed, then the bot will still run as long as the EC2 instance is up. Using
--restart always ensures the bot will restart and run if it has been closed.
We can check on the bot using the following commands in our EC2 instance:
docker ps docker logs <container id>
This will print the logs for that the bot generates.
Congratulations! We have made out bot. Now it is out there retweeting content and helping spread awareness about optometry related posts.
We used Python to create our bot script. We then used Docker to easily package our script and all its dependencies. From there, we were able to relocate our bot on to a forever running cloud service, Amazon AWS EC2, where it can run even with our computer turned off.
I am really hoping other optometrists or anyone related to the field will find this bot useful. I really want to create a stream of information for optometrists allowing ease in work to be shared and received easily.
There is major room for improvement for this bot. And this is only the beginning. If you have any more ideas for this bot, please let me know. It would be greatly appreciated.
I also hope you found this helpful or entertaining in some way. If you think someone else will find this helpful, please share this to them.
- Install Docker Engine on Ubuntu. (2020, August 14). Docker Documentation. https://docs.docker.com/engine/install/ubuntu/
- Miguel, G. (n.d.). How to Make a Twitter Bot in Python With Tweepy – Real Python. Realpython.com. Retrieved December 16, 2020, from https://realpython.com/twitter-bot-python-tweepy/
- Roesslein, J. (n.d.). Streaming With Tweepy — tweepy 3.8.0 documentation. Docs.Tweepy.org. Retrieved January 7, 2021, from http://docs.tweepy.org/en/latest/streaming_how_to.html