This project demonstrates an ETL pipeline that extracts tweets from X (formerly Twitter), transforms them into structured data, and loads them to an AWS S3 bucket. The entire process is orchestrated using Apache Airflow running on an AWS EC2 instance.
The pipeline runs entirely on a Free Tier EC2 instance (Ubuntu AMI, t2.micro) with an added 2 GiB swap file to handle processing.
Category:
Client:
Project Duration:
Architecture
Extraction: Tweets are retrieved from the X API using Tweepy
Transformation: Data is structured into a Pandas DataFrame with fields: user, text, like_count, retweet_count, created_at
Loading: Data is saved to an S3 bucket (rume-airflow-bucket) using s3fs
Orchestration: The ETL process is scheduled and managed with Airflow DAGs
Notes
This project uses AWS Free Tier for EC2 and S3
Swap memory was added to handle t2.micro instance limitations
Ensure your Twitter API token has permissions to access the user timeline
