🛠️ Batch Data Processing with Python, MySQL, Docker, Airflow, and Streamlit

📘 Project Overview

This project showcases an automated data pipeline that scrapes trending posts from technology-focused subreddits using the Reddit API and presents them in an interactive web application built with Streamlit. The backend is powered by Airflow for scheduling, MySQL for storage, and Docker for seamless deployment.

🧠 What the App Does

📰 Browse top daily posts from subreddits like r/datascience, r/MachineLearning, r/deeplearning, and more.
💡 Understand complex tech jargon using Gemini AI-powered explanations.
📅 View posts based on date, score, and subreddit categories.

🧰 Technologies Used

🐍 Python: Scripting and backend logic
🗄️ MySQL: Structured data storage
🐳 Docker: Containerized environment for scraper and frontend
🕹️ Airflow: DAG automation for scraping and exporting
📊 Streamlit: Interactive UI and visualization
🛠️ PRAW: Reddit API wrapper
🧠 Gemini API: Contextual explanations of technical terms

📈 Data Flow Summary

Reddit posts are scraped daily, stored in a MySQL database, and exported to CSV for use in the Streamlit dashboard. The scraper runs via Airflow DAGs, and Docker ensures environment consistency across systems. A separate container hosts the frontend app, offering a seamless experience for users exploring the latest in data science and technology.