This project is part of the course NoSQL Data Modelling in the Data Engineering Nanodegree on Udacity. The final version of the repository will be submitted to Udacity for review and grading.
- Non-Relational Databases & NoSQL
- Distributed Databases
- CAP Theorem
- Denormalization in Apache Cassandra
- Cassandra Query Language
- Primary Keys and Clustering Columns
- WHERE clauses
In this project, I applied what you've learned on data modeling with Apache Cassandra and complete an ETL pipeline using Python. To complete the project, I needed to model your data by creating tables in Apache Cassandra to run queries. I used an ETL pipeline which transfers data from a set of CSV files within a directory to create a streamlined CSV file to model and insert data into Apache Cassandra tables.
A startup called Sparkify wants to analyze the data they've been collecting on songs and user activity on their new music streaming app. The analysis team is particularly interested in understanding what songs users are listening to. Currently, there is no easy way to query the data to generate the results, since the data reside in a directory of CSV files on user activity on the app.
They'd like a data engineer to create an Apache Cassandra database which can create queries on song play data to answer the questions, and wish to bring you on the project. Your role is to create a database for this analysis. You'll be able to test your database by running queries given to you by the analytics team from Sparkify to create the results.
- Pandas
- Cassandra
- Numpy
- Data-Modelling-with-Apache-Cassandra-Project.ipynb: Notebook containing the ETL pipelines.
To use the code written to create the database and tables, open the notebook Data-Modelling-with-Apache-Cassandra-Project.ipynb.
Pull requests to show improvements of the existing code are welcome. For major changes, please open an issue first to discuss what you would like to change.