Mage is an open-source data management platform that helps you clean data and prepare it for training AI/ML models.
The current version of Mage includes a data cleaning UI tool that can run locally on your laptop or can be hosted in your own cloud environment.
Using a data cleaning tool enables you to quickly visualize data quality issues, easily fix them, and create repeatable data cleaning pipelines that can be used in production environments (e.g. online re-training, inference, etc).
$ pip install git+https://github.com/mage-ai/mage-ai.gitLoad your data, connect it to Mage, and launch the tool locally.
From anywhere you can execute Python code (e.g. terminal, Jupyter notebook, etc.), run the following:
import mage_ai
import pandas as pd
df = pd.read_csv('/path_to_data')
mage_ai.connect_data(df, name='name_of_dataset')
mage_ai.launch()Open http://localhost:5000 in your browser to access the tool locally.
To stop the tool, run this command: mage_ai.kill()
After building a data cleaning pipeline from the UI, you can clean your data anywhere you can execute Python code:
import mage_ai
import pandas as pd
df = pd.read_csv('/path_to_data')
mage_ai.clean(df, pipeline_uuid='name_of_cleaning_pipeline') #=> returns cleaned dataframe- Here is a step-by-step guide on how to use the tool.
- Check out the tutorials to quickly become a master of magic.
Inspect your data using different charts (e.g. time series, bar chart, box plot, etc.).
Here’s a list of available charts.
Quickly diagnose data quality issues with summary reports.
Here’s a list of available reports.
Easily add common cleaning functions to your pipeline with a few clicks. Cleaning actions include imputing missing values, reformatting strings, removing duplicates, and many more.
If a cleaning action you need doesn’t exist in the library, you can write and save custom cleaning functions in the UI.
Here’s a list of available cleaning actions.
The tool will automatically suggest different ways to clean your data and improve quality metrics.
Here’s a list of available suggestions.
Big features being worked on or in the design phase.
- Encoding actions (e.g. one-hot encoding, label hasher, ordinal encoding, embeddings, etc.)
- Data quality monitoring and alerting
- Apply cleaning actions to columns and values that match a condition
Here’s a detailed list of features and bugs that are in progress or upcoming.
We welcome all contributions to Mage; from small UI enhancements to brand new cleaning actions. We love seeing community members level up and give people power-ups!
Check out the contributing guide to get started by setting up your development environment and exploring the code base.
Got questions? Live chat with us in Slack.
Anything you contribute, the Mage team and community will maintain. We’re in it together!
We love the community of Magers (/ˈmājər/);
a group of mages who help each other realize their full potential!
To live chat with the Mage team and community, please join the free Mage Slack channel.
For real-time news and fun memes, check out the Mage Twitter.
To report bugs or add your awesome code for others to enjoy, visit GitHub.
WIP