by Sajid Nawaz Khan, Cyber Threat Intelligence Analyst.
This README is designed to support the above workshop which took place in May 2021. A copy of the presentation is available to defenders working within the financial sector.
Please ensure you have installed VirtualBox with the Extension Pack, followed by a virtual machine running x64 Ubuntu 20.04 LTS. Ideally, the machine should be configured with at least 2GB of RAM.
Detailed installation instructions are available on the Mihari Wiki. The code below is provided for convenience to help support the workshop, and is correct for version 2.3.0.
# Core
sudo apt-get update && sudo apt-get upgrade
sudo apt-get install build-essential gcc make perl dkms
sudo apt-get install git
sudo apt-get install sqlite3 libsqlite3-dev
sudo apt-get install ruby-full
sudo apt-get install python3-pip
# Artefact Enrichment
# Update PATH as necessary in /etc/environment
pip3 install requests
pip3 install tldextract
# For Data Analysis
pip3 install jupyter numpy pandas matplotlib seaborn
pip3 install tabulate
pip3 install datasette
# VS Code Plugins (Install VS Code via the Ubuntu Software app)
code --install-extension ms-python.python
code --install-extension ms-toolsai.jupyter
code --install-extension alexcvzz.vscode-sqliteOn a minimal install of Ubuntu, you may also need to install cURL and whois: sudo apt-get install curl whois.
Additionally, you may also wish to clone a copy of this repository:
git clone https://github.com/ssnkhan/infrastructure-hunting.git
# Can take around 5 minutes, please be patient
sudo gem install mihariPost-installation, please configure API keys as detailed within the Mihari Wiki.
In addition to supporting API key configuration via a YAML file, Mihari also supports configuration via environmental variables, e.g., export SHODAN_API_KEY=YOUR_API_KEY within your queries.sh file, or by updating .profile or /etc/environment.
Once a local mihari.db sqlite database has been instantiated, copy or move it to the setup/ directory and run the ./setup.sh script to update its database schema. This is necessary to use the enrichment scripts detailed below.
These scripts are designed to enrich artefacts collected by Mihari with basic whois and passive DNS data.
| File | Description |
|---|---|
update_whois.py |
Implements the Team Cymru whois API to capture a host's ASN, ASN name, BGP prefix, registrar and country. Note that location data is self attested by the ASN, and may not be accurate. |
update_whois_dns.py |
Additionally implements the PassiveTotal API for passive DNS lookups associated with a host within the past ten days (API key required). |
update_domains.py |
Uses the tldextract library to distil a FQDN to its components (subdomain, domain and TLD). |
| File | Description |
|---|---|
Mihari.ipnb |
Jupyter notebook which allows SQL queries to be executed and loaded into a Pandas dataframe. |
Seaborn.ipnb |
Jupyter notebook which allows SQL queries to be executed and graphed as a Seaborn bar chart. |
Queries.md |
SQL queries to support analysis. Use the Jupyter notebooks above, or with Datasette. |
Datasette offers a simple yet powerful way of reviewing and querying the Mihari database through your browser. Start the Datasette server with datasette serve mihari.db -o. Add basic chart generation with pip3 install datasette-vega.
A local Maltego Transform to query an sqlite3 mihari.db, and which returns the detection names and observation dates associated with an IPv4 address.
Run backup.sh manually or via a cronjob to backup the mihari.db sqlite database to the backups folder. Update your paths as necessary.
I'd love to hear your thoughts and feedback. Feel free to say hello on Twitter @snkhan or via LinkedIn.
#cti #threatintelligence #blueteam #threathunting #shodan #censys