Note RF100 is part of the CVinW challenge!
This repository implements the Roboflow 100 benchmark developed by Roboflow. It contains code to download the dataset and reproduce mAP values for YOLOv5 and YOLOv7 Fine-Tuning and GLIP Evaluation on 100 of Roboflow Universe datasets.
RF100 was sponsored with ❤️ by Intel
RF100 paper is on arxiv, if you used RF100 in your research, please consider citing us
@misc{2211.13523,
Author = {Floriana Ciaglia and Francesco Saverio Zuppichini and Paul Guerrie and Mark McQuade and Jacob Solawetz},
Title = {Roboflow 100: A Rich, Multi-Domain Object Detection Benchmark},
Year = {2022},
Eprint = {arXiv:2211.13523},
}
RF100 contains the following datasets, carefully chosen from more than 90'000 datasets hosted on our universe hub. The datasets are splitted in 7 categories: Aerial, Videogames, Microscopic, Underwater, Documents, Electromagnetic and Real World.
| Category | Datasets | Images | Classes |
|---|---|---|---|
| Aerial | 7 | 9683 | 24 |
| Videogames | 7 | 11579 | 88 |
| Microscopic | 11 | 13378 | 28 |
| Underwater | 5 | 18003 | 39 |
| Documents | 8 | 24813 | 90 |
| Electromagnetic | 12 | 36381 | 41 |
| Real World | 50 | 110615 | 495 |
| Total | 100 | 224,714 | 805 |
Note some datasets failed, only on 88
| category | yolov5 | yolov7 | yolov8 | glip |
|---|---|---|---|---|
| aerial | 0.636 | 0.504286 | 0.707286 | 0.0238571 |
| documents | 0.716125 | 0.7225 | 0.79775 | 0.018625 |
| electromagnetic | 0.7423 | 0.639 | 0.7899 | 0.0697 |
| microscopic | 0.650727 | 0.59166 | 0.747455 | 0.0230909 |
| real world | 0.781659 | 0.705276 | 0.842171 | 0.117049 |
| underwater | 0.56 | 0.6624 | 0.7024 | 0.1876 |
| videogames | 0.863333 | 0.783167 | 0.881 | 0.1625 |
| Total | 0.7071 | 0.6583 | 0.7811 | 0.0860 |
🚨 Head over the Appendix at the bottom to see samples from each dataset
We provide a notebook to help you using RF100 with PyTorch
First, clone this repo and go inside it.
git clone https://github.com/roboflow-ai/roboflow-100-benchmark.git
cd roboflow-100-benchmark
git submodule update --init --recursiveYou will need an API key. RF100 can be accessed with any key from Roboflow, head over our doc to learn how to get one.
Then, export the key to your current shell
export ROBOFLOW_API_KEY=<YOUR_API_KEY>Note: The datasets are taken from datasets_links.txt, you can modify that file to add/remove datasets.
The easiest and faster way to download RF100 is using docker and our Dockerfile.
NOTE Be sure to do the post process steps after you installed docker, we will read and write to shared volumes so your user should also be in the docker group.
If you have an NVIDIA GPU, be sure to also install nvidia docker
Be sure to have the ROBOFLOW_API_KEY in your env, then run it (we are passing our user so you shouldn't have permission errors on rf100 folder)
docker run --rm -it \
-e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
-v $(pwd)/rf100:/workspace/rf100 \
-v /etc/group:/etc/group:ro \
-u "$(id -u):$(id -g)" \
roboflow/rf100-downloadroboflow/rf100-download is hosted on our docker hub
Internally, RF100 will downloaded to /app/rf100. You can also specify the format with the -f flag, by default coco is used.
docker run --rm -it \
-e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
-v ${PWD}/rf100:/workspace/rf100 \
-v /etc/group:/etc/group:ro \
-u "$(id -u):$(id -g)" \
roboflow/rf100-download -f yolov5If you want to build the container and not use the image on the hub, run
docker build -t rf100-download -f Dockerfile.rf100.download .To download RF100 in your local enviroment (python >=3.6), you need to install roboflow
pip install roboflowThen,
chmod 770 ./scripts/download_datasets.sh
./scripts/download_datasets.sh
./scripts/download_datasets.sh -f yolov5 $ change format
./scripts/download_datasets.sh -l <path_to_my_location> change download locationSupported formats are
cocoyolov5(used by YOLOv7 as well)
We will use docker to ensure the same enviroment is used.
First, build the container
docker build -t rf100-benchmark -f Dockerfile.rf100.benchmark .
Then, follow the guide for each model.
All results are stored inside ./runs.
Warning The run must be uninterrupted otherwise the downloaded datasets may be unfinished or corrupted
Warning If you have
permission deniederrors, you can either removerunswithsudo rm -rf ./runsor give permissions to everybodysudo chmod -R +x ./runs.
Warning Due to poor code support, random errors and time, we weren't able to run all the models on all datasets. Only 88 datasets were successfully benchmarked on all four models
YOLOv5 Fine-Tuning
Note, we will map the current folder to the container file system to persist data
mkdir -p runs &&
nvidia-docker run --gpus all --rm -it --ipc host --network host --shm-size 64g \
-e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
-v ${PWD}/runs:/workspace/runs \
-v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
rf100-benchmark ./yolov5-benchmark/train.shYOLOv7 Fine-Tuning
Note, we will map the current folder to the container file system to persist data
mkdir -p runs &&
nvidia-docker run --gpus all --rm -d --ipc host --network host --shm-size 64g \
-e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
-v ${PWD}/runs:/workspace/runs \
-v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
rf100-benchmark ./yolov7-benchmark/train.shYOLOv8 Fine-Tuning
Note, Doc coming soon but we have the results (was done internally by another team member)
Note The model failed on these datasets: bone-fracture-7fylg, brain-tumor-m2pbp, cable-damage, circuit-elements and printed-circuit-board
Note The model failed on these datasets: apex-videogame, apples-fvpl5, hand-gestures-jps7z, road-traffic,
wall-damage
mkdir -p runs &&
nvidia-docker run --gpus all --rm -it --ipc host --network host --shm-size 64g \
-e ROBOFLOW_API_KEY=$ROBOFLOW_API_KEY \
-v ${PWD}/runs:/workspace/runs \
-v ${PWD}/datasets_links_640.txt:/workspace/datasets_links_640.txt \
rf100-benchmark ./GLIP-benchmark/train.shNote We create a notebook to analysis why and where GLIP did bad and good
If you are interesting in only the test annotations for rf100, e.g. you are using pycocotools to compute maps, we have create a zip file with root/<dataset_name>/<test_annotations>.json. We hope this will make it easier for you to evaluate your models.
Note 5 datasets failed on v8, so the comparison is only on 95. Check v8 for more info.
Note We only train one seed for each model, take these results with a grain of salt
The following box plots shows mAP@.50 for each YOLOs model on RF100
The following bar plots shows mAP@.50 for each YOLOs model on each RF100 category
We thank all the authors of the original datasets, below a table linking the Roboflow 100 dataset's name to its original counterpart