The project is designed to utilize the Qualcomm® Neural Processing SDK for AI, a deep learning software from Snapdragon platforms for Object Detection in Android. The Android application can be designed to use any built-in/connected camera to capture the objects and use Machine Learning model to get the prediction/inference and location of the respective objects.
- Before starting the Android application, please follow the instructions for setting up Qualcomm Neural Processing SDK using the link provided. https://docs.qualcomm.com/bundle/publicresource/topics/80-63442-2/setup.html?product=1601111740010412
- Android device 6.0 and above which uses below mentioned Snapdragon processors/Snapdragon HDK with display can be used to test the application
- Download CocoDataset 2017 and give its path to Generate_DLC.ipynb. Change variable "dataset_path" in Quantization Section in notebook.
- Snapdragon® SM8550
The above targets supports the application with CPU, GPU and DSP. For more information on the supported devices, please follow this link
demo : Contains demo GIF
app : Contains source files in standard Android app format
app\src\main\assets : Contains Model binary DLC
app\src\main\java\com\qc\objectdetectionYoloNas : Application java source code
app\src\main\cpp : native source code
sdk: Contains openCV sdk
Run jupyter notebook GenerateDLC.ipynb. This notebook will generate YoloNAS quantized dlc.
YoloNAS model is trained on COCO dataset for 80 classes of everyday objects. List of the classes can be found in dataset at : https://cocodataset.org/#explore
This application opens a camera preview, collects all the frames and converts them to bitmap. The network is built via Neural Network builder by passing Quant_YoloNas_s_320.dlc as the input. The bitmap is then given to the model for inference, which returns object prediction and localization of the respective object.
Permission to obtain camera preview frames is granted in the following file:
/app/src/main/AndroidManifest.xml
<uses-permission android:name="android.permission.CAMERA" />In order to use camera2 APIs, add the below feature
<uses-feature android:name="android.hardware.camera2" />Code snippet for neural network connection and loading model:
snpe = snpeBuilder.setOutputLayers({})
.setPerformanceProfile(zdl::DlSystem::PerformanceProfile_t::BURST)
.setExecutionPriorityHint(
zdl::DlSystem::ExecutionPriorityHint_t::HIGH)
.setRuntimeProcessorOrder(runtimeList)
.setUseUserSuppliedBuffers(useUserSuppliedBuffers)
.setPlatformConfig(platformConfig)
.setInitCacheMode(useCaching)
.setCPUFallbackMode(true)
.setUnconsumedTensorsAsOutputs(true) //To get output from two nodes
.build(); //builds the networkThe bitmap image is passed as openCV Mat to native and then converted to BGR Mat of size 320x320x3. Basic image processing depends on the kind of input shape required by the model, then writing that processed image into application buffer.
cv::Mat img320;
cv::resize(img,img320,cv::Size(320,320),cv::INTER_LINEAR);
float inputScale = 0.00392156862745f; //normalization value, this is 1/255
float * accumulator = reinterpret_cast<float *> (&dest_buffer[0]);
//opencv read in BGRA by default
cvtColor(img320, img320, CV_BGRA2BGR);
LOGI("num of channels: %d",img320.channels());
int lim = img320.rows*img320.cols*3;
for(int idx = 0; idx<lim; idx++)
accumulator[idx]= img320.data[idx]*inputScale;This included getting the class with highest confidence for each 2100 boxes and applying Non-Max Suppression to remove overlapping boxes.
for(int i =0;i<(2100);i++)
{
int start = i*80;
int end = (i+1)*80;
auto it = max_element (BBout_class.begin()+start, BBout_class.begin()+end);
int index = distance(BBout_class.begin()+start, it);
std::string classname = classnamemapping[index];
if(*it>=0.5 )
{
int x1 = BBout_boxcoords[i * 4 + 0];
int y1 = BBout_boxcoords[i * 4 + 1];
int x2 = BBout_boxcoords[i * 4 + 2];
int y2 = BBout_boxcoords[i * 4 + 3];
Boxlist.push_back(BoxCornerEncoding(x1, y1, x2, y2,*it,classname));
}
}
std::vector<BoxCornerEncoding> reslist = NonMaxSuppression(Boxlist,0.20);then we just scale the coords for original image
float top,bottom,left,right;
left = reslist[k].y1 * ratio_1; //y1
right = reslist[k].y2 * ratio_1; //y2
bottom = reslist[k].x1 * ratio_2; //x1
top = reslist[k].x2 * ratio_2; //x2 RectangleBox rbox = boxlist.get(j);
float y = rbox.left;
float y1 = rbox.right;
float x = rbox.top;
float x1 = rbox.bottom;
String fps_textLabel = "FPS: "+String.valueOf(rbox.fps);
canvas.drawText(fps_textLabel,10,70,mTextColor);
String processingTimeTextLabel= rbox.processing_time+"ms";
canvas.drawRect(x1, y, x, y1, mBorderColor);
canvas.drawText(rbox.label,x1+10, y+40, mTextColor);
canvas.drawText(processingTimeTextLabel,x1+10, y+90, mTextColor);-
Clone QIDK repo.
-
Run below script, from the directory where it is present, to resolve dependencies of this project.
-
This will copy snpe-release.aar file from $SNPE_ROOT to "snpe-release" directory in Android project.
NOTE - If you are using SNPE version less than 2.12, please change following line in resolveDependencies.sh.
From: cp $SNPE_ROOT/lib/android/snpe-release.aar snpe-release To : cp $SNPE_ROOT/android/snpe-release.aar snpe-release -
Download opencv and paste to sdk directory, to enable OpenCv for android Java.
bash resolveDependencies.sh- Run jupyter notebook GenerateDLC.ipynb to generate DLC(s) for quantized YOLO_NAS DLC. Also, change the dataset_path with Dataset Path.
- This script generates required dlc(s) and paste them to appropriate location.
-
Do gradle sync
-
Compile the project.
-
Output APK file should get generated : app-debug.apk
-
Prepare the Qualcomm Innovators development kit to install the application (Do not run APK on emulator)
-
If Unsigned or Signed DSP runtime is not getting detected, then please check the logcat logs for the FastRPC error. DSP runtime may not get detected due to SE Linux security policy. Please try out following commands to set permissive SE Linux policy.
It is recommended to run below commands.
adb disable-verity
adb reboot
adb root
adb remount
adb shell setenforce 0- Install and test application : app-debug.apk
adb install -r -t app-debug.apk- launch the application
Following is the basic "Object Detection" Android App
- On first launch of application, user needs to provide camera permissions
- Camera will open and pose will be seen, if human is visible on the screen
- User can select appropriate run-time for model, and observe performance difference
Same results for the application are :
- SSD - Single shot Multi box detector - https://arxiv.org/pdf/1512.02325.pdf
- https://github.com/Deci-AI/super-gradients
- https://zenodo.org/record/7789328