CUDA and TensorRT Starter Workspace

This repository guides freshmen who does not have background of parallel programming in C++ to learn CUDA and TensorRT from the beginning.

• CUDA and TensorRT Starter Workspace
  • How to install
  • How to run
  • Chapter description
    • chapter1-build-environment
    • chapter2-cuda-programming
    • chapter3-tensorrt-basics-and-onnx
    • chapter4-tensorrt-optimiztion
    • chapter5-tensorrt-api-basics
    • chapter6-deploy-classification-and-inference-design
    • chapter7-deploy-yolo-detection

This repository is still working in progress(~24/02/21). I will add some more samples and more detailed description in the future. Please feel free to contribute to this repository

How to install

Please pull the repository firstly

git clone git@github.com:kalfazed/tensorrt_starter.git

After clone the repository, please modify the opencv, cuda, cudnn, and TensorRT version and install directory in config/Makefile.config located in the root direcoty of the repository. The recommaned version in this repository is opencv==4.x, cuda==11.6, cudnn==8.9, TensorRT==8.6.1.6

# Please change the cuda version if needed
# In default, cuDNN library is located in /usr/local/cuda/lib64
CXX                         :=  g++
CUDA_VER                    :=  11

# Please modify the opencv and tensorrt install directory
OPENCV_INSTALL_DIR          :=  /usr/local/include/opencv4
TENSORRT_INSTALL_DIR        :=  /home/kalfazed/packages/TensorRT-8.6.1.6

Besides, please also change the ARCH in config/Makefile.config. This parameter will be used by nvcc, which is a compiler for cuda program.

How to run

Inside each subfolder of each chapter, the basic directory structure is as follow: (For some chapters, it will be different)

|-config
    |- Makefile.config
|-src
    |- cpp
        |- xxx.c
    |- python
        |- yyy.py
|-Makefile

Please run make firstly, then it will generate a binary named trt-cuda or trt-infer, depending on different chapters. Pleae run the binary directly or run make run command.

Chapter description

chapter1-build-environment

• 1.0-build-environment

chapter2-cuda-programming

• 2.1-dim_and_index
• 2.2-cpp_cuda_interactive
• 2.3-matmul-basic
• 2.4-error-handler
• 2.5-device-info
• 2.6-nsight-system-and-compute
• 2.7-matmul-shared-memory
• 2.8-bank-conflict
• 2.9-stream-and-event
• 2.10-bilinear-interpolation
• 2.11-bilinear-interpolation-template
• 2.12-affine-transformation
• 2.13-implicit-gemm-conv
• 2.14-pcd-voxelization

chapter3-tensorrt-basics-and-onnx

• 3.1-generate-onnx
• 3.2-export-onnx
• 3.3-read-and-parse-onnx
• 3.4-export-unsupported-node
• 3.5-onnxsurgeon
• 3.6-export-onnx-from-oss
• 3.7-trtexec-analysis

chapter4-tensorrt-optimiztion

• 4.1-polygraphy[WIP]

chapter5-tensorrt-api-basics

• 5.1-mnist-sample
• 5.2-load-model
• 5.3-infer-model
• 5.4-print-structure
• 5.5-build-model
• 5.6-build-sub-graph
• 5.7-custom-basic-trt-plugin
• 5.8-plugin-unit-test

chapter6-deploy-classification-and-inference-design

• 6.0-preprocess-speed-compare
• 6.1-deploy-classification
• 6.2-deploy-classification-advanced
• 6.3-int8-calibration
• 6.4-trt-engine-inspector

chapter7-deploy-yolo-detection

• 7.1-load-save-tensor
• 7.2-affine-transformation
• 7.3-deploy-yolo-basic
• 7.4-quantization-analysis
• 7.5-deploy-yolo-multitask