TensorRT文档学习

Tensorrt的能力

TensorRT operates in two phases. In the first phase, usually performed offline, you provide TensorRT with a model definition, and TensorRT optimizes it for a target GPU. In the second phase, you use the optimized model to run inference.

tensorrt主要有两个阶段，离线进行模型的编译，第二阶段推理

NVIDIA DLA (Deep Learning Accelerator) is a fixed-function accelerator engine targeted for deep learning operations. DLA is designed to do full hardware acceleration of convolutional neural networks. DLA supports various layers such as convolution, deconvolution, fullyconnected, activation, pooling, batch normalization, etc. DLA does not support Explicit Quantization.

DLA其实就是nv家的NPU

内存

tensorrt主要用的是device memory（GPU），资源有限，所以要小心使用

Build阶段

Runtime阶段

分为persist和scratch内存

Timing Cache

To reduce the builder time, TensorRT creates a layer timing cache to keep the layer profiling information during the builder phase. The information it contains is specific to the targeted builder devices, CUDA and TensorRT versions, and BuilderConfig parameters that can change the layer implementation such as BuilderFlag::kTF32 or BuilderFlag::kREFIT. If there are other layers with the same input/output tensor configuration and layer parameters, the TensorRT builder skips profiling and reuses the cached result for the repeated layers. If a timing query misses in the cache, the builder times the layer and updates the cache.

目的是减少build时间，先保存层的profile信息，如果有相同输入输出和参数的，就跳过profile直接用之前的结果

量化

量化工作流

Post-training quantization (PTQ) derives scale factors after the network has been trained. TensorRT provides a workflow for PTQ, called calibration, where it measures the distribution of activations within each activation tensor as the network executes on representative input data, then uses that distribution to estimate a scale value for the tensor.

Quantization-aware training (QAT) computes scale factors during training. This allows the training process to compensate for the effects of the quantization and dequantization operations.
TensorRT’s Quantization Toolkit is a PyTorch library that helps produce QAT models that can be optimized by TensorRT. You can also utilize the toolkit’s PTQ recipe to perform PTQ in PyTorch and export to ONNX.

测试性能

两个指标

Latency

A performance measurement for network inference is how much time elapses from an input being presented to the network until an output is available. This is the latency of the network for a single inference. Lower latencies are better. In some applications, low latency is a critical safety requirement. In other applications, latency is directly visible to users as a quality of service issue. For bulk processing, latency may not be important at all.

Throughput

Another performance measurement is how many inferences can be completed in a fixed unit of time. This is the throughput of the network. Higher throughput is better. Higher throughputs indicate a more efficient utilization of fixed compute resources. For bulk processing, the total time taken will be determined by the throughput of the network.

有一种视角看待这两个指标：设置最大latency然后测量throughput。介于用户体验和系统效率的妥协

选择时间测量的起点和终点很重要。一般有很多选择，对于一个典型的应用分为：pre，infer，post阶段。但是pre和post和特定应用类型有很大的关系，所以我们可以选择关注infer阶段。C++11精确测量时间的代码片段如下

#include <chrono>
auto startTime = std::chrono::high_resolution_clock::now();
context->enqueueV2(&buffers[0], stream, nullptr);
cudaStreamSynchronize(stream);
auto endTime = std::chrono::high_resolution_clock::now();
float totalTime = std::chrono::duration<float, std::milli>
(endTime - startTime).count();