Contents

Video Benchmarking Tool

These scripts are designed to automate the running of benchmarking tests and the collection of the performance data. This performance data includes the FPS of each video stream, and also the CPU %, Memory %, and Memory read/write bandwidth.

The Processor Counter Monitor (PCM) is required for measuring memory read/write bandwidth, which can be downloaded and built here.

If you do not have PCM on your system, those columns will be blank in the output.csv.

Refer README-Using-video-accelerators for using video accelerators and follow the required prerequisites to work with GPU, MYRIAD, and HDDL devices.

Note

  • To run the gstreamer pipeline mentioned in sample_test/config.json([WORK_DIR]/IEdgeInsights/tools/Benchmarking/video-benchmarking-tool/sample_test/config.json), copy the required model files to [WORKDIR]/IEdgeInsights/VideoIngestion/models. For more information, refer to models-readme.

  • In IPC mode, for accelerators: MYRIAD, GPU and USB 3.0 Vision cameras, add user: root in VideoProfiler-docker-compose.yml([WORK_DIR]/IEdgeInsights/tools/VideoProfiler/docker-compose.yml) as the subscriber needs to run as root if the publisher is running as root.

  • For GPU device, there is an initial delay while the model is compiled and loaded. This can affect the first benchmarking results especially on low stream count. This will be reduced on subsequent runs using kernel caching. To ensure that the kernel cache files are created, remove read_only: true in the docker-compose.yml file for VI so that files can be generated.

  • The docker-compose.yml files of VI and VideoProfiler is picked from their respective repos. So any changes needed should be applied in their respective repos.

Steps for running a benchmarking test case:

#. Ensure the VideoProfiler requirements are installed by following the README at README. #.

Start the RTSP server on a separate system on the network:

./stream_rtsp.sh <number-of-streams> <starting-port-number> <bitrate> <width> <height> <framerate>

For example:

./stream_rtsp.sh 16 8554 4096 1920 1080 30

  1. Run execute_test.sh with the desired benchmarking config:

    USAGE:
    ./execute_test.sh TEST_DIR STREAMS SLEEP PCM_HOME [EII_HOME]

Where:
    TEST_DIR-The absolute path to directory containing services.yml for the services to be tested, and the config.json and docker-compose.yml for VI and VA if applicable.
    STREAMS-The number of streams (1, 2, 4, 8, 16)
    SLEEP-The number of seconds to wait after the containers come up
    PCM_HOME-The absolute path to the PCM repository where pcm.x is built
    EII_HOME-[Optional] The absolute path to EII home directory, if running from a non-default location.

    For example:

    sudo -E ./execute_test.sh $PWD/sample_test 16 60 /opt/intel/pcm /home/intel/IEdgeInsights

  2. The execution log, performance logs, and the output.csv will be saved in TEST_DIR/< timestamp >/ so that the same test case can be run multiple times without overwriting the output. If any errors occur during the test, you can view its details from the execution.log file. For successful test, you can view the results in final_output.csv.

Run Video Benchmarking Tool with EdgeVideoAnalyticsMicroservice

Pre-requisites:

  1. Please make sure EVAM services are built on the system as the benchmarking tool will only start the services and not build them.

  2. Please update the RTSP camera IP in pipeline.json([WORK_DIR]/IEdgeInsights/pipeline.json) and in RTSP_CAMERA_IP field .env([WORK_DIR]/IEdgeInsights/.env)

Run evam_execute_test.sh with the desired benchmarking config:

```sh
USAGE:
    ./evam_execute_test.sh TEST_DIR STREAMS SLEEP PCM_HOME [EII_HOME]

WHERE:
    TEST_DIR  - The absolute path to directory containing services.yml for the services to be tested, and the config.json and docker-compose.yml for VI and VA if applicable.
    STREAMS   - The number of streams (1, 2, 4, 8, 16)
    SLEEP     - The number of seconds to wait after the containers come up
    PCM_HOME  - The absolute path to the PCM repository where pcm.x is built
    EII_HOME - [Optional] The absolute path to EII home directory, if running from a non-default location
```


For example:

```sh
sudo -E ./evam_execute_test.sh $PWD/evam_sample_test 16 60 /opt/intel/pcm [WORKDIR]/IEdgeInsights
```