Contents
========


* `Contents <#contents>`__

  * `Video Benchmarking Tool <#video-benchmarking-tool>`__

Video Benchmarking Tool
-----------------------

These scripts are designed to automate the running of benchmarking tests and the collection of the performance data. This performance data includes the FPS of each video stream, and also the CPU %, Memory %, and Memory read/write bandwidth.

The Processor Counter Monitor (PCM) is required for measuring memory read/write bandwidth, which can be downloaded and built `here <https://github.com/opcm/pcm>`_.

If you do not have PCM on your system, those columns will be blank in the output.csv.

Refer `README-Using-video-accelerators </4.0/IEdgeInsights/README.html#use-video-accelerators-in-ingestion-and-analytics-containers>`_ for using video accelerators and follow the required prerequisites to work with GPU, MYRIAD, and HDDL devices.

.. note:: 


   * To run the gstreamer pipeline mentioned in sample_test/config.json(\ ``[WORK_DIR]/IEdgeInsights/tools/Benchmarking/video-benchmarking-tool/sample_test/config.json``\ ), copy the required model files to ``[WORKDIR]/IEdgeInsights/VideoIngestion/models``. For more information, refer to `models-readme </4.0/IEdgeInsights/VideoIngestion/models/README.html>`_.
   * In IPC mode, for accelerators: ``MYRIAD``\ , ``GPU`` and USB 3.0 Vision cameras, add ``user: root`` in VideoProfiler-docker-compose.yml(\ ``[WORK_DIR]/IEdgeInsights/tools/VideoProfiler/docker-compose.yml``\ ) as the subscriber needs to run as ``root`` if the publisher is running as ``root``.
   * For ``GPU`` device, there is an initial delay while the model is compiled and loaded. This can affect the first benchmarking results especially on low stream count. This will be reduced on subsequent runs using kernel caching. To ensure that the kernel cache files are created, remove ``read_only: true`` in the ``docker-compose.yml`` file for VI so that files can be generated.
   * The ``docker-compose.yml`` files of VI and VideoProfiler is picked from their respective repos. So any changes needed should be applied in their respective repos.


Steps for running a benchmarking test case:


#. Ensure the VideoProfiler requirements are installed by following the README at `README </4.0/IEdgeInsights/tools/VideoProfiler/README.html>`_.
#. 
   Start the RTSP server on a separate system on the network:

   .. code-block:: sh

       ./stream_rtsp.sh <number-of-streams> <starting-port-number> <bitrate> <width> <height> <framerate>

   For example:

   .. code-block:: sh

       ./stream_rtsp.sh 16 8554 4096 1920 1080 30

#. 
   Run execute_test.sh with the desired benchmarking config:

   .. code-block:: sh

       USAGE:
           ./execute_test.sh TEST_DIR STREAMS SLEEP PCM_HOME [EII_HOME]

       Where:
           TEST_DIR-The absolute path to directory containing services.yml for the services to be tested, and the config.json and docker-compose.yml for VI and VA if applicable.
           STREAMS-The number of streams (1, 2, 4, 8, 16)
           SLEEP-The number of seconds to wait after the containers come up
           PCM_HOME-The absolute path to the PCM repository where pcm.x is built
           EII_HOME-[Optional] The absolute path to EII home directory, if running from a non-default location.

   For example:

   .. code-block:: sh

       sudo -E ./execute_test.sh $PWD/sample_test 16 60 /opt/intel/pcm /home/intel/IEdgeInsights

#. 
   The execution log, performance logs, and the output.csv will be saved in ``TEST_DIR/< timestamp >/`` so that the same test case can be run multiple times without overwriting the output. If any errors occur during the test, you can view its details from the execution.log file. For successful test, you can view the results in final_output.csv.

Run Video Benchmarking Tool with EdgeVideoAnalyticsMicroservice
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Pre-requisites:


#. 
   Please make sure EVAM services are built on the system as the benchmarking tool will only start the services and not build them.

#. 
   Please update the RTSP camera IP in pipeline.json(\ ``[WORK_DIR]/IEdgeInsights/pipeline.json``\ ) and in ``RTSP_CAMERA_IP`` field .env(\ ``[WORK_DIR]/IEdgeInsights/.env``\ )

Run evam_execute_test.sh with the desired benchmarking config:

.. code-block::

   ```sh
   USAGE:
       ./evam_execute_test.sh TEST_DIR STREAMS SLEEP PCM_HOME [EII_HOME]

   WHERE:
       TEST_DIR  - The absolute path to directory containing services.yml for the services to be tested, and the config.json and docker-compose.yml for VI and VA if applicable.
       STREAMS   - The number of streams (1, 2, 4, 8, 16)
       SLEEP     - The number of seconds to wait after the containers come up
       PCM_HOME  - The absolute path to the PCM repository where pcm.x is built
       EII_HOME - [Optional] The absolute path to EII home directory, if running from a non-default location
   ```


   For example:

.. code-block::

   ```sh
   sudo -E ./evam_execute_test.sh $PWD/evam_sample_test 16 60 /opt/intel/pcm [WORKDIR]/IEdgeInsights
   ```