Multimodal Data Visualization Microservice

This repository contains the source code for the Multimodal Data Visualization microservice used as a part of the Video Analytics Use Case and a component of AI Creek and Ledge Park S.

MDVM allows an EVAM user to visualize both framerate data and the actual recreated video from a video analytics pipeline in EVAM in a Grafana dashboard.

NOTE MDVM EVAM mode is only supported on a open, non-proxied network. Exotic network configurations may lead to issues running the application.

Build the Container Images

MDVM consists of 4 container images. To build the images, run the following command:

docker compose build

Run the Container Images including EVAM

The default docker compose file includes EVAM as a required service. You may need to build EVAM for the service to run. See EVAM for build instructions. If you are using a different EVAM image version, update docker-compose.yml accordingly.

To run the container image, check the environment variables in the .env file and then:

docker compose up

You can access the visualizer through your browser at port 3000:

  1. Go to http://:raw-html-m2r:`<IP-Address>:3000 <http://localhost:3000>`_ to access Grafana.

  2. Login with the credentials: root/evam123.

  3. On the Home Dashboard page, on the left corner, click the Dashboards icon.

  4. Click the Manage Dashboards tab, to view the list of all the preconfigured dashboards.

  5. Select Video Analytics Dashboard, to view the data from the Edge Video Analytics Microservice.

When a new pipeline is started in EVAM, you should see frame data being filling the current pipeline status and historical frame rate data boxes in the Grafana dashboard. To view the video stream, press the refresh button in the Stream data box - a new component will generate and autoconnect to the video source.

Run the Container Images without EVAM

If your EVAM instance is already running or is running on a different host, for example a dedicated inference machine on your local network, then you can point MDVM to that EVAM instance to capture the exported stream information data. Make sure that your EVAM instance is running in WebRTC mode and modify EVAM_ADDRESS in .env to point to your EVAM instance, and then run:

docker compose -f docker-compose-noevam.yml up

WebRTC TURN Server

If you are running MDVM on a host that is not accessible from the internet, or that lies behind an exotic network configuration like an enterprise proxy, you may need to run a TURN server to enable WebRTC communication between the webapp and EVAM. You must modify multimodal-data-visualization-streaming/eva_ui_default/www/webrtc.js to point to your TURN server.

If you have an external TURN server set up, provide those details in the rtc_configuration section at the top of the JavaScript file. For example:

var rtc_configuration = {
  iceServers: [
    {
      urls: "turn:turn.example.com:3478",
      username: "username",
      credential: "password"
    }
  ]
}

You can use the coturn TURN server to set up your own TURN server. Your implementation may require additional configuration to work with your network.

If you modify the JavaScript file, you must rebuild the container image. To do so, run:

docker compose build multimodal-data-visualization-streaming

MDVM Performance

MDVM is on its own lightweight, but does require some processing power to decode the video stream in the browser. Make sure that your browser has hardware acceleration enabled for the best performance

Running EVAM with GPU

You may be able to get better inference performance using the integrated GPU available on your Intel processor, or on a compatible discrete GPU or VPU. Because the core of EVAM is powered by the OpenVINO Toolkit, you can use OpenVINO’s instruction on GPU configuration for guidance on setting up your host. To allow the EVAM container access to your rendering device, you must provide access to the rendering group to the container user: instructions can be found here, under “Configure a Host Non-Root User to Use a GPU Device from an OpenVINO Container on Ubuntu 20 Host”. Once you’ve set up the proper OpenCL drivers and configured user access, you must modify the docker-compose.yml file to mount your rendering device to the container. For example, under the edge_video_analytics_microservice service definition, add:

devices:
  - "/dev/dri:/dev/dri"

If your render group id is different, modify the group_add section of the service as well.

MDVM EII Mode (AI Creek)

MDVM is a compatible component in EII. To run the Multimodal Data Visualization microservice in EII mode, refer to the README_EII.

MDVM Architecture and Implementation

MDVM is a containerized microservice that runs in a Docker container. It is a Grafana dashboard that is used to visualize data from the Edge Video Analytics Microservice. MDVM consists of 3 container images:

  • multimodal-data-visualization: A containerized Grafana that displays the data collected from the Edge Video Analytics Microservice.

  • multimodal-data-visualization-streaming: A containerized webapp that enables video capture from EVAM over WebRTC.

  • webrtc-signaling-server: A STUN server that is used to communicate between the webapp and EVAM and enable negotation and auto-stream capture.

Grafana is the main visualization component, providing a common interface for users to view and interact the captured data. One of the components in our provided dashboard points to the webapp provided by multimodal-data-visualization-streaming

The webapp provided by multimodal-data-visalization-streaming provides a dynamically updating collection of video streams captured from EVAM. The webapp utilized WebRTC to provide a real-time stream of video data from EVAM.

The dashboard also provides live and historical framerate data queried from EVAM’s REST API.

The provided STUN server, based on the DLStreamer pipeline server’s WebRTC example implementation, is used to enable the webapp to communicate with EVAM and provide pipeline data to the webapp.

MDVM for Ledge Park S: Bounding Box Rendering

When ingesting data over EII Message Bus, Visualization is able to handle bounding box rendering for object detection tasks. To configure this, you must modify config.json([WORK_DIR]/IEdgeInsights/Visualizer/multimodal-data-visualization-streaming/eii/config.json) by adding in the specific keys applicable to your usecase.

** For use with GETi metadata, config_geti.json([WORK_DIR]/IEdgeInsights/Visualizer/multimodal-data-visualization-streaming/eii/config.json) has been provided. Replace/rename config.json with this file, then rebuild the EII configuration files.

All availble modifiable keys for bounding box rendering are under the object_detection key. Details are below: |Key|Default|Description| |---|—|---| |detection_name|defects|Defines an array in the metadata message that holds the detections in the matching image frame| |nested_detections|off|By default, MDVM looks for a top level array in the metadata message that holds each detection in the image. If, however, the detection array is nested, this option should be set to "on", in conjunction with a "parent_detection_name", such as ``annotation``\ , which tell MDVM which object to look under for the detection array. Currently, this only supports one level below top.| |parent_detection_name| |The name of the JSON object holding the nested detection array. By default this can be left blank, but should be filled in ``nested_detections`` is set to on. This should be set to the same key as the object holding the detection array, such as ``annotation``\ | |bb_type|tlbr|Defines the type of bounding box information provided in the metadata message. Currently supports ``tlbr``\ , for dual-array bounding box information, and ``geti_bbox`` for depth-4 arrays holding bottom-left and top-right coordinates| |bb_corner_type|["tl","br"]|Helps MDVM define the type of corners, whether top-left and bottom-right, or reverse. If using a 4-depth array bounding box, like with GETi metadata, use ``["bbox",""]``. Currently supports ``["tl","br"]`` and ``["bbox",""]``\ | |label_type| type | Defines the labeling for detections. In the case of the default use-case, type is used to match up with the type key in the JSON metadata. For GETi, label should be used| |label_in_metadata|off|Defines whether the label is present in the actual metadata message. Otherwise, the label is read from predefined values in this configuration file.| |confidence|off|If confidence values are present in the metadata, they can be enabled by setting this to on. Should be used in conjunction with ``conf_name``. MDVM will round confidence numbers to two decimal digits|