Contents

Multimodal Data Visualization Microservice Overview

The Multimodal Data Visualization microservice provides functionality to represent the data graphically. Using this service, you can visualize the video streaming and Time Series data. The following containers run as a part of the Multimodal Data Visualization microservice:

  • multimodal-data-visualization

  • multimodal-data-visualization-streaming

The multimodal-data-visualization-streaming container gets the ingested frames and the inference results from the MsgBus subscriber and it then renders the video to a webpage. This webpage is embedded in Grafana* to visualize the video stream and the Time Series data on the same dashboard.

This directory provides a Docker compose and config file to use the Multimodal Data Visualization microservice with the Edge Insights for Industrial software stack.

Prerequisites

As a prerequisite for the Multimodal Data Visualization microservice, complete the following steps:

  1. EII when downloaded from ESH would be available at the installed location

    cd [EII installed location]/IEdgeInsights

  2. Complete the prerequisite for provisioning the EII stack. For more information, refer to the README.md.

  3. Run the following commands to set the environment and build the ia_configmgr_agent container:

    cd [WORK_DIR]/IEdgeInsights/build

# Execute the builder.py script
python3 builder.py -f usecases/video-streaming.yml

Run the Containers

To pull the prebuilt EII container images and Multimodal Data Visualization microservice images from Docker Hub and run the containers in the detached mode, run the following command:

# Start the docker containers
docker compose up -d

Note

The prebuilt container image for the Multimodal Data Visualization microservice gets downloaded when you run the docker compose up -d command, if the image is not already present on the host system.

Interfaces Section

In the EII mode, the endpoint details for the EII service you need to subscribe from are to be provided in the Subscribers section in the config([WORK_DIR]/IEdgeInsights/Visualizer/config.json) file. For more details on the structure, refer to the EII documentation.

Grafana Overview

Grafana supports various storage backends for the Time Series data (data source). EII uses InfluxDB as the data source. Grafana connects to the InfluxDB data source that is preconfigured as a part of the Grafana setup. The ia_influxdbconnector and ia_webservice service must be running for Grafana to be able to collect the Time Series data and stream the video respectively. After the data source starts working, you can use the preconfigured dashboard to visualize the incoming data. You can also edit the dashboard as required.

After the Multimodal Data Visualization microservice is up, you can access Grafana at http://:3000

Grafana Configuration

The following are the configuration details for Grafana:

  • dashboard.json([WORK_DIR]/IEdgeInsights/Visualizer/multimodal-data-visualization/eii/dashboard.json): This is the dashboard json file that is loaded when Grafana starts. It is preconfigured to display the Time Series data.

  • dashboards.yml([WORK_DIR]/IEdgeInsights/Visualizer/multimodal-data-visualization/eii/dashboards.yml): This is the config file for all the dashboards. It specifies the path to locate all the dashboard json files.

  • datasources.yml([WORK_DIR]/IEdgeInsights/Visualizer/multimodal-data-visualization/eii/datasources.yml): This is the config file for setting up the data source. It has various fields for data source configuration.

  • grafana.ini([WORK_DIR]/IEdgeInsights/Visualizer/multimodal-data-visualization/eii/grafana.ini): This is the config file for Grafana. It specifies how Grafana should start after it is configured.

Note

You can edit the contents of these files based on your requirement.

Run Grafana in the PROD Mode

Note

Skip this section, if you are running Grafana in the DEV mode.

To run Grafana in the PROD mode, import cacert.pem from the build/Certificates/rootca/ directory to the browser certificates. Complete the following steps to import certificates:

  1. In Chrome browser, go to Settings.

  2. In Search settings, enter Manage certificates.

  3. In Privacy and security, click Security.

  4. On the Advanced section, click Manage certificates.

  5. On the Certificates window, click the Trusted Root Certification Authorities tab.

  6. Click Import.

  7. On the Certificate Import Wizard, click Next.

  8. Click Browse.

  9. Go to the IEdgeInsights/build/Certificates/rootca/ directory.

  10. Select the cacert.pem file, and provide the necessary permissions, if required.

  11. Select all checkboxes and then, click Import.

Run Grafana for a Video Use Case

Complete the following steps to run Grafana for a video use case:

  1. Ensure that the endpoint of the publisher, that you want to subscribe to, is mentioned in the Subscribers section of the config([WORK_DIR]/IEdgeInsights/Visualizer/config.json) file.

  2. Use root as the Username and eii123 Password both for first login, password can be changed if required when prompted after logging in.

  3. On the Home Dashboard page, on the left corner, click the Dashboards icon.

  4. Click the Manage Dashboards tab, to view the list of all the preconfigured dashboards.

  5. Select EII Video and Time Series Dashboard, to view multiple panels with topic names of the subscriber as the panel names along with a time-series panel named Time Series.

  6. Hover over the topic name. The panel title will display multiple options.

  7. Click View to view the subscribed frames for each topic.

Note

  • Changing gridPos for the video frame panels is prohibited since these values are altered internally to support multi-instance.

  • Grafana does not support visualization for GVA and CustomUDF streams.

Metadata Configuration

MDVM’s backend, multimodal-data-visualization-streaming, is like all EII componenets controlled by config.json in EII mode. This configuartion file includes a schema that the backend uses to read metadata messages and provide overlays in the visualization stream. Your configuration may vary based on your use case.

Modifying EII Mode with Various Configs

MDVM provides various configs depending on your usecase, available in <root>/multimodal-data-visualization-streaming/eii/configs

Available Configuration Files

  • default_anomlib_config.json: anomalib default detection (default config)

    • This config is the application default. It provides a classification visualization of an anomaly detection pipeline from EVAM

  • geti_udf_config.json: generic GeTI UDF configuration

    • This config

  • old_person_detection_config.json: (deprecated) person detection pipeline configuration

    • This config was used for the original person detection pipeline. It is now deperecated, but provided for posterity

  • person_detection_udf_config.json: person detection udf pipeline configuration

    • This config provides compatiblity with the Geti person detection UDF.

  • weld_udf_config.json: weld detection UDF configuration

    • This config provides compatiblility with the Geti weld detection UDF.

To use one of these premade configs, you can copy the config file to <root>/multimodal-data-visualization-streaming/eii, overwriting the existing config.json.

cp ./multimodal-data-visualization-streaming/eii/configs/person_detection_udf_config.json ./multimodal-data-visualization-streaming/eii/config.json

**Note**: This will overwrite any changes you have made to your existing config.json

Enabling New Configuration File

To enable the new configuration file, the EII config builder will need to be re-run.

cd ../build
python ./builder.py -f usecases/video-streaming-evam.yml
docker compose down -v && docker compose build && docker compose up

Example Metadata for Object Detection

These are generic metadata JSON objects, to provide an example of the schema used by Visualizer.

Metadata for Object Detection (and GeTI UDFs)

"evam_metadata": {
    "task": "object_detection",
    "detection_name":"defects",
    "nested_detections":"off",
    "parent_detection_name":"",
    "bb_type":"tlbr",
    "bb_corner_type":["tl","br"],
    "label_type": "type",
    "label_location":"config",
    "confidence":"off",
    "conf_name":""
},

Metadata for Classification

"evam_metadata": {
    "task": "classification",
    "class_list":["true"],
    "multi_class": "false",
    "label_display":"all"
},