Data Store Microservice ~~~~~~~~~~~~~~~~~~~~~~~ Data Store microservice supports Video and Time series use cases. The Data Store microservice supports two type of Data: * JSON Datatype (InfluxDB) * Blob Datatype (MinIO Object Storage) DataStore Configuration ^^^^^^^^^^^^^^^^^^^^^^^ The configurations for ``Data Store`` service is added in etcd. The configuration details are available in the docker-compose file, under ``AppName`` in the environment section of the app’s service definition. For the scenario, when the ``AppName`` is ``DataStore``, the following example shows how the app’s config will look for ``/DataStore/config`` key in etcd: .. code:: json "datatypes": { "json": { "host" : "ia_influxdb", "port": 8086, "dbname": "datain", "verifySsl": false, "ignoreKeys": [ "defects" ], "tagKeys": [], "retention": "1h", "topics": [ "*" ], "retentionPollInterval": "60s" }, "blob": { "host" : "ia_miniodb", "port": 9000, "dbname": "image-store-bucket", "retention": "1h", "topics": [ "edge_video_analytics_results" ], "retentionPollInterval": "60s", "ignoreKeys": [ "model-registry" ] } } The following are the details of the keys in the above config: - * datatype (required) - The ``host`` is optional parameter in configuration, which is used for connecting the respective Database servers (Local/Remote). If the parameter is not provided, by default JSON Datatype will be selected with ``ia_influxdb`` and Blob Datatype will be selected with ``ia_miniodb`` - The ``port`` is optional parameter in configuration, which is used for connecting the respective Database servers port(Local/Remote). If the parameter is not provided, by default JSON Datatype will be selected with ``8086`` for ``Influx DB`` and Blob Datatype will be selected with ``9000`` for ``Minio Object Storage`` - The ``topics`` key determines which messages are to be processed by the corresponding DB microservice. Only the messages with a topic listed in ``topics`` key are processed by the individual module. If ``topics`` contain ``\*``, then all the messages are processed. - The ``retention`` is required parameter in configuration. The retention parameter specifies the retention policy to apply for the images stored in MinIO object storage. In case of infinite retention time, set it to ““. Suitable duration string value as mentioned at https://golang.org/pkg/time/#ParseDuration. - The ``retentionPollInterval`` is required parameter in configuration. Used to set the time interval for checking images for expiration. Expired images will become candidates for deletion and no longer retained. In case of infinite retention time, this attribute will be ignored. Suitable duration string value as mentioned at https://golang.org/pkg/time/#ParseDuration - The ``ignoreKeys`` is list of string which allows to add keys. For JSON, this list will be used parser to choose whether to parse the nested json or save as string to database. For BLOB, this list will be used for not applying the retention policy for the respective buckets. * restserver - The ``Port`` key is used to configure the port exposed to access the REST APIs. - **Note**\ : Same should be updated under Port section in* docker-compose.yml - The ``cacertname`` key is filename of pem certificate - The ``caprivatekey`` key is filename of pem key - To use different certificates for ``restserver`` change the LHS of the volume mount path in docker-compose.yml for line item ``./Certificates/DataStore_Server/:/run/secrets/RestAPI_Server:ro``, update the ``cacertname`` and ``caprivatekey`` with respective file names in config.json - Refer `Tutorial 2` in :doc:`REST API Endpoints `. ``Swagger(API Documentation)`` is available at ``/docs``. To access swagger, after starting data store open browser and enter url ``https://:8889``. For ``DEV_MODE`` as ``true``, use ``http://:8889`` - **Note**\ : Please note due to some cert issue, the Data Store swagger documentation at https://localhost:8889 in PROD mode OR http://localhost:8889 in DEV mode will not be accessible. By default, both the DBs will be enabled. If you want to disable any of the above DBs, remove the corresponding key and its value from the config. For Example, if you are not using MinIO object storage, you can disable the same and modify the config as below: .. code:: json "datatypes": { "json": { "host" : "ia_influxdb", "port": 8086, "dbname": "datain", "verifySsl": false, "ignoreKeys": [ "defects" ], "tagKeys": [], "retention": "1h", "topics": [ "*" ], "retentionPollInterval": "60s" } } .. For Backuping up the Json and Blob data following configuration is provided in config.json .. code:: json "storage": { "enabled": false, "type": "local", "autoSyncIntervalInSec": 3600, "blobIdentifier": "img_handle" } .. **NOTE**: Enabling storage feature with type ``local`` as above will backup both json and blob data which would lead to high disk space consumption on the host system and the consumption will be more in case of multi streams.For example 1hr of data for a single stream with image resolution as 1920 \* 1200 consumes 3.1Gb of disk space. The following are the details of the keys in the above config: - The ``enabled`` is by default false but if the backup of data is needed then set the ``enabled`` flag as true. - The ``type`` denotes the type of storage needed i.e. either local or cloud. - The ``autoSyncIntervalInSec`` denotes the time interval in seconds indicating how often the database had to be queried to backup the data. - The ``blobIdentifier`` denotes the identifier used for BLOB data. For example below is a Edge Video Analytics Microservice’s data snipet where the blobIdentifier will be ``img_handle`` :: {'height': 1200, 'width': 1920, 'channels': 3, 'img_handle': 'cd3a30e5c3', 'defects': [], 'encoding_type': 'jpeg', 'encoding_level': 95} .. **NOTE** - The backed up data is stored at /opt/intel/eii/data/ds_backup/ folder which consist of two subfolder one for json data and one for blob data. - The json data is stored in the json format with filename is made up of json i.e edge_video_analytics_results_2024-01-23-14_44_58.json - For blob data the format is taken from ‘encoding_type’ field of Edge Video Analytics Microservice’s data with file name is made up of and blob data is stored in the folder blob/ i.e blob/2024-02-01-16:25:32/edge_video_analytics_results_ff875f1df.jpeg JSON Datatype (InfluxDB) ^^^^^^^^^^^^^^^^^^^^^^^^ For nested json data, by default, Data Store will flatten the nested json and push the flat data to InfluxDB to avoid the flattening of any particular nested key mention the tag key in the config.json file. Currently the ``defects`` key is ignored from flattening. Every key to be ignored has to be in a new line. For example, .. code:: sh ignore_keys = [ "Key1", "Key2", "Key3" ] By default, all the keys in the data schema will be pushed to InfluxDB as fields. If tags are present in data schema, it can be mentioned in the `config.json <./config.json>`__ file then the data pushed to InfluxDB, will have fields and tags both. At present, no tags are visible in the data scheme and tag_keys are kept blank in the config file. For Example, .. code:: sh tag_keys = [ "Tag1", "Tag2" ] Blob Datatype (MinIO Object Storage) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The MinIO object storage primarily subscribes to the stream that comes out of the EdgeVideoAnalayticsMicroservice app via Intel® Edge Insights System messagebus and stores the frame into minio for historical analysis. The high-level logical flow of MinIO object storage is as follows: 1. The Intel® Edge Insights System messagebus subscriber in MinIO object storage will subscribe to the EdgeVideoAnalayticsMicroservice published classified result (metadata, frame) on the Intel® Edge Insights System messagebus. The img_handle is extracted out of the metadata and is used as the key and the frame is stored as a value for that key in minio persistent storage. Intel® Edge Insights System Msgbus Interface ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Data Store will start the Intel® Edge Insights System messagebus Publisher, Intel® Edge Insights System messagebus Subscriber threads, and Intel® Edge Insights System messagebus request reply thread based on PubTopics, SubTopics and Server configuration. - Intel® Edge Insights System messagebus Subscriber thread connects to the PUB socket of Intel® Edge Insights System messagebus on which the data is published by EdgeVideoAnalayticsMicroservice and push it to the InfluxDB(Metadata). - Intel® Edge Insights System messagebus Publisher thread will publish the point data ingested by the telegraf and the classifier result coming out of the point data analytics. Intel® Edge Insights System Msgbus/REST Request-Response Interface ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ For a historical analysis of the stored classified images or metadata, Data Store starts a Intel® Edge Insights System Messagebus/gRPC/REST Request-Response Interface server which provides the read, write, update, list, clear, delete interfaces. The payload format for Intel® Edge Insights System Msgbus/gRPC Request-Response Endpoints is available as part of `Tutorial 1` in :doc:`Intel® Edge Insights System Msgbus/gRPC Request-Response Endpoints `. The payload format for Intel® Edge Insights System REST API Endpoints is available as part of `Tutorial 2` in :doc:`REST API Endpoints `. **Note:** 1. The gRPC request-response interface server currently supports **DEV mode** only. DB Server Supported Version ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Currently, DB Handlers are supported and tested with the following versions for respective DB Servers: +-------+---------------+-------------------------+ | S.No | DB Server | Supported Version | +=======+===============+=========================+ | 1 | Influx | 1.8.7 | +-------+---------------+-------------------------+ | 2 | Minio | RELEASE | | | | 2020-12-12T08:39:07Z | +-------+---------------+-------------------------+