Contents

RestDataExport Service

RestDataExport (RDE) service is a data service that serves GET and POST APIs. By default, the RDE service subscribes to a topic from the message bus and serves as GET API Server to respond to any GET requests for the required metadata and frames. By enabling the POST API, the RDE service publishs the subscribed metadata to an external HTTP server.

Important:

RestDataExport service subscribes to classified results from the Video Analytics (video) or the InfluxDB Connector (Time Series) use cases. In the subscriber configuration of the config.json([WORK_DIR]/IEdgeInsights/RestDataExport/config.json) file, specify the required service to subscribe.

Steps to Independently Build and Deploy RestDataExport Service

Note

For running 2 or more microservices, we recommend users to try the use case-driven approach for building and deploying as mentioned at Generate Consolidated Files for a Subset of Edge Insights for Industrial Services.

Steps to Independently Build RestDataExport Service

Note

When switching between independent deployment of the service with and without config manager agent service dependency, one would run into issues with the docker-compose build w.r.t Certificates folder existence. As a workaround, please run the command sudo rm -rf Certificates to proceed with docker-compose build.

To independently build RestDataExport service, complete the following steps:

  1. The downloaded source code should have a directory named RestDataExport:

    cd IEdgeInsights/RestDataExport

  2. Copy the IEdgeInsights/build/.env file using the following command in the current folder

    cp ../build/.env .

    NOTE: Update the HOST_IP and ETCD_HOST variables in the .env file with your system IP.

    # Source the .env using the following command:
set -a && source .env && set +a

  3. Independently build

    docker-compose build

Steps to Independently Deploy RestDataExport Service

You can deploy the RestDataExport service in any of the following two ways:

Deploy RestDataExport Service without Config Manager Agent Dependency

Run the following commands to deploy RestDataExport service without Config Manager Agent dependency:

# Enter the RestDataExport directory
cd IEdgeInsights/RestDataExport

Copy the IEdgeInsights/build/.env file using the following command in the current folder, if not already present.

cp ../build/.env .

Note: Ensure that docker ps is clean and docker network ls doesn’t have EII bridge network.

Update .env file for following:
1. HOST_IP and ETCD_HOST variables with your system IP.
2. `READ_CONFIG_FROM_FILE_ENV` value to `true` and `DEV_MODE` value to `true`.

Source the .env using the following command:
set -a && source .env && set +a

# Run the service
docker-compose -f docker-compose.yml -f docker-compose-dev.override.yml up -d

Note

RestDataExport container restarts automatically when its config is modified in config.json file. If user is updating the config.json file using vi or vim editor, it is required to append the set backupcopy=yes in ~/.vimrc so that the changes done on the host machine config.json gets reflected inside the container mount point.

Deploy RestDataExport Service with Config Manager Agent Dependency

Run the following commands to deploy RestDataExport service with Config Manager Agent dependency:

Note

Ensure that the Config Manager Agent image present in the system. If not, build the Config Manager Agent locally when independently deploying the service with Config Manager Agent dependency.

# Enter the RestDataExport
cd IEdgeInsights/RestDataExport

Copy the IEdgeInsights/build/.env file using the following command in the current folder, if not already present.

cp ../build/.env .

Note

Ensure that docker ps is clean and docker network ls does not have EII bridge network.

Update .env file for following:
1. HOST_IP and ETCD_HOST variables with your system IP.
2. `READ_CONFIG_FROM_FILE_ENV` value is set to `false`.

Copy the docker-compose.yml from IEdgeInsights/ConfigMgrAgent as docker-compose.override.yml in IEdgeInsights/RestDataExport.

cp ../ConfigMgrAgent/docker-compose.yml docker-compose.override.yml

Copy the builder.py with standalone mode changes from IEdgeInsights/build directory

cp ../build/builder.py .

Run the builder.py in standalone mode, this will generate eii_config.json and update docker-compose.override.yml

python3 builder.py -s true

Building the service (This step is optional for building the service if not already done in the Independently buildable step above)

docker-compose build

For running the service in PROD mode, run the below command:

NOTE: Make sure to update DEV_MODE to false in .env while running in PROD mode and source the .env using the command set -a && source .env && set +a.

docker-compose up -d

For running the service in DEV mode, run the below command:

NOTE: Make sure to update DEV_MODE to true in .env while running in DEV mode and source the .env using the command set -a && source .env && set +a.

docker-compose -f docker-compose.yml -f docker-compose-dev.override.yml -f docker-compose.override.yml up -d

Configuration

For more details on the ETCD secrets and message bus endpoint configuration, refer to the following:

HTTP GET API of RDE

The HTTP GET API of RDE allows you to get metadata and images. The following sections provide information about how to request metadata and images using the curl commands.

Get the Classifier Results Metadata

To get the classifier results metadata, refer to the following:

Request to GET Metadata

You can get the metadata for DEV mode and PROD mode.

  • For the DEV mode: GET /metadata

Run the following command:

curl -i -H 'Accept: application/json' http://<machine_ip_address>:8087/metadata

Refer to the following example:

curl -i -H 'Accept: application/json' http://localhost:8087/metadata

  • For the PROD mode: GET /metadata

Run the following command:

curl --cacert ../build/Certificates/rootca/cacert.pem -i -H 'Accept: application/json' https://<machine_ip_address>:8087/metadata

Refer to the following example:

curl --cacert ../build/Certificates/rootca/cacert.pem -i -H 'Accept: application/json' https://localhost:8087/metadata

Output:

The output for the previous command is as follows:

HTTP/1.1 200 OK
Content-Type: text/json
Date: Fri, 08 Oct 2021 07:51:07 GMT
Content-Length: 175
{"channels":3,"defects":[],"encoding_level":95,"encoding_type":"jpeg","frame_number":558,"height":1200,"img_handle":"21af429f85","topic":"camera1_stream_results","width":1920}

Get Images Using the Image Handle

Note

For the image API, the datastore module is mandatory. From the datastore, the server fetches the data, and returns it over the REST API. Include the datastore module as a part of your use case.

Request to GET Images

GET /image

Run the following command:

curl -i -H 'Accept: image/jpeg' http://<machine_ip_address>:8087/image?img_handle=<imageid>

Refer to the following example to store image to the disk using curl along with img_handle:

curl -i -H 'Accept: application/image' http://localhost:8087/image?img_handle=21af429f85 > img.jpeg

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
Dload  Upload   Total   Spent    Left  Speed
100  324k    0  324k    0     0  63.3M      0 --:--:-- --:--:-- --:--:-- 63.3M

Note

You can find the imageid of the image in the metadata API response.

Prerequisites to Run RestDataExport to POST on HTTP Servers

Note

By default, RDE will serve the metadata as GET only request server. By Enabling this, you can get the metadata using the GET request. Also, RDE will post the metadata to an HTTP server.

As a prerequisites, complete the following steps:

  1. Update [WORKDIR]/IEdgeInsights/build/.env file HTTP_METHOD_FETCH_METADATA environment value as follows.

    HTTP_METHOD_FETCH_METADATA="POST"

    Note: Ensure to rerun builder.py post changes, for generating updated deployment yml files.

  2. If you are using the HttpTestServer then ensure that the server’s IP address is added to the no_proxy/NO_PROXY vars in:

    • /etc/environment (Needs restart/relogin)

    • ./docker-compose.yml (Needs to re-run the ‘builder’ step)

    environment:
 AppName: "RestDataExport"
 DEV_MODE: ${DEV_MODE}
 no_proxy: ${ETCD_HOST}, <IP of HttpTestServer>

  3. Run the following command to install python etcd3

    pip3 install -r requirements.txt

  4. Ensure the topics you subscribe to are also added in the config([WORK_DIR]/IEdgeInsights/RestDataExport/config.json) with the HttpServer endpoint specified

  5. Update the config.json file as follows:

    {
 "camera1_stream_results": "http://IP Address of Test Server:8082",
 "point_classifier_results": "http://IP Address of Test Server:8082",
 "http_server_ca": "/opt/intel/eii/cert.pem",
 "rest_export_server_host": "0.0.0.0",
 "rest_export_server_port": "8087"
}

  6. Build and provision EII.

    # Build all containers
docker-compose build
# Provision the EII stack by bringing up ConfigMgrAgent
docker-compose up -d ia_configmgr_agent

  7. Ensure the prerequisites for starting the TestServer application are enabled. For more information, refer to the README.md.

  8. As a prerequisite, before starting RestDataExport service, run the following commands.

    Note: RestDataExport is pre-equipped with a python tool([WORK_DIR]/IEdgeInsights/RestDataExport/etcd_update.py) to insert data into etcd, which can be used to insert the required HttpServer ca cert in the config of RestDataExport before running it.

    set -a && \
source ../build/.env && \
set +a

# Required if running in the PROD mode only
sudo chmod -R 777 ../build/Certificates/

python3 etcd_update.py --http_cert <path to ca cert of HttpServer> --ca_cert <path to etcd   client ca cert> --cert <path to etcd client cert> --key <path to etcd client key> --hostname   <IP address of host system> --port <ETCD PORT>

Example:
# Required if running in the PROD mode
python3 etcd_update.py --http_cert "../tools/HttpTestServer/certificates/ca_cert.pem"   --ca_cert "../build/Certificates/rootca/cacert.pem" --cert "../build/Certificates/root/root_client_certificate.pem" --key "../build/Certificates/root/root_client_key.pem" --hostname <IP address of host system> --port <ETCD PORT>

# Required if running with k8s helm in the PROD mode
python3 etcd_update.py --http_cert "../tools/HttpTestServer/certificates/ca_cert.pem" --ca_cert "../build/helm-eii/eii-deploy/Certificates/rootca/cacert.pem" --cert "../build/helm-eii/eii-deploy/Certificates/root/root_client_certificate.pem" --key "../build/helm-eii/eii-deploy/Certificates/root/root_client_key.pem" --hostname <Master Node IP address of ETCD host system> --port 32379

  9. Start the TestServer application. For more information, refer to the README.md.

  10. Ensure that the DataStore application is running. For more information refer to the README.md.

Launch RestDataExport Service

To build and launch the RestDataExport service, refer to the following:

Configure Environment Proxy Settings

To configure the environment proxy settings for RDE, refer to the following:

  1. To update the host-ip for http, run the following command:

    sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf

  2. To update the host-ip for https, run the following command:

    sudo vi /etc/systemd/system/docker.service.d/https-proxy.conf (update host-ip)

  3. To check if the proxy settings have been applied, run the following command:

    env | grep proxy

  4. To update the no_proxy env variable, run the following command:

    export no_proxy=$no_proxy,<host-ip>

  5. To update docker proxy settings, run the following command:

    sudo vi ~/.docker/config.json (update host-ip in no_proxy)

  6. To reload the docker daemon, run the following command:

    sudo systemctl daemon-reload

  7. To restart the docker service with the updated proxy configurations, run the following command:

    sudo systemctl restart docker

API Documentation

The RestDataExport generates the Open API documentation for all the REST APIs it exposes. This documentation can be accessed at its or documents endpoint.

http://< host ip >:8087/docs