Contents ======== * `Contents <#contents>`__ * `Overview of RestDataExport Service <#restdataexport-service>`__ * `Steps to Independently Build and Deploy RestDataExport Service <#steps-to-independently-build-and-deploy-restdataexport-service>`__ * `Steps to Independently Build RestDataExport Service <#steps-to-independently-build-restdataexport-service>`__ * `Steps to Independently Deploy RestDataExport Service <#steps-to-independently-deploy-restdataexport-service>`__ * `Deploy RestDataExport Service without Config Manager Agent Dependency <#deploy-restdataexport-service-without-config-manager-agent-dependency>`__ * `Deploy RestDataExport Service with Config Manager Agent Dependency <#deploy-restdataexport-service-with-config-manager-agent-dependency>`__ * `Configuration <#configuration>`__ * `HTTP GET API of RDE <#http-get-api-of-rde>`__ * `Get the Classifier Results Metadata <#get-the-classifier-results-metadata>`__ * `Request to get metadata <#request-to-get-metadata>`__ * `Get Images Using the Image Handle <#get-images-using-the-image-handle>`__ * `Request to Get Images <#request-to-get-images>`__ * `Prerequisites for Running RestDataExport to POST on HTTP Servers <#prerequisites-for-running-restdataexport-to-post-on-http-servers>`__ * `API Documentation <#api-documentation>`__ RestDataExport Service ---------------------- RestDataExport (RDE) service is a data service that serves GET and POST APIs. By default, the RDE service subscribes to a topic from the message bus and serves as GET API Server to respond to any GET requests for the required metadata and frames. By enabling the POST API, the RDE service publishs the subscribed metadata to an external HTTP server. .. **Important:** RestDataExport service subscribes to classified results from the Video Analytics (video) or the InfluxDB Connector (Time Series) use cases. In the subscriber configuration of the config.json(\ ``[WORK_DIR]/IEdgeInsights/RestDataExport/config.json``\ ) file, specify the required service to subscribe. Steps to Independently Build and Deploy RestDataExport Service ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. note:: For running 2 or more microservices, we recommend users to try the use case-driven approach for building and deploying as mentioned at `Generate Consolidated Files for a Subset of Edge Insights for Industrial Services `_. Steps to Independently Build RestDataExport Service ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: When switching between independent deployment of the service with and without config manager agent service dependency, one would run into issues with the ``docker-compose build`` w.r.t Certificates folder existence. As a workaround, please run the command ``sudo rm -rf Certificates`` to proceed with ``docker-compose build``. To independently build RestDataExport service, complete the following steps: #. The downloaded source code should have a directory named RestDataExport: .. code-block:: sh cd IEdgeInsights/RestDataExport #. Copy the IEdgeInsights/build/.env file using the following command in the current folder .. code-block:: sh cp ../build/.env . .. **NOTE**\ : Update the HOST_IP and ETCD_HOST variables in the .env file with your system IP. .. code-block:: sh # Source the .env using the following command: set -a && source .env && set +a #. Independently build .. code-block:: sh docker-compose build Steps to Independently Deploy RestDataExport Service ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can deploy the RestDataExport service in any of the following two ways: Deploy RestDataExport Service without Config Manager Agent Dependency """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" Run the following commands to deploy RestDataExport service without Config Manager Agent dependency: ======= .. code-block:: sh # Enter the RestDataExport directory cd IEdgeInsights/RestDataExport .. Copy the IEdgeInsights/build/.env file using the following command in the current folder, if not already present. .. code-block:: sh cp ../build/.env . **Note:** Ensure that ``docker ps`` is clean and ``docker network ls`` doesn't have EII bridge network. .. code-block:: Update .env file for following: 1. HOST_IP and ETCD_HOST variables with your system IP. 2. `READ_CONFIG_FROM_FILE_ENV` value to `true` and `DEV_MODE` value to `true`. Source the .env using the following command: set -a && source .env && set +a .. code-block:: sh # Run the service docker-compose -f docker-compose.yml -f docker-compose-dev.override.yml up -d .. note:: RestDataExport container restarts automatically when its config is modified in ``config.json`` file. If user is updating the config.json file using ``vi or vim`` editor, it is required to append the ``set backupcopy=yes`` in ``~/.vimrc`` so that the changes done on the host machine config.json gets reflected inside the container mount point. Deploy RestDataExport Service with Config Manager Agent Dependency """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" Run the following commands to deploy RestDataExport service with Config Manager Agent dependency: .. note:: Ensure that the Config Manager Agent image present in the system. If not, build the Config Manager Agent locally when independently deploying the service with Config Manager Agent dependency. .. code-block:: sh # Enter the RestDataExport cd IEdgeInsights/RestDataExport Copy the IEdgeInsights/build/.env file using the following command in the current folder, if not already present. .. code-block:: sh cp ../build/.env . .. note:: Ensure that ``docker ps`` is clean and ``docker network ls`` does not have EII bridge network. .. code-block:: Update .env file for following: 1. HOST_IP and ETCD_HOST variables with your system IP. 2. `READ_CONFIG_FROM_FILE_ENV` value is set to `false`. .. Copy the docker-compose.yml from IEdgeInsights/ConfigMgrAgent as docker-compose.override.yml in IEdgeInsights/RestDataExport. .. code-block:: sh cp ../ConfigMgrAgent/docker-compose.yml docker-compose.override.yml .. Copy the builder.py with standalone mode changes from IEdgeInsights/build directory .. code-block:: sh cp ../build/builder.py . Run the builder.py in standalone mode, this will generate eii_config.json and update docker-compose.override.yml .. code-block:: sh python3 builder.py -s true Building the service (This step is optional for building the service if not already done in the ``Independently buildable`` step above) .. code-block:: sh docker-compose build For running the service in PROD mode, run the below command: **NOTE**\ : Make sure to update ``DEV_MODE`` to ``false`` in .env while running in PROD mode and source the .env using the command ``set -a && source .env && set +a``. .. code-block:: sh docker-compose up -d For running the service in DEV mode, run the below command: **NOTE**\ : Make sure to update ``DEV_MODE`` to ``true`` in .env while running in DEV mode and source the .env using the command ``set -a && source .env && set +a``. .. code-block:: sh docker-compose -f docker-compose.yml -f docker-compose-dev.override.yml -f docker-compose.override.yml up -d Configuration ^^^^^^^^^^^^^ For more details on the ETCD secrets and message bus endpoint configuration, refer to the following: * `Etcd_Secrets_Configuration.md `_ * `MessageBus Configuration `_ HTTP GET API of RDE ^^^^^^^^^^^^^^^^^^^ The HTTP GET API of RDE allows you to get metadata and images. The following sections provide information about how to request metadata and images using the curl commands. Get the Classifier Results Metadata ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To get the classifier results metadata, refer to the following: Request to GET Metadata """"""""""""""""""""""" You can get the metadata for DEV mode and PROD mode. * For the DEV mode: ``GET /metadata`` Run the following command: .. code-block:: sh curl -i -H 'Accept: application/json' http://:8087/metadata Refer to the following example: .. code-block:: sh curl -i -H 'Accept: application/json' http://localhost:8087/metadata * For the PROD mode: ``GET /metadata`` Run the following command: .. code-block:: sh curl --cacert ../build/Certificates/rootca/cacert.pem -i -H 'Accept: application/json' https://:8087/metadata Refer to the following example: .. code-block:: sh curl --cacert ../build/Certificates/rootca/cacert.pem -i -H 'Accept: application/json' https://localhost:8087/metadata Output: The output for the previous command is as follows: .. code-block:: sh HTTP/1.1 200 OK Content-Type: text/json Date: Fri, 08 Oct 2021 07:51:07 GMT Content-Length: 175 {"channels":3,"defects":[],"encoding_level":95,"encoding_type":"jpeg","frame_number":558,"height":1200,"img_handle":"21af429f85","topic":"camera1_stream_results","width":1920} Get Images Using the Image Handle ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: For the ``image`` API, the ``datastore`` module is mandatory. From the ``datastore``\ , the server fetches the data, and returns it over the REST API. Include the ``datastore`` module as a part of your use case. Request to GET Images """"""""""""""""""""" ``GET /image`` Run the following command: .. code-block:: sh curl -i -H 'Accept: image/jpeg' http://:8087/image?img_handle= Refer to the following example to store image to the disk using curl along with ``img_handle``\ : .. code-block:: sh curl -i -H 'Accept: application/image' http://localhost:8087/image?img_handle=21af429f85 > img.jpeg % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 324k 0 324k 0 0 63.3M 0 --:--:-- --:--:-- --:--:-- 63.3M .. note:: You can find the ``imageid`` of the image in the ``metadata`` API response. Prerequisites to Run RestDataExport to POST on HTTP Servers ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: By default, RDE will serve the metadata as ``GET`` *only* request server. By Enabling this, you can get the metadata using the ``GET`` request. Also, RDE will post the metadata to an HTTP server. As a prerequisites, complete the following steps: #. Update ``[WORKDIR]/IEdgeInsights/build/.env`` file ``HTTP_METHOD_FETCH_METADATA`` environment value as follows. .. code-block:: sh HTTP_METHOD_FETCH_METADATA="POST" .. **Note:** Ensure to rerun builder.py post changes, for generating updated deployment yml files. #. If you are using the ``HttpTestServer`` then ensure that the server's IP address is added to the ``no_proxy/NO_PROXY`` vars in: * /etc/environment (Needs restart/relogin) * ./docker-compose.yml (Needs to re-run the 'builder' step) .. code-block:: sh environment: AppName: "RestDataExport" DEV_MODE: ${DEV_MODE} no_proxy: ${ETCD_HOST}, #. Run the following command to install ``python etcd3`` .. code-block:: sh pip3 install -r requirements.txt #. Ensure the topics you subscribe to are also added in the config(\ ``[WORK_DIR]/IEdgeInsights/RestDataExport/config.json``\ ) with the ``HttpServer endpoint`` specified #. Update the ``config.json`` file as follows: .. code-block:: json { "camera1_stream_results": "http://IP Address of Test Server:8082", "point_classifier_results": "http://IP Address of Test Server:8082", "http_server_ca": "/opt/intel/eii/cert.pem", "rest_export_server_host": "0.0.0.0", "rest_export_server_port": "8087" } #. Build and provision EII. .. code-block:: sh # Build all containers docker-compose build # Provision the EII stack by bringing up ConfigMgrAgent docker-compose up -d ia_configmgr_agent #. Ensure the prerequisites for starting the TestServer application are enabled. For more information, refer to the `README.md `_. #. As a prerequisite, before starting RestDataExport service, run the following commands. .. **Note:** RestDataExport is pre-equipped with a python tool(\ ``[WORK_DIR]/IEdgeInsights/RestDataExport/etcd_update.py``\ ) to insert data into etcd, which can be used to insert the required ``HttpServer ca cert`` in the config of RestDataExport before running it. .. code-block:: sh set -a && \ source ../build/.env && \ set +a # Required if running in the PROD mode only sudo chmod -R 777 ../build/Certificates/ python3 etcd_update.py --http_cert --ca_cert --cert --key --hostname --port Example: # Required if running in the PROD mode python3 etcd_update.py --http_cert "../tools/HttpTestServer/certificates/ca_cert.pem" --ca_cert "../build/Certificates/rootca/cacert.pem" --cert "../build/Certificates/root/root_client_certificate.pem" --key "../build/Certificates/root/root_client_key.pem" --hostname --port # Required if running with k8s helm in the PROD mode python3 etcd_update.py --http_cert "../tools/HttpTestServer/certificates/ca_cert.pem" --ca_cert "../build/helm-eii/eii-deploy/Certificates/rootca/cacert.pem" --cert "../build/helm-eii/eii-deploy/Certificates/root/root_client_certificate.pem" --key "../build/helm-eii/eii-deploy/Certificates/root/root_client_key.pem" --hostname --port 32379 #. Start the TestServer application. For more information, refer to the `README.md `_. #. Ensure that the ``DataStore`` application is running. For more information refer to the `README.md `_. Launch RestDataExport Service ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To build and launch the RestDataExport service, refer to the following: * `../README.md#generate-deployment-and-configuration-files `_ * `../README.md#provision `_ * `../README.md#build-and-run-eii-videotimeseries-use-cases `_ Configure Environment Proxy Settings ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To configure the environment proxy settings for RDE, refer to the following: #. To update the host-ip for http, run the following command: .. code-block:: sh sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf #. To update the host-ip for https, run the following command: .. code-block:: sh sudo vi /etc/systemd/system/docker.service.d/https-proxy.conf (update host-ip) #. To check if the proxy settings have been applied, run the following command: .. code-block:: sh env | grep proxy #. To update the ``no_proxy env`` variable, run the following command: .. code-block:: sh export no_proxy=$no_proxy, #. To update docker proxy settings, run the following command: .. code-block:: sh sudo vi ~/.docker/config.json (update host-ip in no_proxy) #. To reload the docker daemon, run the following command: .. code-block:: sh sudo systemctl daemon-reload #. To restart the docker service with the updated proxy configurations, run the following command: .. code-block:: sh sudo systemctl restart docker API Documentation ^^^^^^^^^^^^^^^^^ The RestDataExport generates the Open API documentation for all the REST APIs it exposes. This documentation can be accessed at its or documents endpoint. .. code-block:: sh http://< host ip >:8087/docs