EII Configuration Management¶
etcd¶
EII uses etcd (https://etcd.io/) for configuration management. etcd acts as a distributed key-value store for distributed systems. EII applications will contact etcd on startup and read their configurations as set by the user using the provisioning step/etcd UI. Because etcd is distributed, applications can receive their configuration on any node in the etcd cluster, regardless of where they might be deployed.
ConfigMgr¶
ConfigMgr (Config manager) provides CPP/Python/Golang APIs to:
fetch an applications configs values from the KV store.
fetch an applications interface values from KV store for pub, sub, server and client.
watch on application’s config changes.
generate EII MessageBus config.
read and set /GlobalEnv variables.
fetch env variables: appname, dev_mode
All these data are stored into the kv store of EII during the provisioning phase and can be changed dynamically by the admin.
ETCD UI Service¶
Once EII Configuration Management (ia_etcd) service is successfully up, user can access the ETCD web UI through the steps below. This allows user to make configuration changes for respective EII container services.
Open your browser and enter the address: https://< host ip >:7071/etcdkeeper/ (when EII is running in secure mode). In this case, CA cert has to be imported in the browser. For insecure mode i.e. DEV mode, it can be accessed at http://< host ip >:7070/etcdkeeper/.
Click on the version of the title to select the version of ETCD. The default is V3. Reopening will remember your choice.
Right click on the tree node to add or delete.
For secure mode, authentication is required. User name and password needs to be entered in the dialogue box.
Username is ‘root’ and default password is located at ETCD_ROOT_PASSWORD key under environment section in build/provision/dep/docker-compose-provision.override.prod.yml.
This service can accessed from a remote system at address: https://$(HOST_IP):7071 (when EII is running in secure mode). In this case, CA cert has to be imported in the browser. For insecure mode i.e. DEV mode, it can be accessed at http://$(HOST_IP):7071
NOTE:
If ETCD_ROOT_PASSWORD is changed, EII must to be provisioned again. Please follow below command:
$ cd [WORKDIR]/IEdgeInsights/build/provision $ sudo -E ./provision.sh <path_to_eii_docker_compose_file> $ # eq. $ sudo -E ./provision.sh ../docker-compose.yml
Only VideoIngestion and VideoAnalytics based services will have watch for any changes. Any changes done to those keys will be reflected at runtime in EII.
For changes done to any other keys, EII stack needs to be restarted for it to take effect. Please execute below command in working directory build/ to restart EII.
$ cd [WORKDIR]/IEdgeInsights/build $ docker-compose -f docker-compose-build.yml ia_etcd_ui $ docker-compose up
Select etcdkeeper directory
Tool to Generate Configuration Management UI Service¶
By default, after the user provision EII on the edge devices, the configMgr UI service will follow the default configuration on the eii_config.json. Below is the example on changing configuration on ConfigMgr UI and how to save the data back to json format.
Browse the ConfigMgr UI Service as mention at EII Configuration Management Web UI Service Section.
Perform the changes, below example of changing the filter parameter to using dummy filter instead of using the default filter.
For capturing the data back from ETCD Cluster to Json file, run below command
$ cd [WORK_DIR]/IEdgeInsights/build/provision/
$ ./etcd_capture.sh
Below is the example of [WORK_DIR]/IEdgeInsights/build/provision before running the script.
Below is the example of [WORK_DIR]/IEdgeInsights/build/provision after running the script, additional one file of etcd_capture_data.json
Below is the comparison between the original eii_config.json and the updated etcd_capture_data.json file.
Adding new EII service so it gets picked up by Builder¶
Note
Please refer EII sample apps at Samples written in C++, python and golang using EII Core libraries for more details on adding new EII services
Since the builder takes care of registering and running any service present in it’s own directory in the IEdgeInsights directory, this section describes on how to add any new service into the EII stack, subscribe to IEdgeInsights and publish on a new port.
Any service that needs to be added into the EII stack should be added as a new directory in the IEdgeInsights directory. The directory should contain a docker-compose.yml which will be used to deploy the service as a docker container and it should also contain a config.json which contains the required config for the service to run once it is deployed. The config.json will mainly consist of a config section which includes the configuration related parameters required to run the application and an interfaces section which includes the configuration of how this service interacts with other services of the EII stack. The AppName present in environment section in docker-compose.yml file is appended to the config & interfaces like /AppName/config & /AppName/interfaces before being put into the main build/provision/config/eii_config.json.
An example has been provided below on how to write the config.json for any new service, subscribe to VideoAnalytics and publish on a new port:
{
"config": {
"paramOne": "Value",
"paramTwo": [1, 2, 3],
"paramThree": 4000,
"paramFour": true
},
"interfaces": {
"Subscribers": [
{
"Name": "default",
"Type": "zmq_tcp",
"EndPoint": "127.0.0.1:65013",
"PublisherAppName": "VideoAnalytics",
"Topics": [
"camera1_stream_results"
]
}
],
"Publishers": [
{
"Name": "default",
"Type": "zmq_tcp",
"EndPoint": "127.0.0.1:65113",
"Topics": [
"publish_stream"
],
"AllowedClients": [
"ClientOne",
"ClientTwo",
"ClientThree"
]
}
]
}
}
In the above specified config.json, the value of config key is the config required by the service to run and the value of the interfaces key is the config required by the service to interact with other services of EII stack over EII message bus.
The Subscribers value in the interfaces section denotes that this service should act as a subscriber to the stream being published by the value specified by PublisherAppName on the endpoint mentioned in value specified by EndPoint on topics specified in value of Topics key.
The Publishers value in the interfaces section denotes that this service publishes a stream of data after obtaining and processing it from VideoAnalytics. The stream is published on the endpoint mentioned in value of EndPoint key on topics mentioned in the value of Topics key. The services mentioned in the value of AllowedClients are the only clients able to subscribe to the published stream if being published securely over the EIIMessageBus.
Similar to above interface keys, EII services can also have “Servers” and “Clients” interface keys too. For example, check config.json of VideoIngestion service and config.json of SWTriggerUtility tool on how to use.
More details on the interfaces key responsible for the EII MessageBus endpoint configuration
can be found at Etcd_Secrets_Configuration. For more details on Etcd secrets configuration, visit Etcd_Secrets_Configuration
ConfigMgr¶
ConfigMgr (Config manager) provides CPP/Python/Golang APIs to:
fetch an applications configs values from the KV store.
fetch an applications interface values from KV store for pub, sub, server and client.
watch on application’s config changes.
generate EII MessageBus config.
read and set /GlobalEnv variables.
fetch env variables: appname, dev_mode
All these data are stored into the kv store of EII during the provisioning phase and can be changed dynamically by the admin.
Installation¶
The EIIUtils depends on CMake version 3.11+. For Ubuntu 18.04 this is not
the default version installed via apt-get. To install the correct version
of CMake and other ConfigMgr dependencies, please follow eii_libs_installer README
CMAKE_INSTALL_PREFIX needs to be set for the installation:
$ export CMAKE_INSTALL_PREFIX="/opt/intel/eii"
ConfigMgr installs to /opt/intel/eii/lib/. On some platforms this is not included in the LD_LIBRARY_PATH by default. As a result, you must add this directory to your LD_LIBRARY_PATH,
otherwise you will encounter issues using the ConfigMgr. This can
be accomplished with the following export:
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/eii/lib/
Note
You can also specify a different library prefix to CMake through
the CMAKE_INSTALL_PREFIX flag. If different installation path is given via CMAKE_INSTALL_PREFIX, then $LD_LIBRARY_PATH should be appended by $CMAKE_INSTALL_PREFIX/lib.
Install ConfigMgr with Python bindings, Go bindings, Examples, Test suits and Debug Build.¶
$ rm -rf build
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_INCLUDEDIR=$CMAKE_INSTALL_PREFIX/include -DCMAKE_INSTALL_PREFIX=$CMAKE_INSTALL_PREFIX -DWITH_PYTHON=ON -DWITH_GO=ON -DWITH_EXAMPLES=ON -DWITH_TESTS=ON -DCMAKE_BUILD_TYPE=Debug ..
$ make
$ sudo make install
WITH_PYTHON=ON, WITH_GO=ON, WITH_EXAMPLES=ON, WITH_TESTS=ON and CMAKE_BUILD_TYPE=Debug to compile ConfigMgr with Python bindings, Go bindings, Examples, Unit Tests and Debug mode respectively.
Interfaces¶
ConfigMgr parses the data from the kv store (eg: etcd) and the application environment variables for its functionality. It supports Publisher, Subscriber, Server and Client interfaces. Below are the examples for providing different interfaces and different usecases.
Please refer different ways of giving endpoints
{
"Publishers": [
{
"Name": "default",
"Type": "zmq_ipc",
"EndPoint": "/EII/sockets",
"Topics": [
"camera1_stream"
],
"AllowedClients": [
"*"
]
},
{
"Name": "example",
"Type": "zmq_tcp",
"EndPoint": "127.0.0.1:65015",
"Topics": [
"*"
],
"AllowedClients": [
"Visualizer", "VideoAnalytics"
]
}
]
}
Key |
Type |
Required (Mandatory) |
Description |
|---|---|---|---|
|
|
Yes |
Entire publisher interface will be added with in the array. Multiple publish endpoints can be added by adding elements in the array. |
|
|
Yes |
Name of different publisher interfaces |
|
|
Yes |
Specifies ZeroMQ protocol (“zmq_tcp” or “zmq_ipc”) on which data will be published |
|
|
Yes |
In case of TCP or IPC (socket directory only), endpoint should be string as shown in the above examples. In case IPC explicitly specifying socket file, either object or string can be used for EndPoint. Please refer Different ways of specifying endpoint |
|
|
Yes |
Specifying the topics on which data will be published on. Multiple elements in this array can denote multiple topics published on the same endpoint |
|
|
Yes |
Specifying who can subscribe to the the topic on which data is published. If AllowedClients is “*”, then all the provisioned services can receive the data published. |
{
"Subscribers": [
// tcp usecase
{
"Name": "example",
"Type": "zmq_tcp",
"EndPoint": "127.0.0.1:65013",
"PublisherAppName": "VideoIngestion",
"Topics": [
"*"
]
},
// ipc usecase
{
"Name": "default",
"Type": "zmq_ipc",
"EndPoint": ".socks",
"PublisherAppName": "VideoIngestion",
// Topics cannot be "*", if the only IPC directory is given
// if it Topics "*" to be used in ipc, then socket file should be given explicitly.
"Topics": [
"camera1_stream"
]
}
]
}
Key |
Type |
Required (Mandatory) |
Description |
|---|---|---|---|
|
|
Yes |
Entire subscriber interface will be added with in the array. Multiple subscribe endpoints can be added by adding elements in the array |
|
|
Yes |
Name of different subscriber interfaces |
|
|
Yes |
Specifies ZeroMQ protocol (“zmq_tcp” or “zmq_ipc”) through which subscription happens |
|
|
Yes |
In case of TCP or IPC (socket directory only), endpoint should be string as shown in the above examples. In Case IPC explicitly specifying socket file, either object or string can be used for EndPoint. Please refer Different ways of specifying endpoint |
|
|
Yes |
Specifies the publisher’s AppName through which data will be received |
|
|
Yes |
Specifying the topics on which data will be published on. If Topics is “*”, the subscriber receives all the data published on the endpoint, irrespective of the topic names data is published on. Multiple elements in this array can denote multiple topics subscribed on the same endpoint |
{
"Servers": [
// tcp usecase
{
"Name": "default",
"Type": "zmq_tcp",
"EndPoint": "127.0.0.1:9006",
"AllowedClients": [
"VideoAnalytics"
]
},
//ipc usecase
{
"Name": "example",
"Type": "zmq_ipc",
"EndPoint": "/EII/sockets",
"AllowedClients": [
"VideoAnalytics"
]
}
]
}
Key |
Type |
Required (Mandatory) |
Description |
|---|---|---|---|
|
|
Yes |
Entire server interface will be added with in the array. Multiple server endpoints can be added by adding elements in the array |
|
|
Yes |
Name of different server interfaces |
|
|
Yes |
Specifies ZeroMQ protocol (“zmq_tcp” or “zmq_ipc”) through which Server pushes data |
|
|
Yes |
In case of TCP or IPC (socket directory only), endpoint should be string as shown in the above examples. In case IPC explicitly specifying socket file, either object or string can be used for EndPoint. Please refer Different ways of specifying endpoint |
|
|
Yes |
Specifying who can get data is from the Server. If AllowedClients is “*”, then all the provisioned services can connect to Server. |
{
"Clients": [
// tcp usecase
{
"Name": "default",
"ServerAppName": "VideoIngestion",
"Type": "zmq_tcp",
"EndPoint": "127.0.0.1:9006"
},
// ipc usecase
{
"Name": "example",
"ServerAppName": "VideoIngestion",
"Type": "zmq_ipc",
"EndPoint": "/EII/sockets"
}
]
}
Key |
Type |
Required (Mandatory) |
Description |
|---|---|---|---|
|
|
Yes |
Entire client interface will be added with in the array. Multiple client endpoints can be added by adding elements in the array |
|
|
Yes |
Name of different client interfaces |
|
|
Yes |
Server’s AppName to which client connection is established |
|
|
Yes |
Specifies ZeroMQ protocol (“zmq_tcp” or “zmq_ipc”) through which client connection is established |
|
|
Yes |
In case of TCP or IPC (socket directory only), endpoint should be string as shown in the above examples. In case IPC explicitly specifying socket file, either object or string can be used for EndPoint. Please refer Different ways of specifying endpoint |
Overriding of Type and EndPoint:¶
In a given publisher, subscriber, client or server interfaces, the “Type” and “Endpoint” values mentioned in the respective interfaces can be overriden by setting them as env variables.
Let’s take publisher as an example having multiple endpoints and below is its interface:
{
"Publishers": [
{
"Name": "default",
"Type": "zmq_ipc",
"EndPoint": "/EII/sockets",
"Topics": [
"camera1_stream"
],
"AllowedClients": [
"*"
]
},
{
"Name": "example",
"Type": "zmq_tcp",
"EndPoint": "127.0.0.1:65015",
"Topics": [
"*"
],
"AllowedClients": [
"Visualizer", "VideoAnalytics"
]
}
]
}
Let’s say we want to override the Type (“zmq_ipc”) and Endpoint (“/EII/sockets”) of the first publisher’s interface (interface having “Name”:”default”), then env variable should be set with below syntax.
export PUBLISHER_default_TYPE="zmq_tcp"
export PUBLISHER_default_ENDPOINT="127.0.0.1:65013"
In the above command default is the Name of the interface.
Similarly, we can override subscriber’s, client’s and server’s Type and Endpoint.
export SUBSCRIBER_default_TYPE="zmq_tcp"
export SUBSCRIBER_default_ENDPOINT="127.0.0.1:65013"
export SERVER_default_TYPE="zmq_tcp"
export SERVER_default_ENDPOINT="127.0.0.1:65013"
export CLIENT_default_TYPE="zmq_tcp"
export CLIENT_default_ENDPOINT="127.0.0.1:65013"
Let’s take publisher as an example having single endpoint and below is its interface:
{
"Publishers": [
{
"Name": "default",
"Type": "zmq_ipc",
"EndPoint": "/EII/sockets",
"Topics": [
"camera1_stream"
],
"AllowedClients": [
"*"
]
}
}
If we want to override the Type (“zmq_ipc”) and Endpoint (“/EII/sockets”) of publisher’s interface, then env variable should be set with below syntax.
export PUBLISHER_TYPE="zmq_tcp"
export PUBLISHER_ENDPOINT="127.0.0.1:65013"
Similarly, we can override subscriber’s, client’s and server’s Type and Endpoint.
export SUBSCRIBER_TYPE="zmq_tcp"
export SUBSCRIBER_ENDPOINT="127.0.0.1:65013"
export SERVER_TYPE="zmq_tcp"
export SERVER_ENDPOINT="127.0.0.1:65013"
export CLIENT_TYPE="zmq_tcp"
export CLIENT_ENDPOINT="127.0.0.1:65013"
Overriding feature of ConfigMgr will be used in orchestrated scenarios including Kubernetes.
Broker Usecase¶
If publisher and subscriber wants to communicate via broker(ZmqBroker), i.e., if publisher publish data to ZmqBroker and subscriber subscribes from ZmqBroker, then the interfaces of ZmqBroker, subscriber and publisher with respect to zmq_tcp and zmq_ipc protocol as follows.
zmq_tcp Protocol usecase¶
ZmqBroker:¶
{
//X-PUB
"Publishers": [
{
"AllowedClients": [
"*"
],
"EndPoint": "127.0.0.1:5568",
"Name": "backend",
"Topics": [
"*"
],
"Type": "zmq_tcp"
}
],
// X-SUB
"Subscribers": [
{
"AllowedClients": [
"*"
],
"EndPoint": "127.0.0.1:5569",
"Name": "frontend",
"PublisherAppName": "*",
"Topics": [
"*"
],
"Type": "zmq_tcp"
}
]
}
Subscriber¶
{
"Subscribers": [
{
"Name": "default",
"Type": "zmq_tcp",
"EndPoint": "127.0.0.1:5568",
// PublisherAppName will be "ZmqBroker" in case of brokered usecase
"PublisherAppName": "ZmqBroker",
"Topics": [
"*"
]
}
]
}
Publisher¶
{
"Publishers": [
{
"Name": "default",
"Type": "zmq_tcp",
"EndPoint": "127.0.0.1:5569",
"Topics": [
"camera1_stream"
],
// With broker usecase, AllowedClients in Publisher's interface is not required, as it acts as a subscriber to X-SUB
// "brokered" and "BrokerAppName" should be added for ZmqBroker usecase
"brokered": true,
"BrokerAppName" : "ZmqBroker"
}
]
}
zmq_ipc Protocol usecase¶
ZmqBroker:¶
{
"Publishers": [
{
"Name": "backend",
"Type": "zmq_ipc",
"EndPoint": {
"SocketDir": "/EII/sockets",
"SocketFile": "backend-socket"
},
"Topics": ["*"],
"AllowedClients": ["*"]
}
],
"Subscribers": [
{
"Name": "frontend",
"Type": "zmq_ipc",
"EndPoint": {
"SocketDir": "/EII/sockets",
"SocketFile": "frontend-socket"
},
"PublisherAppName": "*",
"Topics": ["*"],
"AllowedClients": ["*"]
}
]
}
Subscriber¶
{
"Subscribers": [
{
"Name": "default",
"Type": "zmq_ipc",
"EndPoint": {
"SocketDir" : "/EII/sockets",
"SocketFile": "backend-socket"
},
// PublisherAppName will be "ZmqBroker" in case of brokered usecase
"PublisherAppName": "ZmqBroker",
// Topics cannot be "*", if the only IPC directory is given
// if it Topics "*" to be used in ipc, then socket file should be given explicitly.
"Topics": [
"*"
]
}
]
}
Publisher¶
{
"Publishers": [
{
"Name": "default",
"Type": "zmq_ipc",
"EndPoint": {
"SocketDir" : "/EII/sockets",
"SocketFile": "frontend-socket"
},
"Topics": [
"camera1_stream"
],
// With broker usecase, AllowedClients in Publisher's interface is not required, as it acts as a subscriber to X-SUB
// "brokered" and "BrokerAppName" should be added for ZmqBroker usecase
"brokered": true,
"BrokerAppName" : "ZmqBroker"
}
]
}
Key |
Type |
Required (Mandatory) |
Description |
|---|---|---|---|
|
|
Yes |
(Required if publishing via ZmqBroker) Specifies if publisher is using broker or not. use “brokered”: true for use with broker. |
|
|
Yes |
(Required if publishing via ZmqBroker) Specifies ZeroMQ Broker app name |
Note “Endpoint” can be given in different ways:¶
zmq_tcp Endpoint should look like below:
"Endpoint":"127.0.0.1:65013"
zmq_ipc Endpoint have different ways of giving.
Specifying just socket directory
"Endpoint":"/EII/sockets"
Specifying socket directory and socket file
"Endpoint":{ "SocketDir" : "/EII/sockets", "SocketFile": "socketfile" } or "Endpoint":"/EII/sockets, socketfile"
Running Examples¶
The ConfigMgr library also supports Cpp APIs and Python & Go bindings. These APIs/bindings can be used in Cpp and Python/Go services in the EII stack to fetch required config/interfaces/msgbus config.
Examples will only be compiled if the WITH_EXAMPLES=ON option is specified while running CMake.
Please refer Examples installation
Refer the interfaces of publisher and server in ./examples/configs/VideoAnalytics_interfaces.json and for subscriber and client, refer ./examples/configs/VideoAnalytics_interfaces.json
Examples on demonstrating the usage of these APIs in the bindings have been given in respective sections below.
CPP Examples¶
$ Navigate to [WORKDIR]/IEdgeInsights/common/libs/ConfigMgr
$ sudo rm -rf build
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_INCLUDEDIR=$CMAKE_INSTALL_PREFIX/include -DCMAKE_INSTALL_PREFIX=$CMAKE_INSTALL_PREFIX -DWITH_EXAMPLES=ON ..
$ make
$ sudo make install
There are currently 5 CPP examples:
examples/sample_pub.cppexamples/sample_sub.cppexamples/sample_server.cppexamples/sample_client.cppexamples/sample_getvalue.cpp
All of the Cpp example executables are in the build/examples/ directory. To run
them, execute the following command:
Before executing any of the examples, please run below command from build/examples/
cd ../../examples/ && source ./env.sh && cd -
Publisher example.
$ ./pub
Subscriber example.
$ ./sub
Server example.
$ ./server
Client example.
$ ./client
Sample_getvalue used to get the values from application’s config
$ ./sample_app
Python Examples¶
$ Navigate to [WORKDIR]/IEdgeInsights/common/libs/ConfigMgr
$ sudo rm -rf build
$ mkdir build
$ cd build
$ cmake -DCMAKE_INSTALL_INCLUDEDIR=$CMAKE_INSTALL_PREFIX/include -DCMAKE_INSTALL_PREFIX=$CMAKE_INSTALL_PREFIX -DWITH_PYTHON=ON ..
$ make
$ sudo make install
There are currently 5 Python examples:
examples/publisher.pyexamples/subscriber.pyexamples/echo_service.pyexamples/echo_client.pyexamples/sample_get_value.py
All of the py examples are in python/examples/ directory. To run
them, execute the following command:
Before executing any of the examples, please run below command from python/examples/
cd ../../examples/ && source ./env.sh && cd -
Publisher example.
$ python3 publisher.py
Subscriber example.
$ python3 subscriber.py
Server example.
$ python3 echo_service.py
Client example.
$ python3 echo_client.py
sample_get_value used to get the values from application’s config
$ python3 sample_get_value.py
Go Examples¶
$ Navigate to [WORKDIR]/IEdgeInsights/common/libs/ConfigMgr/go
$ cp -r ConfigMgr/ $GOPATH/src
$ cd ConfigMgr/examples
$ source go_env.sh
There are currently 5 Go examples:
publisher.gosubscriber.goecho_service.goecho_client.goapp_config.go
All of the go examples are in go/examples/ directory. To run
them, execute the following command:
Before executing any of the examples, please run below command from go/ConfigMgr/examples/
cd ../../../examples/ && source ./env.sh && cd -
source ./go_env.sh
Publisher example.
$ go build publisher.go
$ ./publisher
Subscriber example.
$ go build subscriber.go
$ ./subscriber
Server example.
$ go build echo_service.go
$ ./echo_service
Client example.
$ go build echo_client.go
$ ./echo_client
app_config used to get the values from application’s config
$ go build app_config.go
$ ./app_config
Running Unit Tests¶
The unit tests will only be compiled if the
WITH_TESTS=ONoption is specified when running CMake. Please refer Unit Test installation installation.Provisioning should be done to start etcd server in dev/prod mode and to generate application specific certificates(only in prod mode).
Before executing any of the test files, please run below command from build/tests/
cd ../../examples/ && source ./env.sh && cd -
To run ConfigMgr unit tests .. code-block:
$ ./config_manager_unit_tests $ ./kvstore_client-tests