Config Manager Library¶

ConfigMgr Overview¶

ConfigMgr or Config manager provides CPP, Python, and Golang APIs to:

fetch an application’s configs and interface values from the KV store (in the PROD and DEV mode) or from the config file (only in the DEV mode) based on your input “READ_CONFIG_FROM_FILE_ENV” for the pub, sub, server, and client.
monitor application’s config changes.
generate MessageBus config.
read and set the /config/logging env variables.
fetch the env variables: appname, dev_mode

All the ConfigMgr operations related data is stored in the KV store of Edge Insights for Industrial during the provisioning phase. The admin can dynamically change these data.

ConfigMgr Installation¶

You can install the EII Config Manager library in any of the following ways:

Through published Debian, Fedora, or Alpine APK packages
Installing from source

If you are installing from one of the packages then from the releases assets, select the package to install. Based on the OS that on which you are installing, run the following:

# Debian
sudo apt install libcjson1 libzmq5 zlib1g
sudo dpkg -i <debian package>

# Fedora
sudo dnf install cjson zeromq zlib
sudo rpm -i <rpm package>

# Alpine (NOTE: the depencies get automatically installed by the apk command)
sudo apk add --allow-untrusted <apk package>

In the above commands, installing the cJSON and ZeroMQ dependencies is required, however, in general, installation of the dev module is not required (i.e. the OS packages which include all of the headers for the libraries). If you are compiling an application that is linking to this library, then it is recommended that you install the dev versions of the libraries.

For Ubuntu, install libcjson-dev libzmq3-dev zlib1g-dev
For Fedora, install cjson-devel zeromq-devel zlib-devel
For Alpine, install cjson-dev zeromq-dev zlib-dev

If you want to compile the Config Manager from source, follow the intructions:

The ConfigMgr depends on CMake version 3.11+. For Ubuntu 18.04 this is not the default version installed via apt-get. To install the correct version of CMake and other ConfigMgr dependencies, refer to the eii_libs_installer README

To set CMAKE_INSTALL_PREFIX for the installation, run the following command:

export CMAKE_INSTALL_PREFIX="/opt/intel/eii"

ConfigMgr installs to /opt/intel/eii/lib/. On some platforms, this is not included in the LD_LIBRARY_PATH by default. As a result, you must add this directory to your LD_LIBRARY_PATH, otherwise you will encounter issues using the ConfigMgr. This can be accomplished with the following export:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/intel/eii/lib/

Note

You can also specify a different library prefix to CMake through the CMAKE_INSTALL_PREFIX flag. If different installation path is given via CMAKE_INSTALL_PREFIX, then $LD_LIBRARY_PATH should be appended by $CMAKE_INSTALL_PREFIX/lib.

To install the remaining dependencies for the Message bus, run the following command:

Note

: It is highly recommended that you use a python virtual environment to install the python packages, so that the system python installation doesn’t get altered. For details on setting up and using Python virtual environment, refer to `Python virtual environment <https://www.geeksforgeeks.org/python-virtual-environment/>>`_.

sudo apt install libcjson-dev libzmq5 zlib1g-dev

Note

For Fedora, the packages is cjson-devel zeromq-devel zlib-devel. For Alpine, the package is cjson-dev zeromq-dev zlib-dev.

If you wish to compile the Python binding as well, then you must also install the Python requirements. To do this, run the following pip command:

pip3 install --user -r ./python/requirements.txt

Using System gRPC¶

The ConfigMgr library can be built using the gRPC version already installed on the system. To do this, use the SYSTEM_GRPC flag when running the cmake command. Following is an example of the same.

cmake -DSYSTEM_GRPC=ON ..

Note

The ConfigMgr library depends on a specific gRPC version, 1.29.0. A debian package is provided for this in the grpc-package directory. To install this in Ubuntu run the following command:

sudo dpkg -i grpc-package/grpc-1.29.0-Linux.deb

Compilation¶

The Config Manager utilizes CMake as the build tool for compiling the library. The simplest sequence of commands for building the library are as follows.

mkdir build
cd build
cmake ..
make

This will compile only the C library for the ConfigMgr. To build with the Python binding, specify the WITH_PYTHON flag, when executing the cmake command. Refer to the following:

cmake -DWITH_PYTHON=ON ..

If you wish to include installation of the Go binding with the installation of the EII library, then specify the WITH_GO flag when executing the cmake command.

cmake -DWITH_GO=ON ..

Note

This only copies the Go binding library to your system’s $GOPATH. If you do not have your $GOPATH specified in your system’s environmental variables then an error will occur while executing the cmake command.

In addition to the WITH_PYTHON and WITH_GO flags, the ConfigMgr CMake files add flags for building the C examples and the unit tests associated with the library. The following table specifies all of the available flags that can be given to CMake for building the ConfigMgr.

Flag	Default	Description
`WITH_TESTS`	`OFF`	If set to `ON`, builds the C unit tests with the ConfigMgr compilation
`WITH_EXAMPLES`	`OFF`	If set to `ON`, then CMake will compile the C examples in addition to the library
`WITH_DOCS`	`OFF`	If set to `ON`, then CMake will add a `docs` build target to generate documentation

Note

These flags are in addition to any and all flags that are available for the cmake command. See the CMake documentation for additional flags.
See the Generating Documentation section.

If you wish to compile the ConfigMgr in debug mode, then you can set the the CMAKE_BUILD_TYPE to Debug when executing the cmake command. Refer to the following:

cmake -DCMAKE_BUILD_TYPE=Debug ..

Generating Documentation¶

Generating the documentation has several dependencies which are not installed by the install.sh script. You must install the following packages to generate the documentation:

sudo apt install doxygen texlive-full

Warning: This install way take a very long time. It will install more than 4GB of packages.

If you are building the Python binding by using the WITH_PYTHON flag, then you must also install Sphinx and an extension for Sphinx. This can be accomplished with the following commands:

sudo apt install python3-sphinx
sudo -H -E pip3 install m2r

Note

The commands above assume you already have Python 3.6 and pip installed on your system.

Go documentation generation is WIP.

Once you have completed these steps, the documentation can be generated by running the following make command:

make docs && make docs

Note

Currently you need to run make docs twice so that the table of contents is generated correctly for each of the documents. This will be fixed in the future.

The PDF documents will be in the docs/pdfs/ directory within your build directory. There will be other log files and output files associated with the building of the PDFs. Any file that does not end in .pdf can be ignored.

Packaging¶

Note

If the build is done using the system installed gRPC, then packaging cannot be done . This is due to the various linking issues that can occur in that scenario.

This library supports being packaged as a Debian, RPM, or Alpine APK packages. This is all accomplished via CMake. By default, packaging is disabled. To enable packaging, add the -DPACKAGING=ON flag to your CMake command (see Compilation section). This command will look something like:

cmake -DPACKAGING=ON ..

By default, the packaging utilities will scan the system for the required toolchains it needs to build each package type (Deb, RPM, and APK). If it does not find the required toolsets, then it will disable that form of packaging. The packaging utilities provide CMake flags to force packaging as any of the supported package types. If a given package type, ex. APK, is set to be enabled manually by its CMake flag and its required packaging toolchain does not exist, then CMake will raise a fatal error.

The table provides the required toolchains for each package type as well as the CMake flag to set to ON to manually enable a packaging type:

Package Type	Required Tools	Manual Package Flag
`deb`	`dpkg-deb`	`PACKAGE_DEB`
`rpm`	`rpmbuild`	`PACKAGE_RPM`
`apk`	`docker`	`PACKAGE_APK`

Note

Manually setting a given package type to be built (e.g. setting -DPACKAGE_DEB=ON) still requires that the -DPACKAGING=ON to be set.

After the required toolchains have been installed and CMake has been run with some combination of the packaging flags, the library can be packaged with the following commands:

make package

The command above will build the Debian and RPM packages (depending on the specified CMake flags).

To build the Alpine APK package, run the following command:

make package-apk

IMPORTANT:

The Config Manager depends on the EII Utils and Message bus libraries. To compile the Alpine APK package for the Config Manager it must have the APK packages for the EII Utils and Message bus modules.

To provide this, you must first build or download the Alpine APK package for the EII Utils library (see its repo here([WORK_DIR]/IEdgeInsights/common/util/c/) to obtain the library) and also for the Message bus (see it’s repo here([WORK_DIR]/IEdgeInsights/common/libs/EIIMessageBus/) to obtain the library).

After you have the APKs, create an apks directory at the top level of this repository.

mkdir apks/

Next, place the EII Utils and Message bus APK packages into the, apks, directory. Then run the make package-apk command. If this is not done, then the build will fail.

Note on Alpine APK Packaging¶

To package the library as an Alpine APK package, the packaging utility must use a Docker container to have access to the proper Alpine APK toolchains. This container will automatically be built when the CMake command is ran to configure your build environment.

By default, Alpine 3.14 is used to build the package. However, this version can be changed by setting the APKBUILD_ALPINE_VERSION CMake flag to the version of Alpine you wish to use (ex. -DAPKBUILD_ALPINE_VERSION=3.12).

Install ConfigMgr with Python bindings, Go bindings, Examples, Test suits, and Debug Build¶

rm -rf build
mkdir build
cd build
cmake -DCMAKE_INSTALL_INCLUDEDIR=$CMAKE_INSTALL_PREFIX/include -DCMAKE_INSTALL_PREFIX=$CMAKE_INSTALL_PREFIX -DWITH_PYTHON=ON -DWITH_GO=ON -DWITH_EXAMPLES=ON -DWITH_TESTS=ON -DCMAKE_BUILD_TYPE=Debug ..
make
sudo make install

WITH_PYTHON=ON, WITH_GO=ON, WITH_EXAMPLES=ON, WITH_TESTS=ON and CMAKE_BUILD_TYPE=Debug to compile ConfigMgr with Python bindings, Go bindings, Examples, Unit Tests and Debug mode respectively.

Interfaces¶

ConfigMgr parses the data from the kv store (eg: etcd) and the application environment variables for its functionality. It supports Publisher, Subscriber, Server and Client interfaces. Following are the examples for providing different interfaces and different usecases.

Refer to different ways of giving endpoints

Publisher Interface¶

{
    "Publishers": [
        {
            "Name": "default",
            "Type": "zmq_ipc",
            "EndPoint": "/EII/sockets",
            "Topics": [
                "edge_video_analytics_results"
            ],
            "AllowedClients": [
                "*"
            ]
        },
        {
            "Name": "example",
            "Type": "zmq_tcp",
            "EndPoint": "127.0.0.1:65015",
            "Topics": [
                "*"
            ],
            "AllowedClients": [
                "Visualizer"
            ]
        }
    ]
}

Key	Type	Required (Mandatory)	Description
`Publishers`	`array`	Yes	Entire publisher interface will be added with in the array. Multiple publish endpoints can be added by adding elements in the array.
`Name`	`string`	Yes	Name of different publisher interfaces
`Type`	`string`	Yes	Specifies ZeroMQ protocol (“zmq_tcp” or “zmq_ipc”) on which data will be published
`EndPoint`	`string` or `object`	Yes	In case of TCP or IPC (socket directory only), endpoint should be string as shown in the above examples. In case IPC explicitly specifying socket file, either object or string can be used for EndPoint. Refer to Different ways of specifying endpoint
`Topics`	`array`	Yes	Specifying the topics on which data will be published on. Multiple elements in this array can denote multiple topics published on the same endpoint
`AllowedClients`	`array`	Yes	Specifying who can subscribe to the the topic on which data is published. If AllowedClients is “*”, then all the provisioned services can receive the data published.

Subscriber Interface¶

{
    "Subscribers": [

        // tcp usecase
        {
            "Name": "example",
            "Type": "zmq_tcp",
            "EndPoint": "127.0.0.1:65013",
            "PublisherAppName": "EdgeVideoAnalyticsMicroservice",
            "Topics": [
                "*"
            ]
        },

        // ipc usecase
        {
            "Name": "default",
            "Type": "zmq_ipc",
            "EndPoint": ".socks",
            "PublisherAppName": "EdgeVideoAnalyticsMicroservice",
            // Topics cannot be "*", if the only IPC directory is given
            // if it Topics "*" to be used in ipc, then socket file should be given explicitly.
            "Topics": [
                "edge_video_analytics_results"
            ]
        }
    ]
}

Key	Type	Required (Mandatory)	Description
`Subscribers`	`array`	Yes	Entire subscriber interface will be added with in the array. Multiple subscribe endpoints can be added by adding elements in the array
`Name`	`string`	Yes	Name of different subscriber interfaces
`Type`	`string`	Yes	Specifies ZeroMQ protocol (“zmq_tcp” or “zmq_ipc”) through which subscription happens
`EndPoint`	`string` or `object`	Yes	In case of TCP or IPC (socket directory only), endpoint should be string as shown in the above examples. In Case IPC explicitly specifying socket file, either object or string can be used for EndPoint. Refer to Different ways of specifying endpoint
`PublisherAppName`	`string`	Yes	Specifies the publisher’s AppName through which data will be received
`Topics`	`array`	Yes	Specifying the topics on which data will be published on. If Topics is “*”, the subscriber receives all the data published on the endpoint, irrespective of the topic names data is published on. Multiple elements in this array can denote multiple topics subscribed on the same endpoint

Server Interface¶

{
    "Servers": [
        // tcp usecase
        {
            "Name": "default",
            "Type": "zmq_tcp",
            "EndPoint": "127.0.0.1:9006",
            "AllowedClients": [
                "MultiModalVisualizationStreaming"
            ]
        },
        //ipc usecase
        {
            "Name": "example",
            "Type": "zmq_ipc",
            "EndPoint": "/EII/sockets",
            "AllowedClients": [
                "MultiModalVisualizationStreaming"
            ]
        }
    ]
}

Key	Type	Required (Mandatory)	Description
`Servers`	`array`	Yes	Entire server interface will be added with in the array. Multiple server endpoints can be added by adding elements in the array
`Name`	`string`	Yes	Name of different server interfaces
`Type`	`string`	Yes	Specifies ZeroMQ protocol (“zmq_tcp” or “zmq_ipc”) through which Server pushes data
`EndPoint`	`string` or `object`	Yes	In case of TCP or IPC (socket directory only), endpoint should be string as shown in the above examples. In case IPC explicitly specifying socket file, either object or string can be used for EndPoint. Refer to Different ways of specifying endpoint
`AllowedClients`	`array`	Yes	Specifying who can get data is from the Server. If AllowedClients is “*”, then all the provisioned services can connect to Server.

Client Interface¶

{
    "Clients": [
        // tcp usecase
        {
            "Name": "default",
            "ServerAppName": "EdgeVideoAnalyticsMicroservice",
            "Type": "zmq_tcp",
            "EndPoint": "127.0.0.1:9006"
        },

        // ipc usecase
        {
            "Name": "example",
            "ServerAppName": "EdgeVideoAnalyticsMicroservice",
            "Type": "zmq_ipc",
            "EndPoint": "/EII/sockets"
        }
    ]
}

Key	Type	Required (Mandatory)	Description
`Clients`	`array`	Yes	Entire client interface will be added with in the array. Multiple client endpoints can be added by adding elements in the array
`Name`	`string`	Yes	Name of different client interfaces
`ServerAppName`	`string`	Yes	Server’s AppName to which client connection is established
`Type`	`string`	Yes	Specifies ZeroMQ protocol (“zmq_tcp” or “zmq_ipc”) through which client connection is established
`EndPoint`	`string` or `object`	Yes	In case of TCP or IPC (socket directory only), endpoint should be string as shown in the above examples. In case IPC explicitly specifying socket file, either object or string can be used for EndPoint. Refer to Different ways of specifying endpoint

Overriding of Type and EndPoint¶

In a given publisher, subscriber, client or server interfaces, the “Type” and “Endpoint” values mentioned in the respective interfaces can be overridden by setting them as env variables.

Let’s take publisher as an example having multiple endpoints and its interface:

{
    "Publishers": [
        {
            "Name": "default",
            "Type": "zmq_ipc",
            "EndPoint": "/EII/sockets",
            "Topics": [
                "edge_video_analytics_results"
            ],
            "AllowedClients": [
                "*"
            ]
        },
        {
            "Name": "example",
            "Type": "zmq_tcp",
            "EndPoint": "127.0.0.1:65015",
            "Topics": [
                "*"
            ],
            "AllowedClients": [
                "MultiModalVisualization", "MultiModalVisualizationStreaming"
            ]
        }
    ]
}

Let’s say we want to override the Type (“zmq_ipc”) and Endpoint (“/EII/sockets”) of the first publisher’s interface (interface having “Name”:”default”), then env variable should be set with following syntax.

export PUBLISHER_default_TYPE="zmq_tcp"
export PUBLISHER_default_ENDPOINT="127.0.0.1:65013"

In the above command default is the Name of the interface.

Similarly, we can override subscriber’s, client’s and server’s Type and Endpoint.

export SUBSCRIBER_default_TYPE="zmq_tcp"
export SUBSCRIBER_default_ENDPOINT="127.0.0.1:65013"

export SERVER_default_TYPE="zmq_tcp"
export SERVER_default_ENDPOINT="127.0.0.1:65013"

export CLIENT_default_TYPE="zmq_tcp"
export CLIENT_default_ENDPOINT="127.0.0.1:65013"

Let’s take publisher as an example having single endpoint and following is its interface:

{
    "Publishers": [
        {
            "Name": "default",
            "Type": "zmq_ipc",
            "EndPoint": "/EII/sockets",
            "Topics": [
                "edge_video_analytics_results"
            ],
            "AllowedClients": [
                "*"
            ]
        }
}

If we want to override the Type (“zmq_ipc”) and Endpoint (“/EII/sockets”) of publisher’s interface, then env variable should be set following syntax.

export PUBLISHER_TYPE="zmq_tcp"
export PUBLISHER_ENDPOINT="127.0.0.1:65013"

Similarly, we can override subscriber’s, client’s and server’s Type and Endpoint.

export SUBSCRIBER_TYPE="zmq_tcp"
export SUBSCRIBER_ENDPOINT="127.0.0.1:65013"

export SERVER_TYPE="zmq_tcp"
export SERVER_ENDPOINT="127.0.0.1:65013"

export CLIENT_TYPE="zmq_tcp"
export CLIENT_ENDPOINT="127.0.0.1:65013"

Overriding feature of ConfigMgr will be used in orchestrated scenarios including Kubernetes.

Config Manager Library¶

ConfigMgr Overview¶

ConfigMgr Installation¶

Using System gRPC¶

Compilation¶

Generating Documentation¶

Packaging¶

Note on Alpine APK Packaging¶

Install ConfigMgr with Python bindings, Go bindings, Examples, Test suits, and Debug Build¶

Interfaces¶

Publisher Interface¶

Subscriber Interface¶

Server Interface¶

Client Interface¶

Overriding of Type and EndPoint¶

API Documentation¶