Contents¶

ZeroMQ Broker Overview¶

Using the ZeroMQ broker or broker you can proxy the data coming over the Message Bus ZeroMQ protocol plug-in for publishers and subscribers. Traditionally, each Message Bus ZeroMQ publisher has its own IPC socket or TCP (host, port) combination. The ZeroMQ broker enables subscribers to connect to a single central subscriber TCP or IPC socket to receive incoming messages from the broker, and publishers to connect to the proxy’s publisher TCP or IPC socket to transmit published messages.

The ZeroMQ broker builds from the functionality provided by the ZeroMQ zmq_proxy() API, which uses two separate sockets and hands messages off from one socket to the other.

The following diagram shows a high-level flow of how the ZeroMQ Broker functions.

+-----------+    +-----------+    +-----------+
| Publisher |    | Publisher |    | Publisher |
+-----------+    +-----------+    +-----------+
|  ZMQ_PUB  |    |  ZMQ_PUB  |    |  ZMQ_PUB  |
+-----------+    +-----------+    +-----------+
   connect          connect          connect
     |                |                 |
     +----------------+-----------------+
                      |
                     bind
              +----------------+
              |    ZMQ_XSUB    |
              +----------------+
              |     proxy      |
              +----------------+
              |    ZMQ_XPUB    |
              +----------------+
                     bind
                      |
      +---------------+-------------------+
      |               |                   |
   connect          connect            connect
+------------+    +------------+    +------------+
|  ZMQ_SUB   |    |  ZMQ_SUB   |    |  ZMQ_SUB   |
+------------+    +------------+    +------------+
| Subscriber |    | Subscriber |    | Subscriber |
+------------+    +------------+    +------------+

As shown in the diagram, the broker uses two ZeroMQ sockets:

A socket for publishers to connect that is the ZMQ_XSUB socket that is referred to as the frontend socket.
A socket for the subscribers to connect to that is the ZMQ_XPUB socket that is referred to as the backend socket.

The broker binds to the TCP (host, port) or the IPC socket file (based on its configuration) for each of its sockets. Messages are sent from publishers to the frontend socket and then relayed to the backend socket that forwards the messages to the subscribers.

Security¶

The security for the broker depends on whether the broker is using TCP or IPC for its respective sockets.

Note

The broker can be used with the frontend being a TCP socket and the backend being an IPC socket and vice versa.

The following sections cover the security for the TCP and the IPC modes.

IPC Mode¶

When the broker is using an IPC socket for either the frontend or backend sockets, the ability to send or receive messages is handled by the Linux file permissions. The messages sent over the IPC sockets are not encrypted.

TCP Mode¶

For TCP connections, ZeroMQ broker offers mechanisms for both encryption and authentication. This is accomplished via elliptical curve keys generated when EII is provisioned. These keys provide the encryption and authentication via the ZAP protocol and a list of allowed clients authentication is handled.

The EII Message Bus handles all the encryption and authentication for publishers and subscribers when they connect to the broker. The necessary keys for message decryption and deciding whether a specific publisher or subscriber is permitted to connect are held by the broker via setup of the broker’s messaging interfaces.

See the configuration section for more details on listing out allowed clients and configuring the security for the ZeroMQ broker.

Important Security Implication: For TCP connections, the broker manages a single list of all the publishers and subscribers that are allowed to connect. If a given subscriber can connect, then all the data coming through that broker instance is available to that subscriber to receive. Essentially what this means, is that the broker does not do authentication on a per topic/stream basis, only on a connection basis.

Performance Implications¶

When using the ZeroMQ broker, it is important to understand the implications on the performance of the networking in the EII platform.

By using the broker, an extra network hop is incurred for all messages sent over the Message Bus ZeroMQ protocol plugin. This will increase the latency for the messages sent from the publishers connecting to the broker to the subscribers connecting to the broker.

Important Note: Having the broker running in an EII deployment does not make every message from publishers go through the broker. Each publisher must be configured to connect to the broker. Publishers that are not configured to connect to the broker will still bind to their respective IPC socket or TCP (host, port) combination.

Steps to Independently Build and Deploy the ZeroMQ Broker Service¶

Note

For running 2 or more microservices, we recommend that you try the use case-driven approach for building and deploying. For more information, refer to the Readme.

Steps to Independently Build the ZeroMQ Broker Service¶

Note

When switching between independent deployment of the service with and without the dependency for the config manager agent service, you can run into issues with docker-compose build for the Certificates folder existence. As a workaround, run the sudo rm -rf Certificates command to proceed with docker-compose build.

To build the ZeroMQ broker service independently, complete the following steps:

The downloaded source code should have a directory named ZmqBroker:
```
cd IEdgeInsights/ZmqBroker
```
Copy the IEdgeInsights/build/.env file using the following command in the current folder
```
cp ../build/.env .
```
Note: Update the HOST_IP and ETCD_HOST variables in the .env file with your system IP.
```
# Source the .env using the following command:
set -a && source .env && set +a
```
Independently build
```
docker-compose build
```

Steps to Independently Deploy the ZeroMQ Broker Service¶

You can deploy the ZeroMQ broker service in any of the following ways:

Without Config Manager Agent Dependency
With the Config Manager Agent Dependency

Deploy the ZeroMQ Broker Service without the Config Manager Agent Dependency¶

Run the following commands to deploy the ZeroMQ Broker service without the Config Manager Agent dependency:

# Enter the ZmqBroker directory
cd IEdgeInsights/ZmqBroker

Copy the IEdgeInsights/build/.env file using the following command in the current folder, if not already present.
cp ../build/.env .
Note: Ensure that docker ps is clean and docker network ls doesn’t have EII bridge network.

Update .env file for following:
1. HOST_IP and ETCD_HOST variables with your system IP.
2. `READ_CONFIG_FROM_FILE_ENV` value to `true` and `DEV_MODE` value to `true`.

Source the .env using the following command:
set -a && source .env && set +a

# Run the service
docker-compose -f docker-compose.yml -f docker-compose-dev.override.yml up -d

Note

The ZmqBroker container restarts automatically when its config is modified in the config.json file. To update the config.json file using the vi or the vim editor, append the set backupcopy=yes in ~/.vimrc. This allows the changes made on the host machine config.json file to reflect in the container mount point.

Deploy the ZeroMQ Broker Service with the Config Manager Agent Dependency¶

Complete the following steps to deploy the ZeroMQ broker service with the Config Manager Agent dependency.

Note

Ensure that the Config Manager Agent image present in the system. If not, build the Config Manager Agent locally when independently deploying the service with Config Manager Agent dependency.

# Enter the ZmqBroker directory
cd IEdgeInsights/ZmqBroker

Copy the IEdgeInsights/build/.env file using the following command in the current folder, if not already present.
cp ../build/.env .
Note: Ensure that docker ps is clean and docker network ls does not have EII bridge network.

Update .env file for following:
1. HOST_IP and ETCD_HOST variables with your system IP.
2. `READ_CONFIG_FROM_FILE_ENV` value is set to `false`.

Copy the docker-compose.yml from IEdgeInsights/ConfigMgrAgent as docker-compose.override.yml in IEdgeInsights/ZmqBroker.

cp ../ConfigMgrAgent/docker-compose.yml docker-compose.override.yml

Copy the builder.py with standalone mode changes from IEdgeInsights/build directory
cp ../build/builder.py .
Run the following command to run the builder.py in the standalone mode. This will generate the eii_config.json file and update the docker-compose.override.yml file.

python3 builder.py -s true

Building the service (This step is optional for building the service if not already done in the Independently buildable step above)
docker-compose build
For running the service in PROD mode, run the below command:

NOTE: Make sure to update DEV_MODE to false in .env while running in PROD mode and source the .env using the command set -a && source .env && set +a.
docker-compose up -d
For running the service in DEV mode, run the below command:

NOTE: Make sure to update DEV_MODE to true in .env while running in DEV mode and source the .env using the command set -a && source .env && set +a.
docker-compose -f docker-compose.yml -f docker-compose-dev.override.yml -f docker-compose.override.yml up -d

Configuration¶

Note

The JSON schema for the configuration properties of the ZeroMQ Broker can be found in the schema.json file.

The configuration of the ZeroMQ Broker is predominantly accomplished from the EII Configuration Manager APIs. However, it can also be configured via JSON files, as described in the Usage section. The configuration via JSON files are only for development or debugging purposes. This section will focus on the production configuration of the broker.

The configuration of the broker falls into two main areas:

Application/Service Configuration - Configuration of the runtime behavior.
Interface Configuration - Configuration of the networking interfaces that is the frontend and the backend sockets.

Aside from the interface and the service configuration, there are two properties that are set via environmental variables: AppName and DEV_MODE. The AppName variable determines how the service queries its configuration using the Configuration Manager APIs. For docker-compose, this is set in the environment section of the docker-compose.yml file under the ia_zmq_broker section. The default value for this is ZmqBroker.

The DEV_MODE variable sets whether the broker will attempt to use certificates with the Configuration Manager APIs and over any TCP connections. This is also obtained from the environment section of the docker-compose.yml file.

Important Note: There are some other configuration properties in the environment section of the docker-compose.yml file. These are provided to the service from the build/.env file. For more information, refee to the EII User Guide.

The following sections cover the various configuration properties that the broker supports for interfaces and application configuration.

Application or Service Configuration¶

The broker supports two configuration parameters; one for the Linux scheduler policy and the other for the scheduler priority. These two properties are applied to the main thread which is running the ZeroMQ proxy.

The configuration values must be specified in the config.json before running the builder.py script or eii_config.json after using the builder.py script.

The following table specifies the details for these parameters. Note that neither of these are required in the configuration and should only be set if the user knows they need to have a certain scheduling policy and priority for the broker.

Key	Type	Description
`sched_policy`	`string`	Scheduling policy to set for the main thread. Must be: `SCHED_OTHER`, `SCHED_IDLE`, `SCHED_BATCH`, `SCHED_FIFO`, or `SCHED_RR`.
`sched_priority`	`integer`	Sets the priority for the thread. Must be between `0` and `99`. Only valid for `SCHED_FIFO` and `SCHED_RR` policies.

An example of applying these two settings to the ZeroMQ Broker’s configuration is shown as follows. In this example, the configuration is an excerpt from the, IEdgeInsights/build/provision/config/eii_config.json, generated while using the builder.py script.

{
    // ... omitted ...

    "/ZmqBroker/config": {
        "sched_policy": "SCHED_FIFO",
        "sched_priority": 42
    }

    // ... omitted ...
}

Note

There will also be an /ZmqBroker/interfaces section associated with the broker, but that is covered in the next section.

The configuration shown in the example sets the Linux scheduler policy to SCHED_FIFO and its priority to 42. This can also be left entirely empty, and the broker thread will default to the Linux default thread scheduling policy or priority.

Interface Configuration¶

The configuration of the networking interfaces for the ZeroMQ Broker is divided into two sections

Subscribers - This section subscribes to the messages coming from publishers and sets the configuration for the frontend.
Publishers - section determines the configuration for the backend socket.

The Configuration Manager API assumes that the Subscribers and the Publishers section is a list. However, since the ZeroMQ broker does not use multiple publisher and subscriber sockets, the lists must only contain one element. The single JSON object that is expected to be in the Subscribers and the Publishers sections is essentially the same.

The following example shows the interfaces configuration for a broker instance with TCP sockets for both the frontend and backend.

{
    "Subscribers": [
        {
            "Name": "frontend",
            "Type": "zmq_tcp",
            "EndPoint": "0.0.0.0:60514",
            "Topics": ["*"],
            "PublisherAppName": "*",
            "AllowedClients": ["*"]
        }
    ],
    "Publishers": [
        {
            "Name": "backend",
            "Type": "zmq_tcp",
            "EndPoint": "0.0.0.0:60515",
            "Topics": ["*"],
            "AllowedClients": ["*"]
        }
    ]
}

The Name and Topics fields differ, which is the main difference to note. The string frontend must be used as the value for the Name key in the Subscribers section. There can be just one value of * in the Topics list. The same guidelines apply to the object called Publishers. Notably, the word * appears in the phrase AllowedClients.You can modify this to a list of the app names of the services that are permitted to connect if you only want to allow specific services to connect to the broker. As a result, the broker can use the defined names frontend and backend to query specific configuration items. The AllowedClients list should be changed to only include the app names of the services that are authorized to connect if the deployment needs to restrict the services that can connect to the broker.

For IPC sockets, the configuration is very similar, except that the EndPoint and the Type values are different, as shown in the following example:

{
    "Subscribers": [
        {
            "Name": "frontend",
            "Type": "zmq_ipc",
            "EndPoint": {
                "SocketDir": "/tmp/socks",
                "SocketFile": "frontend-socket"
            },
            "PublisherAppName": "*",
            "Topics": ["*"],
            "AllowedClients": ["*"]
        }
    ],
    "Publishers": [
        {
            "Name": "backend",
            "Type": "zmq_ipc",
            "EndPoint": {
                "SocketDir": "/tmp/socks",
                "SocketFile": "backend-socket"
            },
            "Topics": ["*"],
            "AllowedClients": ["*"]
        }
    ]
}

In this example, note that the Type value is zmq_ipc and the EndPoint is a JSON object that defines the socket directory and file to use for the sockets.

Important Notes

There are two important notes to be aware of with the configurations presented earlier:

Although the configurations above showcase using IPC and TCP for both the frontend and backend sockets, they can be used in any combination. For example, the broker can use an IPC socket for the frontend and TCP for the backend.

When using IPC sockets, the name of the SocketFile must also be specified for all subscribers or publishers that will be connecting to that socket.

Docker¶

The ZeroMQ broker is integrated into the EII docker-compose ecosystem. To build and use the Docker container, follow the defined building or provisioning instructions for EII .

If you are using a YAML file with the builder.py script, then ensure to add ZmqBroker under the AppName section of the YAML file. This will include the broker into the resulting docker-compose.yml file and also include the broker’s configuration in the eii_config.json which is used while provisioning.

Connecting the EII Services to the ZeroMQ Broker¶

To connect the EII services, such as the VideoIngestion or VideoAnalytics services, you must edit the interfaces configuration of the services to tell them the broker instance to connect to. Namely, you need to add two keys to the Publishers configuration to have two additional keys:

BrokerAppName - This specifies the AppName of the targeted broker instance.
brokered - This tells the Message Bus that the given publisher instance is brokered.

As an example, following is the changes in the interfaces configuration for enabling the VideoIngestion and VideoAnalytics to communicate via ZmqBroker.

Note

:

Though the example shows VideoIngestion and VideoAnalytics using the ZmqBroker, it is not recommended for video streaming use cases due to the extra hop added to the video frames that will affect performance.

Changes to VideoIngestion/config.json which is a publisher.

The following adds the BrokerAppName and the brokered keys to the default publisher’s configuration.

"interfaces": {
    "Publishers": [
        {
            "Name": "default",
            "Topics": [
                "camera1_stream"
            ],
            "Type": "zmq_tcp",
            "EndPoint": "ia_zmq_broker:60514",
            "brokered": true
            "BrokerAppName": "ZmqBroker",
            "AllowedClients": [
                "*",
            ],
        }
    ]
 }

For the subscriber that is VideoAnalytics, there won’t be much changed other than changing the endpoint to point to ZmqBroker along with PublisherAppName.

"interfaces": {
    "Subscribers": [
        {
            "Name": "default",
            "Type": "zmq_tcp",
            "EndPoint": "ia_zmq_broker:60515",
            "PublisherAppName": "ZmqBroker",
            "Topics": [
                "camera1_stream"
            ],
            "zmq_recv_hwm": 50
        }
    ]
}

Bare Metal¶

Important Note: Running the broker using bare-metal is not recommended for production environments. This way of running the broker is meant purely for development and debugging purposes.

Compilation¶

The ZeroMQ Broker is written in C++ and as such utilizes the CMake build system for building the binary for the broker.

The ZeroMQ Broker has the following dependencies:

CMake 3.15+
IntelSafeString
EII Utils
libzmq
EII ConfigMgr
MessageBus (only required for unit tests)

Before building the ZeroMQ Broker, you must install these libraries on the target system. This can be done using the common/eii_libs_installer.sh script.

After the dependencies for the broker are installed on your system, complete the following steps to build the ZeroMQ Broker.

To create the build directory, run the following command:
```
mkdir build/
```
Change directories into the newly created build/ directory
```
cd build/
```
Run the CMake command to set up the build environment (see subsections).
1. To compile without the unit tests run the following command:
```
cmake ..
```
1. To compile the unit tests, run the following command. Note that it is recommended to build in the Debug mode.
```
cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_TESTS=ON ..
```
To build the ZeroMQ broker binary, run the following command:
```
make
```

Usage¶

After building the ZeroMQ broker you will have a binary named, “zmq-broker”. Use this binary to run the ZeroMQ broker.

The ZeroMQ broker can be configured from the EII Configuration Manager as well as from environmental variables and JSON configuration files.

To use the configuration obtained via the EII Configuration Manager, simply start the binary with no parameters; however, ensure to set the AppName and the DEV_MODE environmental variables. As mentioned at the beginning of the Bare Metal section, this way of running the broker should only be done in development environments. As such, it is assumed that the DEV_MODE environmental variable will be set to true and no security is used with the Configuration Manager or over Message Bus ZeroMQ TCP connections. Additionally, for running in this way, the ia_etcd container must be running.

# Set DEV_MODE environmental variable to "true"
 export DEV_MODE=true

# Set the AppName environmental variable to "ZmqBroker"
 export AppName=ZmqBroker

# Run the broker
 ./zmq-broker

The broker also supports receiving its configuration through JSON configuration files and environmental variables. To set the log level, Linux scheduler policy and priority set the following environmental variables. Refer to the details in the [Configuration] (#configuration) section.

Note

: None of the following environmental variables are required to run the broker.

C_LOG_LEVEL - Log level (will default to ERROR log level)
SCHED_POLCIY - Scheduler policy
SCHED_PRIORITY - Scheduler priority

When using JSON files, the broker will expect two positional command line arguments. The first will be the configuration of the frontend socket and the second will be for the backend configuration.

REMINDER: The frontend refers to the configuration for socket which publishers shall connect to, and the backend refers to the configuration for the socket which subscribers shall connect to.

The contents of the JSON is analogous to the JSON configuration files used in the Message Bus (see the IEdgeInsights/common/libs/EIIMessageBus/README.md for more information). The contents of the JSON files depends on whether if the given socket is supposed to use TCP or IPC.

For IPC configurations, the frontend and the backend configurations need to look respectively as follows:

Frontend:

{
    // Which ZeroMQ protocol to use (TCP vs. IPC)
    "type": "zmq_ipc",

    // Socket directory to create the socket file in
    "socket_dir": "/tmp",

    // Specifies the IPC endpoint config (i.e. socket directory and file)
    "": {
        // IPC socket file to bind to for the frontend socket
        "socket_file": "frontend-sock"
    }
}

Note

:

This example configuration is stored in examples/ipc_frontend_example.json file and can be used with the examples/configs/ipc_publisher_brokered.json Message Bus configuration file.

The following table desribes the purpose of each key in the JSON configuration file.

Key	Description
`type`	Specifies the ZeroMQ protocol to use, must be either `zmq_ipc` or `zmq_tcp`.
`socket_dir`	Specifies the directory in which to create all of the IPC socket files.
`""`	Gives the IPC socket file configuration for the frontend IPC socket.
`socket_file`	This key specifies the name of the socket file to bind to.

For the socket_file key, when connecting a publisher, the publisher’s configuration must also use this same socket_file key in the configuration for it. Otherwise, it will fail to connect because it will attempt to bind/connect to its topic name (see the Message Bus’s documentation for more details).

Backend:

{
    // Which ZeroMQ protocol to use (TCP vs. IPC)
    "type": "zmq_ipc",

    // Socket directory to create the socket file in
    "socket_dir": "/tmp",

    // Specifies the IPC endpoint config (i.e. socket directory and file)
    "": {
        // IPC socket file to bind to for the backend socket
        "socket_file": "backend-sock"
    }
}

Note

: This example configuration is stored in the examples/ipc_backend_example.json file and can be used with the examples/configs/ipc_subscriber_brokered.json Message Bus configuration file.

The configuration for the backend socket is the same as for the frontend.

For TCP configurations, the frontend and the backend configurations need to look respectively as follows:

Frontend:

{
    // Which ZeroMQ protocol to use (TCP vs. IPC)
    "type": "zmq_tcp",

    // List of curve public keys for publishers allowed to connect to the
    // frontend socket
    "allowed_clients": ["4J4?(I13cwJgqi+T5nxg:Dyr5)l&reK]cxxTfa9V"],

    // Specifies the configuration for the frontend socket
    "": {
        // TCP host
        "host": "0.0.0.0",

        // TCP port
        "port": 5568,

        // Server secret key to use for encryption / authentication
        "server_secret_key": "H[J1%f:#?0Y1cpUL<nEkPKm.]:n@JI>j]*!u3N#9"
    }
}

Note

This example configuration is stored in the examples/tcp_frontend_example.json file and can be used with the examples/configs/tcp_publisher_brokered_with_security.json Message Bus configuration file.

The following table describes the purpose of each key in the JSON configuration file.

Key	Description
`type`	Specifies the ZeroMQ protocol to use, must be either `zmq_ipc` or `zmq_tcp`.
`host`	TCP host.
`port`	TCP port.
`allowed_clients`	Specifies the list of public keys for the publishers that are allowed to connect.
`""`	Gives security configuration for the frontend TCP socket.
`server_secret_key`	Secret key for the frontend socket for encryption / authentication.

Important Note:

When using JSON files for the configuration of the ZeroMQ Broker, the JSON file is the configuration used by the Message Bus, rather than the normal EII interface configurations. This is why the JSON configuration differs from the configuration given via the eii_config.json. The JSON shown earlier for the frontend, and the following backend configuration, represent how the configuration manager alters that configuration to be in these forms.

Backend:

{
    // Which ZeroMQ protocol to use (TCP vs. IPC)
    "type": "zmq_tcp",

    // List of curve public keys for subscribers allowed to connect to the
    // backend socket
    "allowed_clients": ["y-G]27SC}{!mC4(]}=c=KH1M{x)Kd/{i%o]j7YT3"],

    // Specifies the configuration for the frontend socket
    "zmq_tcp_publish": {
        // TCP host
        "host": "0.0.0.0",

        // TCP port (note that this is a different port from the frontend)
        "port": 5569

        // Server secret key to use for encryption / authentication
        "server_secret_key": "qydUsM#PP4r<E*2]<]kbLRwk1IX:H^y{Gk56e:tc"
    }
}

Note

: This example configuration is stored in the examples/tcp_backend_example.json file and can be used with the examples/configs/tcp_subscriber_with_security.json Message Bus configuration file.

All the keys mentioned earlier have the same purpose and meaning as for the frontend configuration. Except there is one key difference; instead of the empty string key, the backend configuration has the zmq_tcp_publish key. This is because the broker adopts the same configuration definitions as the Message Bus ZeroMQ protocol plugin. The ZeroMQ protocol plugin uses the zmq_tcp_publish key for setting the TCP (host, port) combinations for all publishers under a single message bus context. The broker uses the same configuration to keep it consistent between the two entities.

After the frontend and the backend JSON configuration files have been created, run the broker binary as follows:

# IPC Example
$ ./zmq-broker examples/ipc_frontend_example.json examples/ipc_backend_example.json

# TCP Example
$ ./zmq-broker examples/tcp_frontend_example.json examples/tcp_backend_example.json

Note

It is assumed the configurations are named as frontend_config.json and backend_config.json respectively. You can name the configurations anything. For more examples of configurations, see the JSON files in the tests/configs/ directory.

Running Unit Tests¶

If the ZeroMQ broker was built with the -DWITH_TESTS=ON flag set for the CMake command, then do the following to run the unit tests:

Note

The following commands assume you are in the build/ directory that was created when compiling the broker.

Run the following command to go to the tests/ directory
```
cd tests/
```
To execute the unit tests, run the following command:
```
./broker-tests
```

To run the unit tests with a different log level, provide one of the following log levels: DEBUG, INFO, WARN, ERROR with the command-line argument --log-level. An example of using the DEBUG log level is shown as follows:

./broker-tests --log-level DEBUG