Contents ======== * `Contents <#contents>`__ * `EII Provision and Deployment <#eii-provision-and-deployment>`__ * `Prerequisites <#prerequisites>`__ * `Updating the Helm Charts Directory <#updating-the-helm-charts-directory>`__ * `Provision and deploy EII in the Kubernetes Node <#provision-and-deploy-eii-in-the-kubernetes-node>`__ * `Provision and Deploy Mode in Times Switching between Dev and Prod Mode or Changing the Usecase <#provision-and-deploy-mode-in-times-switching-between-dev-and-prod-mode-or-changing-the-usecase>`__ * `For Running Helm Charts and Deploying kube pods with specific namespace <#for-running-helm-charts-and-deploying-kube-pods-with-specific-namespace>`__ * `Steps for Enabling Accelarators <#steps-to-enable-accelarators>`__ * `Steps for Enabling GiGE Camera with Helm <#steps-for-enabling-gige-camera-with-helm>`__ * `Setting up Multus CNI and Enabling it <#setting-up-multus-cni-and-enabling-it>`__ * `Accessing Web Visualizer and EtcdUI <#accessing-web-visualizer-and-etcdui>`__ * `Prerequisites on K8s MultiNode Cluster Environment <#pre-requisites-on-k8s-multinode-cluster-environment>`__ EII Provision and Deployment ---------------------------- For deployment of Edge Insights for Industrial(EII), helm charts are provided for both provision and deployment. .. note:: \ : * In this document, there are labels of 'Edge Insights for Industrial (EII)' for file names, paths, code snippets,etc. * Same procedure has to be followed for single or multi node. * Login to /configure docker registry before running helm. This is required when public docker hub for accessing images is not used. Prerequisites ------------- **Note**\ : * K8s installation on single or multi node must be done as pre-requisite to continue the following deployment. Note: The kubernetes cluster is set up with ``kubeadm``\ , ``kubectl`` and ``kubelet`` packages on single and multi nodes with ``v1.23.4``. Refer to tutorials such as https://adamtheautomator.com/install-kubernetes-ubuntu/#Installing_Kubernetes_on_the_Master_and_Worker_Nodes and many other online tutorials to setup kubernetes cluster on the web with host OS as ubuntu 18.04/20.04. * For helm installation, refer to `helm website `_ .. For time series usecase make sure ia_mqtt_broker and ia_mqtt_publisher are running. For preparing the necessary files required for the provision and deployment, execute the build and provision steps on an Ubuntu 18.04 / 20.04 machine. Follow the Docker prerequisites, EII Prerequisites, Provision EII and Build and Run EII mentioned in `README.md `_ on the Ubuntu dev machine. As EII does not distribute all the docker images on docker hub, run into issues of those pods status showing ``ImagePullBackOff`` and few pods status like visualizer, factory ctrl etc., showing ``CrashLoopBackOff`` due to additional configuration required. For ``ImagePullBackOff`` issues, follow the steps mentioned at [../README.md#distribution-of-eii-container-images]> (../README. md#distribution-of-eii-container-images) to push the images that are locally built to the docker registry of choice. Ensure to update the ``DOCKER_REGISTRY`` value in ``[WORKDIR]/IEdgeInsights/build/.env`` file and re-run the ../builder.py(\ ``[WORK_DIR]/IEdgeInsights/build/builder.py``\ ) script to regenerate the helm charts for provision and deployment. .. note:: For deploying EII pods across nodes with access to persistent volumes, one can use the "Network File System" distributed storage as explained at `README_NFS.md `_ Update the Helm Charts Directory -------------------------------- #. Edit the "EII_HOME_DIR" in .env(\ ``[WORK_DIR]/IEdgeInsights/build/.env``\ ) with /home/username/\`_ with the preferred usecase for generating the consolidated helm charts for the provisioning and deployment. .. code-block:: sh cd [WORKDIR]/IEdgeInsights/build python3 builder.py -f usecases/.yml #. Following steps are required both in DEV and PROD mode: Provision and Deploy EII in the Kubernetes Node ----------------------------------------------- .. note:: Ensure older certificates and eii-certs secret from the k8s cluster is ``deleted``. #. Copy the helm charts in helm-eii/ directory to the node. .. **Note:** For Edge Video Analytics Microservice helm deployment, follow https://github.com/intel-innersource/applications.services.esh.edge-video-analytics-microservice/blob/main/README.md for downloading the ``models``. Once models folder created, copy the ``models``\ , ``resources`` and ``eii/pipelines`` directories from EdgeVideoAnalyticsMicroservice repo to ``/opt/intel/eii`` directory on the host machine. #. Install deploy helm chart .. code-block:: sh cd [WORKDIR]/IEdgeInsights/build/helm-eii/ helm install eii-deploy eii-deploy/ .. The Certificates/ directory contains sensitive information. So post the installation of eii-deploy helm chart, it is recommended to delete the Certificates from it. **Note:** Visualizer won't show the frames and to see the frames coming in, it is required to update the localhost IP with the worker node IP where the visualizer streaming pod is running (\ ``kubectl get nodes`` command will help them get the node IP address where the visualizer streaming pod is running), find the detailed steps on updating IP address on Grafana dashboard referred in the Visualizer README.md at ``[WORKDIR]/IEdgeInsights/Visualizer/multimodal-data-visualization/README.md##accessing-visualizer-from-other-node`` section. Set the ingestion pipeline for the video ingestion pod, and install the deploy helm chart as follows: .. code-block:: sh helm install --set env.PIPELINE="" eii-deploy eii-deploy/ For Example(USB Camera), .. code-block:: sh helm install --set env.PIPELINE="/dev/video0" eii-deploy eii-deploy/ **Note:** ConfigMgrAgent service needs to be initialized before other services during runtime. In case other services are initialized before ConfigMgrAgent, notice "cfgmgr initialization failed" exception. After generating this exception the services must restart and continue to run. Verify all the pod are running: .. code-block:: sh kubectl get pods EII is now successfully deployed. Provision and Deploy Mode in Times Switching between Dev and Prod Mode or Changing the Use case ----------------------------------------------------------------------------------------------- #. Set the DEV_MODE as "true/false" in .env(\ ``[WORK_DIR]/IEdgeInsights/build/.env``\ ) depending on dev or prod mode. #. Run builder to copy templates file to eii-deploy/templates directory and generate consolidated values.yaml file for eii-services: .. code-block:: sh cd [WORKDIR]/IEdgeInsights/build python3 builder.py -f usecases/.yml #. Remove the etcd storage directory .. code-block:: sh sudo rm -rf /opt/intel/eii/data/* Repeat helm install of deploy chart as per previous section. .. note:: \ : During re-deployment (\ ``helm uninstall`` and ``helm install``\ ) of helm chart, wait for all the pervious pods to terminate. For Running Helm Charts and Deploying Kube Pods with Specific Namespace ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. note:: \ : By default, all our helm charts are deployed with ``default`` namespace, below commands will help us to deploy helm chart and kube pods with specific namespace .. code-block:: sh helm install --set namespace= / --namespace --create-namespace For Example, * For Deploying ``eii-deploy`` helm chart with ``eii`` namespace. .. code-block:: sh helm install --set namespace=eii eii-deploy eii-deploy/ --namespace eii --create-namespace * Now all the ``pods`` and ``helm`` charts are deployed under ``eii`` namespace * For listing ``helm charts`` deployed with specific namespace .. code-block:: sh helm ls -n * For listing ``kube pods`` deployed with specific namespace .. code-block:: sh kubectl get pods -n Steps for Enabling Accelarators ------------------------------- .. note:: \ : ``nodeSelector`` is the simplest recommended form of node selection constraint. ``nodeSelector`` is a field of PodSpec. It specifies a map of key-value pairs. For the pod to be eligible to run on a node, the node must have each of the indicated key-value pairs as labels (it can have additional labels as well). The most common usage is one key-value pair. #. Setting the ``label`` for a particular node .. code-block:: sh kubectl label nodes = #. For NCS2 dependenecies, follow the steps for setting ``labels``. * For NCS2 .. code-block:: sh kubectl label nodes ncs2=true .. note:: \ : Here the node-name is your worker node machine hostname #. Open the ``[WORKDIR]/IEdgeInsights/VideoIngestion/helm/values.yaml`` or ``[WORKDIR]/IEdgeInsights/VideoAnalytics/helm/values.yaml`` file. #. Based on your workload preference. Add ncs2 to accelerator in values.yaml of video-ingestion or video-analytics. * For NCS2 .. code-block:: yml config: video_ingestion: . . accelerator: "ncs2" . . #. Set device as "MYRIAD" in case of ncs2 and as HDDL in case of hddl in the VA config(\ ``[WORK_DIR]/IEdgeInsights/VideoAnalytics/config.json``\ ) * In case of ncs2. .. code-block:: yml "udfs": [{ . . . "device": "MYRIAD" }] #. Run the ``[WORKDIR]/IEdgeInsights/build/builder.py`` for generating latest consolidated ``deploy`` yml file based on your ``nodeSelector`` changes set in the respective Modules. .. code-block:: sh cd [WORKDIR]/IEdgeInsights/build/ python3 builder.py #. Follow the `Deployment Steps <#provision-and-deploy-in-the-kubernetes-node>`__ #. Verify the respecitve workloads are running based on the ``nodeSelector`` constraints. Steps for Enabling GiGE Camera with Helm ---------------------------------------- **Note:** For more information on ``Multus`` refer to git https://github.com/intel/multus-cni Skip installing multus if it is already installed. Prequisites ^^^^^^^^^^^ For enabling gige camera with helm. Helm pod networks should be enabled ``Multus`` Network Interface to attach host system network interface access by the pods for connected camera access. **Note**\ : Follow the steps and ensure ``dhcp daemon`` is running fine. If there is an error on ``macvlan`` container creation on accessing the socket or if socket was not running. Execute the steps again. .. code-block:: sh sudo rm -f /run/cni/dhcp.sock cd /opt/cni/bin sudo ./dhcp daemon .. note:: When CNI issue occurs make sure to cleanup the cluster and re-initialize the cluster. Setting up Multus CNI and Enabling it ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * Multus CNI is a container network interface (CNI) plug-in for Kubernetes that enables attaching multiple network interfaces to pods. Typically, in Kubernetes each pod only has one network interface (apart from a loopback) -- with Multus you can create a multi-homed pod that has multiple interfaces. This is accomplished by Multus acting as a "meta-plug-in", a CNI plug-in that can call multiple other CNI plug-ins. #. Get the name of the ``ethernet`` interface in which gige camera and host system connected **Note**\ : Identify the network interface name by following command .. code-block:: sh ifconfig #. Execute the following Script with Identified ``ethernet`` interface name as Argument for ``Multus Network Setup`` **Note:** Pass the ``interface`` name without ``quotes`` .. code-block:: sh cd [WORKDIR]/IEdgeInsights/build/helm-eii/gige_setup sudo -E sh ./multus_setup.sh .. **Note**\ : Verify ``multus`` pod is in ``Running`` state .. code-block:: sh kubectl get pods --all-namespaces | grep -i multus #. Set gige_camera to true in values.yaml .. code-block:: sh $ vi [WORKDIR]/IEdgeInsights/VideoIngestion/helm/values.yaml . . gige_camera: true . . #. Follow the `Deployment Steps <#provision-and-deploy-in-the-kubernetes-node>`__ #. Verify ``pod``\ ip & ``host`` ip are same as per Configured ``Ethernet`` interface by using below command. .. code-block:: sh kubectl exec -it -- ip -d address .. note:: * Deploy with ``root`` user rights for GPU device, MYRIAD(NCS2) device and GenICam USB3.0 interface cameras. * Refer the below configuration file snip to deploy with ``root`` user rights using ``runAsUser`` field. .. code-block:: apiVersion: apps/v1 kind: Deployment ... spec: ... spec: ... containers: .... securityContext: runAsUser: 0 Steps for Enabling CustomUdfs ----------------------------- For deploying the Custom UDFs, complete the following steps: Enable the required CustomUDF services. As per the EII default scenario, the sample custom UDF containers are not mandatory containers to run, hence the ``builder.py`` must run the ``video-streaming-all-udfs.yml`` use case. #. Make sure Visualizer(mulitimodaldatavisulization/streaming) subscribing w.r.t usecases. For running **Multimodal Data Visualization/Streaming** with **CustumUdfs** as a publisher, the config can be updated to subscribe to the EndPoint and topic of **CustumUdfs** in the following way. Refer to the following example multimodalvisualization configuration to subscribe **PySafetyGearAnalytics** CustomUDF results. Open the ``[WORKDIR]/IEdgeInsights/Visulizer/multimodal-data-visualization-streaming/eii/config.json`` and ``[WORKDIR]/IEdgeInsights/Visulizer/multimodal-data-visualization/eii/config.json`` .. code-block:: javascript { "interfaces": { "Subscribers": [ { "Name": "default", "Type": "zmq_tcp", "EndPoint": "ia_python_safety_gear_analytics:65019", "PublisherAppName": "PySafetyGearAnalytics", "Topics": [ "py_safety_gear_stream_results" ] } ] } } #. Helm MultiModalVisulation templete changes for CustomUdfs. Open the ``[WORKDIR]/IEdgeInsights/Visulizer/multimodal-data-visualization-streaming/eii/helm/templates/multimodal-data-visualization-streaming.yaml`` Replace the following lines in multimodal-data-visualization-streaming.yaml from 97 to 99 and 106 to 108 and 115 to 118 instead of video_analytics with pythonsafetygearanalytics.For example refer the below example and change w.r.t CustomUdfs. .. code-block:: yml {{- if ($global.Values.config.pythonsafetygearanalytics) }} {{- $subscriber_port = $global.Values.config.pythonsafetygearanalytics.publish_port }} {{- end }} .. code-block:: yml {{- if ($global.Values.config.pythonsafetygearanalytics) }} {{- $subscriber_port = add $global.Values.config.pythonsafetygearanalytics.publish_port $instance_idx }} {{- end }} .. code-block:: yml {{- if ($global.Values.config.pythonsafetygearanalytics) }} - name: SUBSCRIBER_ENDPOINT value: "{{ $.Values.config.pythonsafetygearanalytics.name }}:{{ $subscriber_port }}" {{- end }} #. Run builder to copy templates file to eii-deploy/templates directory and generate consolidated values.yaml file for eii-services: .. code-block:: sh cd [WORKDIR]/IEdgeInsights/build python3 builder.py -f usecases/video-streaming-all-udfs.yml .. note:: The [NativePclIngestion](../CustomUdfs/NativePclIngestion] UDF does not support helm deployment. Accessing Visualizer and EtcdUI ------------------------------- Environment EtcdUI and Visualizer will be running in following ports. * EtcdUI * ``https://master-nodeip:30010/`` * Visualizer * PROD Mode -- ``https://master-nodeip:30001/`` * DEV Mode -- ``http://master-nodeip:30001/`` Environment VideoAnalytics/VideoIngestion and EVAM will be running in following ports. * VideoAnalytics/VideoIngestion * PROD Mode -- ``https://master-nodeip:30008/camera1_stream_results`` * DEV Mode -- ``http://master-nodeip:30009/camera1_stream_results`` * EVAM * PROD Mode -- ``https://master-nodeip:30008/edge_video_analytics_results`` * DEV Mode -- ``http://master-nodeip:30009/edge_video_analytics_results`` Prerequisites on K8s MultiNode Cluster Environment -------------------------------------------------- .. note:: * For running EII in Prod Mode, it is required to have self-signed certificates that are getting generated need to go in as kubernetes secrets. To make this happen, it is mandatory that ConfigManager Agent Pod & generating Certs Pod gets scheduled in k8s Cluster's Master/Control Plane node * By default, EII deployment charts are deployed in PROD mod and only ConfigManager Agent and generating Certs pods gets scheduled on Master/Control Plane node of K8s Cluster * Ensure that the Master Node is *tainted* to schedule the pod .. code-block:: sh kubectl taint nodes --all node-role.kubernetes.io/master- .. note:: For deploying EII pods across nodes with access to persistent volumes, use the "Network File System" distributed storage as explained at `README_NFS.md `_.