Data Collection and Annotation as a Service (DCaaS)
===================================================

DCaaS is a microservice that helps in collection and annotation of input
video data. The collected/annotated data can then be used for AI model
training/fine-tuning and statistical analysis.

It supports 3 design patterns which is targeted to cover different
deployment scenarios:

+---------------------------+------------------------------------------+
| Design Pattern            | Remarks                                  |
+===========================+==========================================+
| Simple Data Collection    | Data ingest from source and save in      |
|                           | local database                           |
+---------------------------+------------------------------------------+
| Simple Data Collection    | Data ingest from source and data         |
| with Human Annotation     | annotation by a user along with review   |
+---------------------------+------------------------------------------+
| Simple Data Collection    | Data ingest from source and data         |
| with Auto-Annotation      | annotation by an (AI) algorithm          |
+---------------------------+------------------------------------------+

Below figure outlines architecture of DCaaS. Please click
`here <docs/overview-data-collection-and-annotation-as-a-service.rst>`__
for more details

.. raw:: html

   <p align="left">

.. raw:: html

   <p>

Contents
--------

-  `How to run DCaaS <#how-to-run-dcaas>`__

   -  `Basic pre-requisites <#basic-pre-req>`__
   -  `Get Code base from GitHub <#get-code-base-from-github>`__
   -  `Install pre-requisites and proxy setup <#install-prereqs>`__
   -  `Configure, Build and Run the Services <#config-build-run>`__
   -  `Using DCaaS in other modes <#dcaas-other-modes>`__

-  `License <#license>`__
-  `More Details <#more-details>`__
-  `Troubleshooting <#troubleshooting>`__

How to run DCaaS 
----------------

Basic pre-requisites 
~~~~~~~~~~~~~~~~~~~~

-  Git

-  

   ============ ======
   OS           Python
   ============ ======
   Ubuntu 20.04 3.8
   Ubuntu 22.04 3.10
   ============ ======

Get Code base from GitHub 
~~~~~~~~~~~~~~~~~~~~~~~~~

Note: See `Setup for Development <USAGE.md#setup-for-development>`__ for
details 1. Install repo tool.
``sh       curl https://storage.googleapis.com/git-repo-downloads/repo > repo       sudo mv repo /bin/repo       sudo chmod a+x /bin/repo``
2. Create a working directory and initialize the eii-manifest repo. For
more details, read
`here <https://github.com/intel-innersource/applications.industrial.edge-insights.manifests/blob/v4.0-RC6/README.md>`__
``sh       mkdir -p [WORKDIR]           cd [WORKDIR]       repo init -u "https://github.com/intel-innersource/applications.industrial.edge-insights.manifests.git" -b main       repo sync``

4. Clone DCaaS repository.
   ``sh       cd IEdgeInsights       git clone https://github.com/intel-innersource/frameworks.ai.edgecsp.data-collection-as-a-service DataCollectionMicroservice       cd DataCollectionMicroservice       git checkout develop``

Install pre-requisites and proxy setup 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Install docker utilities and python packages required during first
   installation
   ``sh     cd [WORKDIR]/IEdgeInsights/build     sudo -E ./pre_requisites.sh     ## run with proxy if needed     # sudo -E ./pre_requisites.sh --proxy="<Proxy-DNS>:<port>"``

2. Create proxy configuration for docker. Create
   ``[HOMEDIR]/.docker/config.json`` and add the below with the
   appropriate ``<Proxy-DNS>:<port>`` \```json { “proxies”: { “default”:
   { “httpProxy”: “http://:”, “httpsProxy”: “http://:”, “noProxy”:
   “intel.com,*.intel.com,10.0.0.0/8,192.168.0.0/16,172.16.0.0/12” } } }

   \``\`

Configure, Build and Run the Services 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1. Edit the ``.env`` file and add entries for the following:
   :literal:`sh     # DEB packages source location     PKG_SRC=http://eii-nightly-devops.iind.intel.com/latest     # Host ip address to be updated here     HOST_IP=     ETCD_HOST=     # Service credentials     ETCDROOT_PASSWORD=     INFLUXDB_USERNAME=     INFLUXDB_PASSWORD=     MINIO_ACCESS_KEY=     MINIO_SECRET_KEY=     # These are required to be updated in PROD mode only     WEBVISUALIZER_USERNAME=     WEBVISUALIZER_PASSWORD=     # Path where remote backup will happen     DCAAS_STORAGE_DIR=/path/to/persistent/remote/storage     # set the below variable if DEV_MODE is set to `true` in the current .env file     DCAAS_UDF_DIR=/udfs`
   Edit the ``.env`` file further as mentioned
   `here <USAGE.md#human-annotation-mode>`__

   Add ``"DataCollectionMicroservice": ""`` to the ``"subscriber_list"``
   in [WORKDIR]/IEdgeInsights/build/builder_config.json

   **Note**: Set ``DCAAS_STORAGE_DIR`` to the directory where you wish
   to remotely save the annotations and images (ex:
   /path/to/persistent/remote/storage) and run the below. Azure users
   need additional setup before being able to upload data to azure.
   Refer `Remote Storage <USAGE.md#remote-storage>`__ for more details.
   ``sh    sudo chmod 777 /path/to/persistent/remote/storage``

2. Run the builder script to generate deployment and configuration
   files. To learn more, click
   `here <https://github.com/intel-innersource/applications.industrial.edge-insights.core/tree/v4.0-RC6#task-2-generate-the-deployment-and-the-configuration-files>`__

   .. code:: sh

      (venv) python3 builder.py -f ../DataCollectionMicroservice/artifacts/sample-usecase.yml

3. Build the OEI usecase. To learn more, click
   `here <https://github.com/intel-innersource/applications.industrial.edge-insights.core/tree/v4.0-RC6#task-3-build-and-run-the-edge-insights-for-industrial-video-and-time-series-use-cases>`__
   ``sh     (venv) docker compose -f docker-compose.yml build``

4. Start License Manager

   Before DCaaS or any other services can be started, we need to start
   the license manager agent. DCaaS will do a license check during the
   start of the app as well as during runtime periodically. The license
   manager agent will allow DCaaS to check if the user has valid
   entitlement for usage. Failure during this check will exit DCaaS
   gracefully.

   For 1st time setup, please see
   `wiki <https://wiki.ith.intel.com/display/SBS/Setting+up+of+Network+Lease+with+Thales+Sentinel+RMS+SDK>`__

   If you have already done the first time setup, open a different
   terminal and navigate to the directory where the ``LM_SCP_Start.sh``
   script is located (ideally the extracted directory during the 1st
   time setup) ``sh  ./LM_SCP_Start.sh``

5. Start DCaaS and other Services ``sh     (venv) ./run.sh`` Once the
   above script runs, you should see the below prints. To check DCaaS
   container running logs and for more information, please refer
   `Viewing logs <USAGE.md#viewing-logs>`__

6. Stop the Services ``sh     (venv) docker compose down -v``

Using DCaaS in other modes 
--------------------------

-  To run DCaaS in “Auto Annotation Mode” follow instructions
   `here <USAGE.md#auto-annotation-mode>`__

-  To run DCaaS in “Simple Storage Mode” (stores the frames from
   “data_filter” in DataStore)

   -  Remove ``"DataCollectionMicroservice": ""`` from the
      ``"subscriber_list"`` in
      [WORKDIR]/IEdgeInsights/build/builder_config.json if it already
      exists
   -  Edit the default `config.json <config.json>`__ as below

      -  ``enabled``

         -  Ensure that you have set ``"enabled": true`` for
            “simple_storage”
         -  Ensure that you have set ``"enabled": false`` for both
            “auto_annotation” and “human_annotation”

License 
-------

License details can be found in `License.pdf <License.pdf>`__

More Details 
------------

Please refer `USAGE.md <USAGE.md>`__ for more details

Troubleshooting 
---------------

If you run into any issue, please refer
`TROUBLESHOOTING.md <TROUBLESHOOTING.md>`__