Edge-to-Cloud Bridge for Microsoft Azure* Service

ID 729768
Updated 11/30/2022
Version Latest
Public

author-image

By

Overview

The Edge-to-Cloud Bridge for Microsoft Azure* service is an open-source microservice that serves as a bridge between an edge device and the Microsoft Azure ecosystem†. It sends analytics data to the cloud, i.e., Microsoft Azure IoT Hub by connecting with Microsoft Azure IoT Edge runtime and sends image data to Microsoft Azure Blob storage using Azure IoT Blob storage runtime. It subscribes to the data publishing microservices via MQTT broker or OEI Message Bus and receives the data to send to Azure cloud.

When deployed with Edge Insights for Industrial (EII) microservices, the Edge-to-Cloud Bridge microservice subscribes to OEI Message Bus for receiving analytics data and incoming video images. It also obtains the configurations for EII from its Azure IoT Hub digital twin and translates them to the etcd configuration manager via EII Configuration Manager APIs.

When deployed independently, the Edge-to-Cloud Bridge microservice subscribes to an MQTT broker for analytics data. The data can be published to the MQTT broker or OEI Message Bus by other microservices such as the Edge Video Analytics Microservice, which performs video analytics.

Select Configure & Download to download the microservice and the software listed below.

Configure & Download

Cloud Connector Icon

  • Time to Complete: 45 minutes
  • Programming Language: Python* 3
  • Available Software: 
    • Edge-to-Cloud Bridge for Microsoft Azure*
    • Edge Video Analytics Microservice

Target System Requirements

  • One of the following processors:
    • 6th to 11th generation Intel® Core™ processors
    • 1st to 3rd generation of Intel® Xeon® Scalable processors
    • Intel Atom® processor with Intel® Streaming SIMD Extensions 4.2 (Intel® SSE4.2) 
  • At least 8 GB RAM.
  • At least 64 GB hard drive.
  • An Internet connection.
  • One of the following Linux* distributions:
    • Ubuntu* 18.04.3 LTS Kernel 5.0†
    • Ubuntu* 20.04 LTS Kernel 5.4†

 

† Use Kernel 5.8 for 11th generation Intel® Core™ processors.


How It Works 

Edge-to-Cloud Bridge for Microsoft Azure* Service

The Edge-to-Cloud Bridge for Microsoft Azure* service is a Python microservice used for connecting the edge devices to Azure ecosystem using Azure IoT Edge Runtime and Azure Blob Storage Runtime. It runs as an Azure IoT Edge module and uses Azure IoT Edge SDK to communicate with Azure IoT Edge Runtime and IoT Hub.

 

Architecture Diagram
Figure 1: Architecture Diagram

 

 

The microservice uses an MQTT client or OEI Message Bus library to connect to the available message bus and subscribe to the channel, both specified in the mosquitto.json for MQTT and the config.json file for OEI Message Bus. It can subscribe to the publishers like Edge Video Analytics Microservice and OEI Video Analytics microservice. At runtime, the microservice automatically determines the availability of OEI Message Bus or MQTT Bus. Once the microservice is connected to the publishers, it waits for the messages. Once the microservice receives the data on the subscribed topics, it validates the message with a schema in the schema.json file. Once validated, the incoming messages are pushed to Azure IoT Edge runtime via the Azure IoT Edge SDK. The Azure IoT Edge Runtime connects to Azure IoT. The flow for the microservice is described below.

 

Flow Diagram
Figure 2: Flow Diagram

 

 


Get Started

Step 1: Configure your Environment

The Edge-to-Cloud Bridge for Microsoft Azure* service runs as an Azure IoT Edge Module. Before you use the service, you must do some setup steps to configure your Azure Cloud account and your development or edge deployment system.

Run the following tutorials to set up your development system:

Once the development system is set up, you will deploy the Microservices to Azure IoT Runtime.

 

NOTE: In the tutorials above, you will set the names for your Azure IoT Hub and Azure IoT Device. Save these for later, as they will be important during the deployment.

 

Step 2: Install the Microservice

Select Configure & Download to download the microservice and then follow the steps below to install it.  

Configure & Download

NOTE: The Docker image for the microservice Edge-to-Cloud Bridge for Microsoft Azure* service can be downloaded directly from Docker Hub using the following command:
docker pull intel/edge_to_azure_bridge:latest
For using the microservice with EII, pull all other required EII Docker images from the Docker Hub at https://hub.docker.com/u/openedgeinsights

 

  1. Open a new terminal, go to the downloaded folder and unzip the downloaded package:  
    unzip edge_to_cloud_bridge_for_microsoft_azure_service.zip

     

  2.  Go to the edge_to_cloud_bridge_for_microsoft_azure_service/ directory:
    cd edge_to_cloud_bridge_for_microsoft_azure_service
      
  3. Change permission of the executable edgesoftware file: 
    chmod 755 edgesoftware
     
  4. Run the command below to install the microservice: 
    sudo ./edgesoftware install

     

  5. During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download. 

     

    Screenshot of product key
    Figure 3: Product Key


     

  6. When the installation is complete, you see the message “Installation of package complete” and the installation status for each module. 

     

    Screenshot of installation complete
    Figure 4: Installation Complete

     

 

Step 3: Run Edge-to-Cloud Bridge for Microsoft Azure* Service

 

NOTE:
If you downloaded and installed the software package as mentioned in Step 2 above, you installed the Edge_to_Azure_Bridge_Resources folder in addition to the edge-to-azure-bridge container image. The folder contains configuration files required for this step.
If you pulled the edge-to-azure-bridge container image from Docker Hub, you must click on Configure & Download and then go through Customize Download options to select and download Edge to Azure Bridge Resources.

 

Edge-to-Cloud Bridge for Microsoft Azure* service is deployed as an Azure IoT Edge Module. The microservice connects to the MQTT broker and listens to the messages published to the topics defined in the configuration file (mosquitto.json). Once the microservice receives a message on the subscribed topics, it validates the message with the schema (schema.json) and sends it to Azure IOT Hub. The configuration files are mounted to the service during runtime via the Azure IoT Hub deployment manifest. To change the topics and schema, refer to the Optional Step below.

To deploy Edge-to-Cloud Bridge for Microsoft Azure service, follow the below steps:

  1. Update the Deployment manifest file (Edge_to_Azure_Bridge_Resources/config/deployment.template.json) with the absolute path to the Edge-to-Cloud Bridge for Microsoft Azure service config files. Replace the placeholder tag in the configuration with the absolute path to the Resources folder. 
    1. Navigate to the Edge_to_Azure_Bridge_Resources folder:
      cd <Path to EDGE_TO_AZURE_BRIDGE_RESOURCES Folder> 
    2. Run the following command that will replace the string EDGE_TO_AZURE_BRIDGE_RESOURCES with the absolute path to the config folder: 
      sed -i "s^EDGE_TO_AZURE_BRIDGE_RESOURCES^$PWD^g" config/deployment.template.json 
  2. Deploy the manifest file using the command below:
    az iot edge set-modules -n <azure-iot-hub-name> -d <azure-iot-edge-device-name> -k config/deployment.template.json

    Replace <azure-iot-hub-name> and <azure-iot-edge-device-name> with the names created in step 1.

    NOTE: This command may open the Azure login page in your web browser.

  3. Check the deployment status with the command:
    sudo iotedge list

    The deployment manifest deploys MQTTBroker and EdgeToAzureBridge as Azure IoT Edge Modules.
    You will see output similar to:
    Screenshot of Deployment Status
     
  4. Run the following command to monitor the data being sent to Azure IOT Hub:
    az iot hub monitor-events --output table --device-id <azure-iot-edge-device-name> --hub-name <azure-iot-hub-name> 
    Replace <azure-iot-hub-name> and <azure-iot-edge-device-name> with the names created in step 1.
     
  5. Now, you can create messages in the schema format provided in /Edge_to_Azure_Bridge_Resources/schema.json, publish them to the MQTT broker, and the Edge-to-Azure Bridge service will send it to the Azure IoT Hub.
    Use one of the following methods to publish data to the MQTT Broker: 
    1. Use any OpenSource MQTT Client such as MQTTExplorer. Install the MQTT client by following their instructions, run the client and send a message.
      A sample message with the default schema provided in the config folder supports the messages in the format below: 
      {
      "objects":["Person", "car", "bike"],
      "Description": "Objects Detected",
      "Location":"SB_FF_06",
      "Temperature":30
      } 

      Here's a screenshot for MQTT Explorer:


      Screenshot of Monitor Data

    2. From your code, use any MQTT client SDK, for example: Python or JavaScript.
    3. Use a microservice, such as the Edge Video Analytics microservice: Follow Tutorial 1 in this guide to deploy the Edge Video Analytics Microservice as an Azure IoT Edge Module and send analytics data to IoT Hub.

 

(Optional) Changing Topics and Message Schema for the Microservice

Edge-to-Cloud Bridge for Microsoft Azure* can be configured to listen to the messages from multiple topics. To change the topics to which the microservice should subscribe for listening to the messages, update the mosquitto.json file with the list of topics.

All the messages should follow the schema provided in schema.json. To change the message schema, update the schema.json file.

Follow the steps below to change topics and message schema:

 

  1. Let's assume an application publishes to a topic named camera1. We can update the configuration file (mosquitto.json) as below:
    {
        "topics": ["camera1"],
        "BrokerHost": "MQTTBroker",
        "BrokerPort": 1883
    } 
  2. Let’s also assume the application publishes location and people count to the topic as below: 
    {
      "location": "SB_2_1",
      "people_count": 3
    } 

     

  3. Update the schema (schema.json) as below to validate the messages coming to the service:
    {
        "$schema": "https://json-schema.org/draft/2020-12/schema",
        "properties": {
          "location": {
            "description": "Location of the camera",
            "type": "string"
          },
          "people_count": {
            "description": "No of people",
            "type": "integer"
          }
        },
        "required": [ "location", "people_count"]
      } 


    NOTE: Learn more about defining the schema at this link: JSON Schema.

  4. To update the service with these changes, restart the service with the command:

    sudo iotedge restart EdgeToAzureBridge
    
  5. Monitor Azure IoT Hub Data with the command:

    az iot hub monitor-events --output table --device-id <Device ID> --hub-name <IoT HUB name>
    

     

  6. Send the message with the new schema to the topic camera1 and verify:

    Screenshot of Verify Data

     

 


Tutorial

 

Tutorial 1: Run Edge-to-Cloud Bridge for Microsoft Azure Service with Edge Video Analytics Microservice

The steps below showcase running the Edge-to-Cloud Bridge for Microsoft Azure* service along with the Edge Video Analytics Microservice. Both microservices will be deployed as Azure IoT Edge Modules. The Edge Video Analytics Microservice executes a video analytics pipeline and publishes the inference results to the MQTT message broker. It provides RESTful APIs to discover, start, stop, customize, and monitor pipeline execution. The Edge-to-Cloud Bridge microservice connects to the MQTT broker and sends the data received from Edge Video Analytics Microservice to Azure IoT Hub.

  1. Download microservices: Use the Customize Download option and select the Edge Video Analytics along with Run Edge-to-Cloud Bridge for Microsoft Azure service while downloading the package. 

    Screenshot of selecting EVAM to download
     
  2. Install: Follow the procedure in Step 2 above, this will install both microservices. 
     
  3. Download models: The Edge Video Analytics Microservice requires deep learning models to run the video analytics pipeline.
    1. Go to the working directory created after installing the package and add file permissions to run the scripts: 
      cd Edge_Video_Analytics_Resources/
      sudo chmod +x docker/run.sh
      sudo chmod +x tools/model_downloader/model_downloader.sh 
    2. Download the required models from Open Model Zoo in the working directory by running the following command: 
      sudo ./tools/model_downloader/model_downloader.sh --model-list models_list/models.list.yml


      View the list of models by opening the models_list/models.list.yml file.
       

    3. Check that the download is successful by browsing to the models directory: 

      Screenshot of Model Directory

       

  4. Update Edge-to-Cloud Bridge microservice configurations. 

    1. Edge Video Analytics Microservice pipeline requests will publish to vaserving topic. Update the configuration file as below:

      {
          "topics": ["vaserving"],
          "BrokerHost": "MQTTBroker",
          "BrokerPort": 1883
      } 

       

    2. Update the schema (schema.json) as below to validate the messages coming to the Edge Video Analytics Microservice: 

      {
          "properties": {
            "objects": {
              "description": "The unique identifier for a inference metadata.",
              "type": "array"
            }
          },
          "required": [ "objects" ]
        } 

       

  5. Update the Deployment manifest file (config/deployment_evam.template.json).

    1. From the Edge_to_Azure_Bridge_Resources folder, run the command below to update the deployment manifest with the resources path that is used in Edge Video Analytics Microservice: 

      sed -i "s^EVAM_RESOURCES^<Path to EVAM Resources>^g" config/deployment_evam.template.json 


      For example: 

      sed -i "s^EVAM_ RESOURCES^/home/intel/video_analytics/Video_Analytics_2021.4.2/Edge_Video_Analytics_Resources^g" config/deployment_evam.template.json
      


       

    2. From the Edge_to_Azure_Bridge_Resources folder, run the command below to update the deployment template with the absolute path to the configuration files:

      sed -i "s^EDGE_TO_AZURE_BRIDGE_RESOURCES^$PWD^g" config/deployment_evam.template.json 
      

       

  6. Deploy the configured deployment manifest file using the command: 

    az iot edge set-modules -n <azure-iot-hub-name> -d <azure-iot-edge-device-name> -k config/deployment_evam.template.json

     

  7. Check the deployment status with the command: 

    sudo iotedge list 
    


    You will see output similar to: 

    Screenshot of Deployment with EVAM status
     

  8. Monitor Azure IOT Hub Data using the command:

    az iot hub monitor-events --output table --device-id <Device ID> --hub-name <IoT HUB name> 
  9. Start a new pipeline and monitor Azure IoT Hub data. (Before running the pipeline, update the <SYSTEM_IP_ADDRESS> in line 11 below.)

    curl --location -X POST 'http://localhost:8080/pipelines/object_detection/person_vehicle_bike' \
    --header 'Content-Type: application/json' \
    --data-raw '{
      "source": {
          "uri": "https://github.com/intel-iot-devkit/sample-videos/raw/master/person-bicycle-car-detection.mp4?raw=true",
          "type": "uri"
      },
      "destination": {
          "metadata": {
            "type": "mqtt",
            "host": "<SYSTEM_IP_ADDRESS>:1883",
            "topic": "vaserving"
          }
      }
    }'
    


     

  10. For PRC users, try the following command. (Before running the pipeline, update the <SYSTEM_IP_ADDRESS> in line 11 below.)

    curl --location -X POST 'http://localhost:8080/pipelines/object_detection/person_vehicle_bike' \
    --header 'Content-Type: application/json' \
    --data-raw '{
      "source": {
          "uri": "file:///app/resources/classroom.mp4",
          "type": "uri"
      },
      "destination": {
          "metadata": {
            "type": "mqtt",
            "host": "<SYSTEM_IP_ADDRESS>:1883",
            "topic": "vaserving"
          }
      }
    }'
     

     

  11. The logs being monitored will look similar to:

    Screenshot of Monitoring Data with EVAM

 

 


Summary and Next Steps

By following this guide, you learned how to use the Edge-to-Cloud Bridge for Microsoft Azure service to bridge between an edge device and the Microsoft Azure ecosystem. By running the tutorial, you also learned how to integrate it with other microservices such as Edge Video Analytics Microservice to build a video analytics application that runs on an edge device and Azure cloud.

As a next step, see the items in Learn More to continue your knowledge journey.   

 


Learn More 

To continue learning, see the following guides and software resources: 


Troubleshooting

  • Make sure that the downloaded package does not have spaces in the path.
    • Spaces in path will lead to incorrect deployment manifest created in the following step:
      sed -i "s^EDGE_TO_AZURE_BRIDGE_RESOURCES^$PWD^g" config/deployment.template.json
    • To overcome this issue, manually update the EDGE_TO_AZURE_BRIDGE_RESOURCES placeholder with the absolute path to the Edge to Azure Bridge Resources folder.
    • For example, the below screenshot denotes an incorrect deployment manifest with incorrect bind path. To solve it, use a backslash for each space in the name and then the space and update the path.

      Screenshot of error caused by spaces in filename
       

 

Support Forum 

If you're unable to resolve your issues, contact the Support Forum.  

 

†You are responsible for payment of all third-party charges, including payment for use of Microsoft Azure services. 

For the most up-to-date information on Microsoft® Azure products, see the Microsoft Azure website.