Multimodal Data Visualization Microservice

ID 737299
Updated 11/30/2022
Version 1.0
Public

author-image

By

Overview

Multimodal Data Visualization Microservice visualizes the video analytics and time series analytics data using the Grafana* dashboard. The visualization microservice receives video frames via Open Edge Insights (OEI) Message Bus, WebRTC* or HTTP protocols and displays it using the dashboard, streaming multiple videos at once in the same dashboard. It can also subscribe to OEI Message Bus to receive the time series analytics data on the Grafana dashboard.

Multimodal Data Visualization Microservice can be used independently with Edge Video Analytics Microservice, or with the Edge Insights for Industrial (EII) microservices ecosystem.

Select Configure & Download to download the microservice and the software listed below.

Configure & Download

Icon representing situational monitoring. It is a circle with a tilted triangle in the center. There is an exclamation point, circle, and check mark located on the outer circle.

  • Time to Complete: 15 minutes
  • Programming Language: Python* 3
  • Available Software:  Edge Video Analytics Microservice

 Target System Requirements

  • One of the following processors:
    • 6th to 11th generation Intel® Core™ processors
    • 1st to 3rd generation of Intel® Xeon® Scalable processors
    • Intel Atom® processor with Intel® Streaming SIMD Extensions 4.2 (Intel® SSE4.2) 
  • At least 8 GB RAM.
  • At least 64 GB hard drive.
  • An Internet connection.
  • Ubuntu* 20.04 LTS Kernel 5.4†

 

† Use Kernel 5.8 for 11th generation Intel® Core™ processors.


How It Works 

The Multimodal Data Visualization Microservice consists of two components packaged as Docker container images:

  • Multimodal-data-visualization-streaming 
  • Multimodal-data-visualization 

The multimodal-data-visualization-streaming container receives videos frames via HTTP, WebRTC or OEI Message Bus and streams them to a web page. The multimodal-data-visualization container, which uses Grafana*, embeds the web page in Grafana dashboard using AJAX panel to visualize the video stream in the Dashboard. It can display multiple videos in the same Grafana dashboard. Also, in addition to the videos, it can display metrics related to video analytics and time-series analytics.

When the multimodal-data-visualization container starts, it loads a dashboard configuration in Grafana according to the virtualization microservice mode selected. The multimodal-data-visualization component uses Grafana service which exposes the Grafana web interface on port 3000.

The architecture is represented by a block diagram. The top block has a header of Multimodal-data-visualization and it includes a box with Grafana*, Video Streaming and Time Series Data. The box below this has a header of Multimodal-data-visualization-streaming. It contains a box on the left with Nginx, EII mode, Gunicorn Server, EII Config Manager, and OEI Message Bus Library. On the right the box contains EVA Mode, WebRTC Client, and Signaling Server.
Figure 1: Architecture Diagram

 

The Multimodal Data Visualization Microservice can be deployed in the following three modes using an environment variable: 

  • Standalone mode (uses HTTP to receive the video frames) 
  • EVA mode (uses WebRTC to receive the video frames) 
  • EII mode (uses OEI Message Bus to receive the video frames) 

In Standalone mode, the visualization microservices accepts video streams via HTTP and displays them on Grafana dashboard.  

In EVA mode, the WebRTC framework is used to get the processed video from a microservice such as Edge Video Analytics Microservice and streams it to a web page. Edge Video Analytics Microservices sends the video frames with the inference results such as object detection bounding boxes overlaid on them. In this mode, the visualization microservice also displays video analytics metrics such as pipeline status and performance obtained by sending a REST request to the video analytics microservice. 

In EII mode, the visualization microservice subscribes to the OEI Message Bus to receive the video frames and inference results and renders the video to a web page after overlaying the inference results, such as object detection bounding boxes on the video frames. In this mode, the visualization microservice also displays Time Series Dashboard for visualizing the timeseries data received from EII Message Bus from EII. 

The visualization microservice allows developer to specify the configurations.


 Get Started

Step 1: Configure your Environment

Make sure all the requirements are met with Ubuntu* 20.04 OS. 

 Step 2: Install the Microservice

Select Configure & Download to download the microservice and then follow the steps below to install it. 

Configure & Download

 

  1. Open a new terminal, go to the downloaded folder and unzip the downloaded package:  
    unzip multimodal_data_visualization.zip

     

  2.  Go to the multimodal_data_visualization/ directory:
    cd multimodal_data_visualization
      
  3. Change permission of the executable edgesoftware file: 
    chmod 755 edgesoftware
     
  4. Run the command below to install the microservice: 
    sudo ./edgesoftware install

     

  5. During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download. 

     

    Screenshot of terminal with text that says: “Please enter the Product Key. The Product Key is contained in the email you received from Intel confirming your download.” The Product Key is blocked out in the image, and the text after it says, “Starting the setup,” confirming the Product Key was entered successfully.
    Figure 2: Product Key


     

  6. When the installation is complete, you see the message “Installation of package complete” and the installation status for each module. 

     

    Screenshot of terminal with text that shows different modules are being installed, and then it says “Installation of package complete. Recommended to reboot system after installation.” A table lists all the modules, their IDs, and their status as “success.”
    Figure 3: Installation Complete

     

Step 3: Verify the Installation

Check for successful installation: 

  1. All the modules installed should show Success status.
  2. Below images should be available locally. (Check the images using command docker images).
    1. multimodal-data-visualization 
    2. multimodal-data-visualization-streaming 
  3. Path multimodal_data_visualization/Multimodal_Data_Visualization_Microservice_<Version No>/Multimodal_Data_Visualization_Resources/ is available for running the Multimodal Data Visualization Microservices. 
     
    Screenshot of terminal with text that shows different modules are being installed, and then it says “Installation of package complete. Recommended to reboot system after installation.” A table lists all the modules, their IDs, and their status as “success.”
    Figure 4: Successful Install Verified

     


Tutorials

Tutorial 1: Run Multimodal Data Visualization Microservice in Standalone mode  

 

  1. Navigate to Multimodal_Data_Visualization_Microservice_<Version No>/Multimodal_Data_Visualization_Resources/ directory. 

     

NOTE: By default, the Multimodal Data Visualization Microservice is configured in standalone mode. 

  1. Run the Docker Compose up command: docker-compose up 
  2. Go to http://localhost:3000 to access Grafana. 
  3. On the Home Dashboard page, click the Dashboards icon in the left corner. 
    1. Click the Manage Dashboards tab to view the list of all the preconfigured dashboards.
    2. Click on Sample Visualization Dashboard, It will render a sample video from Google* API, so the Grafana dashboard will be loaded with the video. 
      Screen depicting the sample visualization dashboard. It shows a car in a parking lot.
      Figure 5: Sample Visualization Dashboard


       

NOTE: To stream with a different video, replace the video url with the other video's url and click on the Stream button on the dashboard. 

Tutorial 2: Run Multimodal Data Visualization Microservice in EVA mode 

The steps below detail how to run the Multimodal Data Visualization Microservices along with the Edge Video Analytics Microservice.

The Edge Video Analytics Microservice executes a video analytics pipeline. It provides RESTful APIs to discover, start, stop, customize, and monitor pipeline execution. The Multimodal Data Visualization Microservices uses the pipeline processed data to show in a Grafana dashboard. 

Step 1: Configure Client Browser for EVA Mode 

Mozilla Firefox*: Assign settings by navigating to about:config

  1. Confirm media.peerconnection.enabled is set to true. This is assigned by default. 
  2. Confirm media.peerconnection.ice.obfuscate_host_addresses is set to false. This allows successful operation with http://localhost. 
  3. The first time navigating to the site, click the shield at the left of the address bar in your browser to turn off Enhanced Tracking Protection. 

Step 2: Download and Install the Package with Edge Video Analytics Microservice 

  1. Download microservices: Click Customize Download option and select Edge Video Analytics along with Run the Multimodal Data Visualization services when downloading the package. 
    Screen showing the Customize Download option in the download tool. The modules available to download are listed.
    Figure 6: Customize Download

     

  2. Install: Follow Get Started: Install the Microservice. This will install both microservices. 
    Screenshot of terminal with text that shows different modules are being installed, and then it says “Installation of package complete. Recommended to reboot system after installation.” A table lists all the modules, their IDs, and their status as “success.”
    Figure 7: Successful Installation

     

  3. Verify the installation: List the Docker images that are downloaded using the following command: docker images 
    Screenshot of terminal with text that shows different modules are being installed, and then it says “Installation of package complete. Recommended to reboot system after installation.” A table lists all the modules, their IDs, and their status as “success.”
    Figure 8: Docker Images Download

     

Step 3: Bring up the Containers 

  1. The Edge Video Analytics Microservice requires deep learning models to run the video analytics pipeline. Follow these steps to download the models:
    1. Go to the working directory created after installing the package and add file permissions to run the scripts. 
      cd Edge_Video_Analytics_Resources/ 
      sudo chmod +x docker/run.sh 
      sudo chmod +x tools/model_downloader/model_downloader.sh

       

    2. Download the required models from Open Model Zoo in the working directory by running the following command:  
      sudo ./tools/model_downloader/model_downloader.sh --model-list models_list/models.list.yml

       

    3. View the list of models by opening the models_list/models.list.yml file.  
    4. Check that the download is successful by browsing to the models directory. 
      Screen showing multiple folders in the Models directory
      Figure 9: Models Directory

       

  2. Navigate to Multimodal_Data_Visualization_Microservice_<Version No>/Multimodal_Data_Visualization_Resources directory:  
    cd ../Multimodal_Data_Visualization_Resources/

     

  3. Configure Host IP:
    export HOST_IP=xxx.xxx.xx.xxx

     

  4.  Run the docker-compose up command for EVA mode: 
    docker-compose -f docker-compose-eva.yml up

     

  5. Open a new terminal and check that the containers are running: 
    docker ps --format 'table{{.Image}}\t{{.Status}}\t{{.Names}}'
    Screen showing entering the command to see that containers are running
    Figure 10: Container Status

     

Step 4: Run Grafana in EVA Mode 

  1. Go to http://localhost:3000 to access Grafana. 
  2. On the Home Dashboard page, in the left corner, click the Manage Dashboards tab to view the list of all the preconfigured dashboards. 
  3. Select Pipeline Server Sample Dashboard to view the data from the Edge Video Analytics Microservice. 
  4. If there are no pipelines running, on the Grafana dashboard it will show “Currently there are no pipelines running.” 
    Screen showing "There is currently no pipeline running!"
    Figure 11: Pipeline Status

     

  5. Open a new terminal. 
  6. Send a REST request using the curl API to run the object detection pipeline: 
    curl localhost:8080/pipelines/object_detection/person_vehicle_bike -X POST -H \ 
    'Content-Type: application/json' -d \ 
    '{ 
        "source": { 
            "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true", 
            "type": "uri" 
        }, 
        "destination": { 
            "metadata": { 
                "type": "file", 
                "format": "json-lines", 
                "path": "/tmp/results.jsonl" 
            }, 
            "frame": { 
                "type": "webrtc", 
                "peer-id": "peer_test_1" 
            } 
        }, 
        “parameters”:{ 
            “detection-device”: “CPU” 
        } 
    }' 

    For PRC users, use the command below: 

    curl localhost:8080/pipelines/object_detection/person_vehicle_bike -X POST -H \ 
    'Content-Type: application/json' -d \ 
    '{ 
        "source": { 
            "uri": " https://ghproxy.com/github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true", 
            "type": "uri" 
        }, 
        "destination": { 
            "metadata": { 
                "type": "file", 
                "format": "json-lines", 
                "path": "/tmp/results.jsonl" 
            }, 
            "frame": { 
                "type": "webrtc", 
                "peer-id": "peer_test_1" 
            } 
        }, 
        “parameters”:{ 
            “detection-device”: “CPU” 
        } 
    }' 

     

NOTE: Multiple pipeline requests can be sent in parallel by providing a unique value for the Destination Peer ID (peer-id) for every request to launch a new pipeline. 

  1. Once the pipeline starts in running state, navigate to the Grafana dashboard to visualize the streaming. Click Refresh on the panel to get the video streaming for the running pipelines. 
    Screen of 4 video streams of an empty parking lot.
    Figure 12: Video Streaming

     

  2. Once the status for all the pipelines has changed to Completed, video streaming will be stopped as shown in the image below. 
    Screen with blank boxes where videos should be streaming
    Figure 13: Video Streaming Stopped

     

  3. Click Refresh again for any new pipeline request or to remove the completed pipelines from the panel. 
    Screen of three videos streaming. One is a car in a parking lot, one is a person with a bounding box in the same parking lot, and the third is the empty parking lot.
    Figure 14: Refreshed Video Screen

     

 

Tutorial 3: Install Multimodal Data Visualization Helm* Chart

NOTE: This tutorial assumes you have a Kubernetes cluster available.

In this tutorial, you will deploy the Multimodal Data Visualization Microservice on a Kubernetes cluster using Helm charts. The Helm chart packages Multimodal Data Visualization and Multimodal Data Visualization Streaming for a sample deployment.

This tutorial explains how to use the Multimodal Data Visualization Helm Chart in EVA Mode and in Standalone Mode. You must complete the Tutorial Prerequisites for both modes.

 

Tutorial Prerequisites

  1. Complete steps 1 and 2 in the Get Started section. To download the Helm chart, you must select the Customize Download option during the Configure & Download step. 
    When the installation is complete, you see the message "Installation of package complete" and the installation status for each module. 
     
    Screenshot of terminal with text that shows different modules are being installed, and then it says “Installation of package complete. Recommended to reboot system after installation.” A table lists all the modules, their IDs, and their status as “success.” 
    Figure 15: Successful Installation including Helm Charts

     
  2. Add the system's host IP address in values.yaml file of EVAM Helm chart folder.
  3. Add the system's host IP address in values.yaml file of multimodal data visualization Helm chart folder.

 

Multimodal Data Visualization Helm Chart in EVA Mode

  1. Enter the command: 
    helm install multimodal-data-visualization ./multimodal-data-visualization-chart-3.0/


    You will see output similar to: 

    A console window showing system output after running the “helm install” command. The system displays status messages and ends with a list of deployed modules.
    Figure 16: Run Helm Chart


     

  2. Get the IP addresses of the pods running in the cluster with the command:

    kubectl get pods -o wide


    You will see output similar to: 

    A console window showing system output after running the “kubectl get pods” command. The system displays a list of all the pods and their status. The expected status is “Running”.
    Figure 17: Pod Status after Helm Chart Installation


     

  3. Go to http://<HOST_IP>:30000 to access Multimodal-Data-Visualization.
    NOTE: Connecting to Grafana may take some time (up to 2 minutes) based on the setup.

  4. On the Home Dashboard page, on the left corner, click the Dashboards icon.

  5. Click the Manage Dashboards tab, to view the list of all the preconfigured dashboards.

  6. Select Video Analytics Dashboard to view the data from the Edge Video Analytics Microservice.

    A web app dashboard showing the Video Analytics Dashboard before the pipeline is triggered.
    Figure 18: Video Analytics Dashboard before pipeline is triggered


     

  7. Send the pipeline request command to trigger a pipeline for streaming and FPS results on the dashboard using the command:

    curl localhost:30007/pipelines/object_detection/person_vehicle_bike -X POST -H \
    'Content-Type: application/json' -d \
    '{
        "source": {
            "uri": "https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true",
            "type": "uri"
        },
        "destination": {
            "metadata": {
                "type": "file",
                "format": "json-lines",
                "path": "/tmp/results.jsonl"
            },
            "frame": {
                "type": "webrtc",
                "peer-id": "peer_test_1"
            } 
        }, 
        “parameters”:{ 
            “detection-device”: “CPU” 
        } 
    
    }' 


    NOTE: For PRC users, change the pipeline request URI to: https://ghproxy.com/https://github.com/intel-iot-devkit/sample-videos/blob/master/person-bicycle-car-detection.mp4?raw=true.
     

  8. Click the Refresh button on the webRTC Stream panel to get the streaming for running pipelines.

    A web app dashboard showing the Video Analytics Dashboard showing pipeline data in bar charts and graph form in the top half. The lower half shows 2 camera outputs.
    Figure 19: Video Analytics Dashboard showing streaming for the running pipelines



     

Multimodal Data Visualization Helm Chart in Standalone Mode 

 

  1. Add the system's host IP address in values.yaml file, which is located in the multimodal data visualization helm chart folder.
  2. Set the environment variable MODE to empty in values.yaml file, which is located in the Multimodal-Data-Visualization-<version> folder, where <version> is the software version downloaded.
    MODE:

     

  3. Install the Helm chart with the command:

    helm install multimodal-data-visualization ./multimodal-data-visualization-chart-3.0/

     

  4. Go to http://<HOST_IP>:30000 to access Multimodal-Data-Visualization. 
    NOTE: Connecting to Grafana may take some time (up to 2 minutes) based on the setup. 
     

  5. To view the sample video streaming on the Grafana dashboard, select Sample Visualization Dashboard.
     

    A web app dashboard showing the Sample Visualization Dashboard with 1 camera view showing a parking lot with 1 car and 1 person with a bicycle.
    Figure 20: Sample Visualization Dashboard

     

 

 


Summary and Next Steps

By following this guide, you learned how to visualize the video analytics and time series analytics data using the Grafana dashboard.

By running the tutorials, you also learned how to visualize the video in different modes and stream multiple videos in the same Grafana dashboard using Edge Video Analytics Microservice.

As a next step, see the items in Learn More to continue your knowledge journey.   

 


Learn More 

To continue learning, see the following guides and software resources: 


Troubleshooting

  • If streaming is not shown in Grafana, make sure to update the Firefox* setting in about:config -> media.peerconnection.ice.obfuscate_host_addresses to False.

    Screen showing "false" for the host address
     
  • If video is not streaming on the browser and shows an error about installing video codecs, then install the below package through the terminal and restart the browser.
     
    sudo apt-get install ffmpeg

    Screenshot showing video codecs error
     
  • If streaming is not visible on the browser for “RUNNING” state pipelines shown with FPS, then make sure HOST_IP is exported before running the containers.
     
    export HOST_IP=xxx.xxx.xx.xxx

 

Support Forum 

If you're unable to resolve your issues, contact the Support Forum.  

 

†You are responsible for payment of all third-party charges, including payment for use of Microsoft Azure services. 

For the most up-to-date information on Microsoft® Azure products, see the Microsoft Azure website.