Smart VR – Live Streaming of Immersive Media Reference Implementation

ID 658731
Updated 3/21/2022
Version 3.0.0
Public

author-image

By

Overview

This Reference Implementation (RI) provides a smart VR solution leveraging edge platforms for media ingestion from multiple streams, media processing (360 view construction, 360 frames analytics), and interactive media distribution based on Field of View (FoV) request.

To run the reference implementation, you will need to first download and install the Intel® Smart Edge Open Developer Experience Kit.

Once you have installed the Intel® Smart Edge Open Developer Experience Kit, select Configure & Download to download the reference implementation and the software listed below.  

Configure & Download

Smart VR - Live Streaming - Image of running the RI

  • Time to Complete: Approximately 1 hour
  • Programming Language: Python*, C++, Golang
  • Available Software: 
    • Intel® Smart Edge Open version 21.12
    • Intel® Collaboration Suite for WebRTC (Intel® CS for WebRTC) with Open WebRTC Toolkit (OWT)
    • Intel® Advanced 360 Video Library
    • Intel® Distribution of OpenVINO™ toolkit
    • Scalable Video Technology for High Efficiency Video Coding (SVT-HEVC) 

Target System Requirements

Intel® Smart Edge Open Nodes

  • One of the following processors:
    • Intel® Xeon® scalable processor
    • Intel® Xeon® processor D
  • At least 64 GB RAM
  • At least 128 logic cores (CPU)
  • At least 512 GB hard drive
  • An Internet connection
  • Ubuntu* 20.04 LTS Server
  • IP camera or pre-recorded video(s)

How It Works

This reference implementation is an Omnidirectional MediA Format (OMAF) immersive (VR) media application run on cloud infrastructure for end-to-end (E2E) live streaming and supports file content VR streaming. It runs on WebRTC stack with Linux* (Ubuntu* 20.04). The Intel® Smart Edge Open platform infrastructure is used to deploy the application pods.  

Includes the following main functionality systems:  

  • Media Ingestion Function (MIF): Captures stream from 4k camera, 360 stitching, high- and low-resolution encoding.  
  • Media Distribution Function (MDF): Performs tile re-packing and distributes the content.  
  • Media Control Function (MCF): Responsible for scheduling and managing network units, signaling transmission between MIF and MDF.  
  • Media Analytic Function (MAF): Enables the media analytic pipeline using the OpenVINO™ model. This media analytics acceleration building block allows CPU and GPU/VPU acceleration for frame analytics. 
  • Client (Linux/Android*): Does Supplemental Enhancement Information (SEI) parsing, decoding, low- and high-resolution tiles re-ordering, FoV, rendering, etc.

Intel® Smart Edge Open 21.12 or above supports on-premise cluster deployment with MIF, MAF and MDF on the same host. MCF control function is located on the cloud cluster to provide centralized control function and web UI. 

L3 switch will be supported in Real 360 Camera mode.

Smart VR - Live Streaming of Immersive Media RI

Figure 1: Architecture Diagram

 


Get Started

Prerequisites

Ensure that the following conditions are met properly for a smooth installation process.

  1. Hardware Requirements
    Make sure you have a fresh Ubuntu 20.04 LTS Server installation with the hardware specified in the Target System Requirements section.
  2. To run the reference implementation, you will need to first download and install the Intel® Smart Edge Open Developer Experience Kit.

Step 1: Install the Reference Implementation

Select Configure & Download to download the reference implementation and then follow the steps below to install it.  

Configure & Download

  1. Make sure that the Target System Requirements are met properly before proceeding further.  
    • For single-device mode, only one machine is needed. (Both controller and edge node will be on the same device.) 
    • NOTE: Multi-device mode is not supported in the current release. 
  2. Create a user account for Intel® Smart Edge Open: 
    sudo adduser "smartedge-open"
    sudo passwd "smartedge-open"
    echo "smartedge-open ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/smartedge-open 
  3. Open a new terminal as smartedge-open user:  
    su - smartedge-open
  4. Generate an SSH key under smartedge-open user:
    ssh-keygen
    ssh-copy-id smartedge-open@target_server_IP 
  5. Move the downloaded zip package to the /home/smartedge-open folder: 
    sudo mv <path-of-downloaded-directory>/Smart_VR_Live_Streaming_of_Immersive_Media.zip /home/smartedge-open
  6. Go to the /home/smartedge-open directory using the following command and unzip the RI:
    cd /home/smartedge-open 
    unzip Smart_VR_Live_Streaming_of_Immersive_Media.zip 
  7. Go to the Smart_VR_Live_Streaming_of_Immersive_Media/ directory: 
    cd Smart_VR_Live_Streaming_of_Immersive_Media /
  8. Change permission of the executable edgesoftware file: 
    chmod 755 edgesoftware 
  9. Run the command below to install the Reference Implementation:  
    ./edgesoftware install
  10. During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download.  
    NOTE: Installation logs are available at: /var/log/esb-cli/Immersive_Media_<version>/Immersive_Media/install.logScreenshot of product key

    Figure 2. Product Key

     

  11. When the installation is complete, you see the message Installation of package complete and the installation status for each module.  

    NOTE: This step may take about 5 minutes, depending on your network connection.
     

    Screenshot of install success

    Figure 3. Installation Complete

     

If Intel® Smart Edge Open is installed, running the following command should show output similar to the image below. All the pods should be either in the running or completed stage.  

kubectl get pod -A

 

Screenshot of pod status

Figure 4. Pod Status

 

Run the Application

1. Check the Smart_VR_Live_Streaming_of_Immersive_Media pod with the command:

kubectl get pod -n smartedge-apps

You will see output similar to: 

Screenshot of application status

Figure 5. Application Status

2. In a browser, go to https://<SmartEdge-node-ip>:31004/ to verify the Smart_VR_Live_Streaming_of_Immersive_Media app result.

Follow the steps below to verify the certificate: 

1. Select Click this for testing certificate and refresh.
 Screenshot of test certificate

Figure 6. Test Certificate

 

2. Select Proceed to... 
 Screenshot of test certificate - proceed

Figure 7. Test Certificate - Proceed

 

3. You will see output similar to:
 Screenshot of test certificate complete

Figure 8. Test Certificate - Complete

4. Visit URL to allow camera and microphone of the device: https://\<SmartEdge-node-ip\>:31004/.

Screenshot of Allowing or Blocking Mic and camera

Figure 9. Allow camera and microphone

5. Close the tab and open a new one with https://<SmartEdge-node-ip>:31004/?publish=false.

6. Select Analytics Demo.

Screenshot of application output

Figure 10. Click Analytics Demo

You will see the analytic web:

Screenshot of Analytics Demo

Figure 11. Analytic Web

 

  • Select 62xxxxxxxxxxx-common in the video from drop-down, and then click the startAnalytics button.
  • Select 9D1Axxxxxxxxx-common in the subscribe video drop-down, and then click the subscribe button.

You will see the final result:

Screenshot of application video output

Figure 12. Application Video Output

 

NOTE: When you finish viewing the video or someone reopens a page to check the video, click the stopAnalytics button to stop analytics.

 

Uninstall the Application

  1. List the installed RI module with the command:
    ./edgesoftware list

    You will see output similar to: 

    Screenshot of edgesoftware list

    Figure 11. Edgesoftware List

     

  2. Uninstall the RI module using the <module-id> from the previous step in the following command: 

    ./edgesoftware uninstall <module-id>

    You will see output similar to: 

    Screenshot of uninstall ri

    Figure 12. Uninstall RI

 

  1. Run the commands below to make sure the block-all-ingres network policy is deleted after uninstalling
    $ kubectl get netpol
    NAME                POD-SELECTOR   AGE
    block-all-ingress   <none>         58d

Post deletion only the below network policy should be present.

$ kubectl get netpol -n smartedge-apps
NAME                         POD-SELECTOR   AGE
deny-from-other-namespaces   <none>         18h

 

Node Feature Discovery (NFD) Feature

Immersive Media uses Intel® Distribution of OpenVINO™ toolkit which is optimized for Intel® processors that support special instructions like AVX512VNNI for optimized performance. The deployment of this application will require the node with this feature supported on the node along with Ubuntu 20.04 OS. This NFD features ensures to deploy the application on the node supported with these features. NFD is installed by Intel® Smart Edge Open Developer Experience Kit and running as two pods on Intel® Smart Edge Open.

$ kubectl get pods -A | grep node-feature-discovery

smartedge-system         nfd-release-node-feature-discovery-master-7f4ffd8f6b-ckwrx   1/1     Running   6 (25h ago)      58d
smartedge-system         nfd-release-node-feature-discovery-worker-m69h7              1/1     Running   64 (25h ago)     58d

 

Immersive Media pod scheduled and running successfully on Intel® Smart Edge Open node based on hardware capabilities of Intel® Xeon® Scalable server.

$ kubectl get pods -n smartedge-apps | grep im360

im360-mcf-85744f5b5d-nnrk6    1/1     Running   0          5m26s
im360-mdf-7d8455677f-fpkjl    1/1     Running   0          5m26s
im360-mif-79674f8978-m52bf    1/1     Running   0          5m26s
im360-mqtt-6b6cdd7cfc-8dmcs   1/1     Running   0          5m26s

 

The following output shows a description of im360 pod, which shows that it is running successfully with the NFD feature.

$ kubectl describe pods im360-mcf-85744f5b5d-nnrk6 -n smartedge-apps
... ...
Node-Selectors:              feature.node.kubernetes.io/cpu-cpuid.AVX2=true
                             feature.node.kubernetes.io/cpu-cpuid.AVX512VNNI=true
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  32m   default-scheduler  Successfully assigned smartedge-apps/im360-mcf-c9587755c-9dkds to ubuntu252
  Normal  Pulled     32m   kubelet            Container image "openvisualcloud/xeon-ubuntu2004-smartedge-im360:2.9.3" already present on machine
  Normal  Created    32m   kubelet            Created container im360-mcf
  Normal  Started    32m   kubelet            Started container im360-mcf

 

 

Public Helm Registry for Helm Charts

Installation of Immersive Media Reference Implementation Intel® Smart Edge Open is accomplished using Helm charts. Earlier Helm charts used to be a part of Reference Implementation installation package. Now a global Helm repo is issued, so that Reference Implementation helm charts can be accessible from private and public network. This will speed up and ease the process of introducing updates and their integration with the Reference Implementation.

 


Summary and Next Steps

This reference implementation successfully demonstrates the following features: 

  • Video analytics pipeline based on Intel® Distribution of OpenVINO™ toolkit and Intel® Deep Learning Streamer (Intel® DL Streamer). 
  • Media streams can be forwarded and analyzed through distributed pipeline. 
  • K8s* based deployment (single node, two pods, multi-containers) on Intel® Smart Edge Open 21.12. 
  • Enhanced Android* player receives and processes analytics metadata. 
  • Web UI for simulator.
  • For Developers and end users: 
    • Work on real time video stitching for 4k/8k after VR camera input.
    • Work on H.265 OMAF encoding format support, which supports high and low-definition mixing.
    • Work on FOV field of view support for dynamically switching to HD.
       

As a next step, try one of the following: 

  • Support the multi-clusters deployment. 
  • Media analytic pipeline qualities’ optimization.

Learn More

To continue learning, see the following guides and software resources:


Troubleshooting

Installation Confirmation

Check the log of im360-mif to see if the stream is added successfully: 

kubectl logs <im360-deployment-PodName> -n smartedge-apps -c im360-mif 

Uninstall Confirmation

Execute the command helm list to check whether there is im360-app or kubectl get pod -n smartedge-apps to check whether there are related pods.

Subscribed but Video is not displayed

If you have clicked the Subscribe button but no video is displayed, disable the firewall.

## Please turn off the firewall of your target server
sudo ufw disable

 

Video not displayed in browser

If you cannot see the video in the browser, you may need to update the browser version. Refer to the table below for supported browsers on different operating systems. 

  Windows* Ubuntu* macOS*
Chrome* 89 Supported Supported Supported
FIrefox* 87 Supported Supported Supported
Safari* 14.0.1     Supported
Microsoft Edge* 89 Supported    

 

You must grant the necessary permission by selecting Continue allowing <url> to access your camera and microphone.

Screenshot of granting permissions

Internal Server Error

If there is a error in the analytics id field, Internal Server Error, complete the steps as shown in the following screen:

Screenshot of Internal Server Error

Make sure there is only one xxx-common item and another three common items.

 

Network Policy

The Smart_VR_Live_Streaming_of_Immersive_Media application uses the host network, which will use the host's port 31004, 31080. (Port 31004 is for the view side; the port 31080 is for the testing certification.)

 

Support Forum 

If you're unable to resolve your issues, contact the Support Forum.  

To attach the installation logs with your issue, execute the command below to consolidate a list of the log files in tar.gz compressed format, e.g., Smart_VR_Live_Streaming_of_Immersive_Media.tar.gz

tar -czvf Smart_VR_Live_Streaming_of_Immersive_Media.tar.gz /var/log/esb-cli/Immersive_Media_3.0.0/immersive_media/