Smart VR – Live Streaming of Immersive Media Reference Implementation

Version: 2.0.0   Published: 03/02/2021  

Last Updated: 10/20/2021

Overview

This Reference Implementation (RI) provides a smart VR solution leveraging edge platforms for media ingestion from multiple streams, media processing (360 view construction, 360 frames analytics), and interactive media distribution based on Field of View (FoV) request.

To run the reference implementation, you will need to first download and install the Intel® Smart Edge Open Developer Experience Kit.

Once you have installed the Intel® Smart Edge Open Developer Experience Kit, select Configure & Download to download the reference implementation and the software listed below.  

Configure & Download


Time to Complete

Programming
Language

Available Software

Approximately 1 hour

Python*, C++, Golang


Intel® Smart Edge Open version 21.09
Intel® Collaboration Suite for WebRTC (Intel® CS for WebRTC) with Open WebRTC Toolkit (OWT)
Intel® Advanced 360 Video Library
Intel® Distribution of OpenVINO™ toolkit
Scalable Video Technology for High Efficiency Video Coding (SVT-HEVC) 

 


Target System Requirements

Intel® Smart Edge Open Nodes

  • One of the following processors:
    • Intel® Xeon® scalable processor
  • At least 64 GB RAM
  • At least 128 logic cores (CPU)
  • At least 512 GB hard drive
  • An Internet connection
  • Ubuntu* 20.04 LTS Server
  • IP camera or pre-recorded video(s)

How It Works

This reference implementation is an Omnidirectional MediA Format (OMAF) immersive (VR) media application run on cloud infrastructure for end-to-end (E2E) live streaming and supports file content VR streaming. It runs on WebRTC stack with Linux* (Ubuntu* 20.04). The Intel® Smart Edge Open platform infrastructure is used to deploy the application pods.  

Includes the following main functionality systems:  

  • Media Ingestion Function (MIF): Captures stream from 4k camera, 360 stitching, high- and low-resolution encoding.  
  • Media Distribution Function (MDF): Performs tile re-packing and distributes the content.  
  • Media Control Function (MCF): Responsible for scheduling and managing network units, signaling transmission between MIF and MDF.  
  • Media Analytic Function (MAF): Enables the media analytic pipeline using the OpenVINO™ model. This media analytics acceleration building block allows CPU and GPU/VPU acceleration for frame analytics. 
  • Client (Linux/Android*): Does Supplemental Enhancement Information (SEI) parsing, decoding, low- and high-resolution tiles re-ordering, FoV, rendering, etc.

Intel® Smart Edge Open 21.09 supports on-premise cluster deployment with MIF, MAF and MDF on the same host. MCF control function is located on the cloud cluster to provide centralized control function and web UI. 

L3 switch will be supported in Real 360 Camera mode.

Smart VR - Live Streaming of Immersive Media RI
Figure 1: Architecture Diagram

 


Get Started

Prerequisites

Ensure that the following conditions are met properly for a smooth installation process.

  1. Hardware Requirements
    Make sure you have a fresh Ubuntu 20.04 LTS Server installation with the hardware specified in the Target System Requirements section.
  2. To run the reference implementation, you will need to first download and install the Intel® Smart Edge Open Developer Experience Kit.

Step 1: Install the Reference Implementation

Select Configure & Download to download the reference implementation and then follow the steps below to install it.  

Configure & Download

  1. Make sure that the Target System Requirements are met properly before proceeding further.  
    • For single-device mode, only one machine is needed. (Both controller and edge node will be on the same device.) 
    • NOTE: Multi-device mode is not supported in the current release. 
  2. Create a user account for Intel® Smart Edge Open: 
    adduser "smartedge-open"
    passwd "smartedge-open"
    echo "smartedge-open ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/smartedge-open 
  3. Open a new terminal as smartedge-open user:  
    su - smartedge-open
  4. Generate an SSH key under smartedge-open user:
    ssh-keygen
    ssh-copy-id smartedge-open@target_server_IP 
  5. Move the downloaded zip package to the /root folder: 
    mv <path-of-downloaded-directory>/Smart_VR_Live_Streaming_of_Immersive_Media.zip /home/smartedge-open
  6. Go to the /home/smartedge-open directory using the following command and unzip the RI:
    cd /home/smartedge-open 
    unzip Smart_VR_Live_Streaming_of_Immersive_Media.zip 
  7. Go to the Smart_VR_Live_Streaming_of_Immersive_Media/ directory: 
    cd Smart_VR_Live_Streaming_of_Immersive_Media /
  8. Change permission of the executable edgesoftware file: 
    chmod 755 edgesoftware 
  9. Run the command below to install the Reference Implementation:  
    ./edgesoftware install
  10. During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download.  
    NOTE: Installation logs are available at: /var/log/esb-cli/Immersive_Media_<version>/Immersive_Media/install.log
    Figure 2. Product Key

     

  11. When the installation is complete, you see the message Installation of package complete and the installation status for each module.  

    NOTE: This step may take about 5 minutes, depending on your network connection.
     

    Figure 3. Installation Complete

     

If Intel® Smart Edge Open is installed, running the following command should show output similar to the image below. All the pods should be either in the running or completed stage.  

kubectl get pod -A

 

Figure 4. Pod Status

 

Run the Application

1. Check the Smart_VR_Live_Streaming_of_Immersive_Media pod with the command:

kubectl get pod -n smartedge-apps

You will see output similar to: 

Figure 5. Application Status

2. In a browser, go to https://<SmartEdge-node-ip>:3004/?publish=false to verify the Smart_VR_Live_Streaming_of_Immersive_Media app result.

Follow the steps below to verify the certificate: 

  1. Click the link that says "Click this for testing certificate and refresh":
     
    Figure 6. Test Certificate

     

  2. Click the link that says "Proceed to... ":
     
    Figure 7. Test Certificate - Proceed

     

  3. You will see output similar to:
     
    Figure 8. Test Certificate - Complete

     

3. Reopen the page to visit https://<SmartEdge-node-ip>:3004/?publish=false

You will see output similar to:

Figure 9. Application Output

 

Select 61xxxxxxxxxxx-common in the video from drop-down, and then click the startAnalytics button.
Select 9D1Axxxxxxxxx-common in the subscribe video drop-down, and then click the subscribe button.

You will see output similar to:

Figure 10. Application Video Output

 

NOTE: When you finish viewing the video or someone reopens a page to check the video, click the stopAnalytics button to stop analytics.

 

Uninstall the Application

  1. List the installed RI module with the command:
    ./edgesoftware list

    You will see output similar to: 

    Figure 11. Edgesoftware List

     

  2. Uninstall the RI module with the command using the <module-id> from the previous step: 

    ./edgesoftware uninstall <module-id>

    You will see output similar to: 

    Figure 12. Uninstall RI

 


Summary and Next Steps

This reference implementation successfully demonstrates the following features: 

  • Video analytics pipeline based on Intel® Distribution of OpenVINO™ toolkit and DL Streamer. 
  • Media streams can be forwarded and analyzed through distributed pipeline. 
  • K8s* based deployment (single node, two pods, multi-containers) on Intel® Smart Edge Open 21.09. 
  • Enhanced Android* player receives and processes analytics metadata. 
  • Web UI for simulator.
  • For Developers and end users: 
    • Work on real time video stitching for 4k/8k after VR camera input.
    • Work on H.265 OMAF encoding format support, which supports high and low-definition mixing.
    • Work on FOV field of view support for dynamically switching to HD.
       

As a next step, try one of the following: 

  • Support the multi-clusters deployment. 
  • Media analytic pipeline qualities’ optimization.

Learn More

To continue learning, see the following guides and software resources:


Troubleshooting

Installation Confirmation

Check the log of im360-mif to see if the stream is added successfully: 

kubectl logs <im360-deployment-PodName> -n smartedge-apps -c im360-mif 

Uninstall Confirmation

Execute the command helm list to check whether there is im360-app or kubectl get pod -n smartedge-apps to check whether there are related pods.

Subscribed but Video is not displayed

If you have clicked the Subscribe button but no video is displayed, disable the firewall.

## Please turn off the firewall of your target server
sudo ufw disable

 

Video not displayed in browser

You may need to update the browser version. Refer to the table below for supported browsers on different operating systems. 

  Windows* Ubuntu* macOS*
Chrome* 89 Supported Supported Supported
FIrefox* 87 Supported Supported Supported
Safari* 14.0.1     Supported
Microsoft Edge* 89 Supported    

Network Policy

The Smart_VR_Live_Streaming_of_Immersive_Media application uses the host network, which will use the host's port 3001,3004, 8080, 5001 to 5006. (Port 3004 is for the view side; the port 8080 is for the testing certification.)

 

Support Forum 

If you're unable to resolve your issues, contact the Support Forum.  

To attach the installation logs with your issue, execute the command below to consolidate a list of the log files in tar.gz compressed format, e.g., Smart_VR_Live_Streaming_of_Immersive_Media.tar.gz

tar -czvf Smart_VR_Live_Streaming_of_Immersive_Media.tar.gz /var/log/esb-cli/Immersive_Media _2.0.0/Immersive_Media / /var/log/esb-cli/Smart_Edge_Open_Developer_Experience_Kits_1.0.0/Smart_Edge_Open_Developer_Experience_Kits/ 

Product and Performance Information

1

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.