This Reference Implementation (RI) provides a smart VR solution leveraging edge platforms for media ingestion from multiple streams, media processing (360 view construction, 360 frames analytics), and interactive media distribution based on Field of View (FoV) request.
To run the reference implementation, you will need to first download and install the Intel® Smart Edge Open Developer Experience Kit.
Once you have installed the Intel® Smart Edge Open Developer Experience Kit, select Configure & Download to download the reference implementation and the software listed below.
- Time to Complete: Approximately 1 hour
- Programming Language: Python*, C++, Golang
- Available Software:
- Intel® Smart Edge Open version 21.12
- Intel® Collaboration Suite for WebRTC (Intel® CS for WebRTC) with Open WebRTC Toolkit (OWT)
- Intel® Advanced 360 Video Library
- Intel® Distribution of OpenVINO™ toolkit
- Scalable Video Technology for High Efficiency Video Coding (SVT-HEVC)
Intel® Smart Edge Open Nodes
- One of the following processors:
- Intel® Xeon® scalable processor
- Intel® Xeon® processor D
- At least 64 GB RAM
- At least 128 logic cores (CPU)
- At least 512 GB hard drive
- An Internet connection
- Ubuntu* 20.04 LTS Server
- IP camera or pre-recorded video(s)
How It Works
This reference implementation is an Omnidirectional MediA Format (OMAF) immersive (VR) media application run on cloud infrastructure for end-to-end (E2E) live streaming and supports file content VR streaming. It runs on WebRTC stack with Linux* (Ubuntu* 20.04). The Intel® Smart Edge Open platform infrastructure is used to deploy the application pods.
Includes the following main functionality systems:
- Media Ingestion Function (MIF): Captures stream from 4k camera, 360 stitching, high- and low-resolution encoding.
- Media Distribution Function (MDF): Performs tile re-packing and distributes the content.
- Media Control Function (MCF): Responsible for scheduling and managing network units, signaling transmission between MIF and MDF.
- Media Analytic Function (MAF): Enables the media analytic pipeline using the OpenVINO™ model. This media analytics acceleration building block allows CPU and GPU/VPU acceleration for frame analytics.
- Client (Linux/Android*): Does Supplemental Enhancement Information (SEI) parsing, decoding, low- and high-resolution tiles re-ordering, FoV, rendering, etc.
Intel® Smart Edge Open 21.12 or above supports on-premise cluster deployment with MIF, MAF and MDF on the same host. MCF control function is located on the cloud cluster to provide centralized control function and web UI.
L3 switch will be supported in Real 360 Camera mode.
Figure 1: Architecture Diagram
Ensure that the following conditions are met properly for a smooth installation process.
- Hardware Requirements
Make sure you have a fresh Ubuntu 20.04 LTS Server installation with the hardware specified in the Target System Requirements section.
- To run the reference implementation, you will need to first download and install the Intel® Smart Edge Open Developer Experience Kit.
Step 1: Install the Reference Implementation
Select Configure & Download to download the reference implementation and then follow the steps below to install it.
- Make sure that the Target System Requirements are met properly before proceeding further.
- For single-device mode, only one machine is needed. (Both controller and edge node will be on the same device.)
- NOTE: Multi-device mode is not supported in the current release.
- Create a user account for Intel® Smart Edge Open:
sudo adduser "smartedge-open" sudo passwd "smartedge-open" echo "smartedge-open ALL=(ALL) NOPASSWD: ALL" | sudo tee /etc/sudoers.d/smartedge-open
- Open a new terminal as smartedge-open user:
su - smartedge-open
- Generate an SSH key under smartedge-open user:
ssh-keygen ssh-copy-id smartedge-open@target_server_IP
- Move the downloaded zip package to the /home/smartedge-open folder:
sudo mv <path-of-downloaded-directory>/Smart_VR_Live_Streaming_of_Immersive_Media.zip /home/smartedge-open
- Go to the /home/smartedge-open directory using the following command and unzip the RI:
cd /home/smartedge-open unzip Smart_VR_Live_Streaming_of_Immersive_Media.zip
- Go to the Smart_VR_Live_Streaming_of_Immersive_Media/ directory:
cd Smart_VR_Live_Streaming_of_Immersive_Media /
- Change permission of the executable edgesoftware file:
chmod 755 edgesoftware
- Run the command below to install the Reference Implementation:
- During the installation, you will be prompted for the Product Key. The Product Key is contained in the email you received from Intel confirming your download.
NOTE: Installation logs are available at: /var/log/esb-cli/Immersive_Media_<version>/Immersive_Media/install.log
Figure 2. Product Key
- When the installation is complete, you see the message Installation of package complete and the installation status for each module.
NOTE: This step may take about 5 minutes, depending on your network connection.
Figure 3. Installation Complete
If Intel® Smart Edge Open is installed, running the following command should show output similar to the image below. All the pods should be either in the running or completed stage.
kubectl get pod -A
Figure 4. Pod Status
Run the Application
1. Check the Smart_VR_Live_Streaming_of_Immersive_Media pod with the command:
kubectl get pod -n smartedge-apps
You will see output similar to:
Figure 5. Application Status
2. In a browser, go to https://<SmartEdge-node-ip>:31004/ to verify the Smart_VR_Live_Streaming_of_Immersive_Media app result.
Follow the steps below to verify the certificate:
1. Select Click this for testing certificate and refresh.
Figure 6. Test Certificate
2. Select Proceed to...
Figure 7. Test Certificate - Proceed
3. You will see output similar to:
Figure 8. Test Certificate - Complete
4. Visit URL to allow camera and microphone of the device: https://\<SmartEdge-node-ip\>:31004/.
Figure 9. Allow camera and microphone
5. Close the tab and open a new one with https://<SmartEdge-node-ip>:31004/?publish=false.
6. Select Analytics Demo.
Figure 10. Click Analytics Demo
You will see the analytic web:
Figure 11. Analytic Web
- Select 62xxxxxxxxxxx-common in the video from drop-down, and then click the startAnalytics button.
- Select 9D1Axxxxxxxxx-common in the subscribe video drop-down, and then click the subscribe button.
You will see the final result:
Figure 12. Application Video Output
NOTE: When you finish viewing the video or someone reopens a page to check the video, click the stopAnalytics button to stop analytics.
Uninstall the Application
- List the installed RI module with the command:
You will see output similar to:
Figure 11. Edgesoftware List
Uninstall the RI module using the <module-id> from the previous step in the following command:
./edgesoftware uninstall <module-id>
You will see output similar to:
Figure 12. Uninstall RI
- Run the commands below to make sure the block-all-ingres network policy is deleted after uninstalling
$ kubectl get netpol NAME POD-SELECTOR AGE block-all-ingress <none> 58d
Post deletion only the below network policy should be present.
$ kubectl get netpol -n smartedge-apps NAME POD-SELECTOR AGE deny-from-other-namespaces <none> 18h
Node Feature Discovery (NFD) Feature
Immersive Media uses Intel® Distribution of OpenVINO™ toolkit which is optimized for Intel® processors that support special instructions like AVX512VNNI for optimized performance. The deployment of this application will require the node with this feature supported on the node along with Ubuntu 20.04 OS. This NFD features ensures to deploy the application on the node supported with these features. NFD is installed by Intel® Smart Edge Open Developer Experience Kit and running as two pods on Intel® Smart Edge Open.
$ kubectl get pods -A | grep node-feature-discovery smartedge-system nfd-release-node-feature-discovery-master-7f4ffd8f6b-ckwrx 1/1 Running 6 (25h ago) 58d smartedge-system nfd-release-node-feature-discovery-worker-m69h7 1/1 Running 64 (25h ago) 58d
Immersive Media pod scheduled and running successfully on Intel® Smart Edge Open node based on hardware capabilities of Intel® Xeon® Scalable server.
$ kubectl get pods -n smartedge-apps | grep im360 im360-mcf-85744f5b5d-nnrk6 1/1 Running 0 5m26s im360-mdf-7d8455677f-fpkjl 1/1 Running 0 5m26s im360-mif-79674f8978-m52bf 1/1 Running 0 5m26s im360-mqtt-6b6cdd7cfc-8dmcs 1/1 Running 0 5m26s
The following output shows a description of im360 pod, which shows that it is running successfully with the NFD feature.
$ kubectl describe pods im360-mcf-85744f5b5d-nnrk6 -n smartedge-apps ... ... Node-Selectors: feature.node.kubernetes.io/cpu-cpuid.AVX2=true feature.node.kubernetes.io/cpu-cpuid.AVX512VNNI=true Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 32m default-scheduler Successfully assigned smartedge-apps/im360-mcf-c9587755c-9dkds to ubuntu252 Normal Pulled 32m kubelet Container image "openvisualcloud/xeon-ubuntu2004-smartedge-im360:2.9.3" already present on machine Normal Created 32m kubelet Created container im360-mcf Normal Started 32m kubelet Started container im360-mcf
Public Helm Registry for Helm Charts
Installation of Immersive Media Reference Implementation Intel® Smart Edge Open is accomplished using Helm charts. Earlier Helm charts used to be a part of Reference Implementation installation package. Now a global Helm repo is issued, so that Reference Implementation helm charts can be accessible from private and public network. This will speed up and ease the process of introducing updates and their integration with the Reference Implementation.
Summary and Next Steps
This reference implementation successfully demonstrates the following features:
- Video analytics pipeline based on Intel® Distribution of OpenVINO™ toolkit and Intel® Deep Learning Streamer (Intel® DL Streamer).
- Media streams can be forwarded and analyzed through distributed pipeline.
- K8s* based deployment (single node, two pods, multi-containers) on Intel® Smart Edge Open 21.12.
- Enhanced Android* player receives and processes analytics metadata.
- Web UI for simulator.
- For Developers and end users:
- Work on real time video stitching for 4k/8k after VR camera input.
- Work on H.265 OMAF encoding format support, which supports high and low-definition mixing.
- Work on FOV field of view support for dynamically switching to HD.
As a next step, try one of the following:
- Support the multi-clusters deployment.
- Media analytic pipeline qualities’ optimization.
To continue learning, see the following guides and software resources:
Check the log of im360-mif to see if the stream is added successfully:
kubectl logs <im360-deployment-PodName> -n smartedge-apps -c im360-mif
Execute the command helm list to check whether there is im360-app or kubectl get pod -n smartedge-apps to check whether there are related pods.
Subscribed but Video is not displayed
If you have clicked the Subscribe button but no video is displayed, disable the firewall.
## Please turn off the firewall of your target server sudo ufw disable
Video not displayed in browser
If you cannot see the video in the browser, you may need to update the browser version. Refer to the table below for supported browsers on different operating systems.
|Microsoft Edge* 89||Supported|
You must grant the necessary permission by selecting Continue allowing <url> to access your camera and microphone.
Internal Server Error
If there is a error in the
analytics id field,
Internal Server Error, complete the steps as shown in the following screen:
Make sure there is only one
xxx-common item and another three common items.
The Smart_VR_Live_Streaming_of_Immersive_Media application uses the host network, which will use the host's port 31004, 31080. (Port 31004 is for the view side; the port 31080 is for the testing certification.)
If you're unable to resolve your issues, contact the Support Forum.
To attach the installation logs with your issue, execute the command below to consolidate a list of the log files in tar.gz compressed format, e.g., Smart_VR_Live_Streaming_of_Immersive_Media.tar.gz.
tar -czvf Smart_VR_Live_Streaming_of_Immersive_Media.tar.gz /var/log/esb-cli/Immersive_Media_3.0.0/immersive_media/
Product and Performance Information
Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.