Intel® Collaboration Suite for WebRTC (Intel® CS for WebRTC) Media Analytics Server Guide

Overview

Introduction

Welcome to the Media Analytics Server User Guide for Intel® Collaboration Suite for WebRTC (Intel® CS for WebRTC). This guide describes how to install and configure the Intel Media Analytics Server to work with MCU. This guide also explains how to create and integrate plugins used by the analytics server to perform customer-defined media analytics.

The Media Analytics Server can host customer defined media analytics plugins to perform analytics on streams from MCU, push back analyzed media back to MCU for real-time streaming and send out real-time analytics events. The proposed usage scenario for real-time media analytics includes but is not limited to movement or object detection in surveillance and remote health care, customer and audience analyzing in retail and remote education, etc.

The following list briefly explains the purpose of each section in this guide:

• Section 1. Introduction and conventions used in this guide.
• Section 2. Installing and configuring the Media Analytics Server for Intel CS for WebRTC.
• Section 3. Steps to perform analytics with Media Analytics Server.
• Section 4. Guide on creating media analytics plugins and integrate with Media Analytics Server.

Installation requirements and dependencies for the MCU is described in MCU server guide document separately.

Terminology

This manual uses the following acronyms and terms:

For more information

For more information, please visit the following Web pages:

• Intel HTML Developer Zone
• Intel Collaboration Suit for WebRTC
  • Intel Collaboration Suite for WebRTC
  • Intel Collaboration Suite for WebRTC Forum
  • Intel Collaboration Suite for WebRTC China Forum
• The Intel Engineering Task Force(IETF) Working Group
• W3C WebRTC Working Group
• WebRTC Open Project

Media Analytics Server Installation

Introduction

This section describes the system requirements for installing the Media Analytics Server.

Requirements and compatibility

The following table describes the minimum system requirements for installing the Media Analytics Server.

The Media Analytics Server includes a distribution of clCaffe and we recommend Ubuntu 16.04 LTS 64-bit to use that library when creating your own video analytics plugins.

The Media Analytics Server is compatible with any form of client that is capable of sending RESTful requests. If the Media analytics client also interacts with MCU for video/audio subscription and publication, please refer to MCU Server Guide for the full list of compatible client types.

Install the Media Analytics Server

This section describes the dependencies and steps for installing the Analytics Server.

Dependencies

The following table describes Analytics Server dependencies:

All dependencies, except system libraries and node, are provided with the release package.

All essential system libraries are installed when you install the Video Analytics Server package using the Ubuntu* package management system.

Regarding Node.js*, make sure it's installed in your system prior to installing the Video Analytics Server. We recommend version 8.11.4444. Refer to http://nodejs.org/ for the details and installation.

Before installing the Video Analytics Server, make sure your login account has sys-admin privileges; i.e. the ability to execute sudo.

Install the Analytics Server package

In the server machine, directly unarchive the package file:

tar xf CS_WebRTC_Media_Analytics_Server.v<Version>.tgz

Configure the Analytics Server

Before starting the media analytics server, perform following configuration:

Update the super service information for RESTful for token fetching from MCU

Modify sampleAnalyticsServer.js, update the line starting with icsREST.API.init, and replace the super service ID, super service key, and URL with information corresponding to your MCU.

Change server settings

Following settings can be updated in analytics/analytics.toml:

Launch the Media Analytics server

To launch the Video Analytics server on one machine, follow steps below:

• Initialize the Video Analytics package for the first time execution to initialize rabbitmq services.
  For general Media Analytics Server installation, use following command:

  bin/init-all.sh [--deps]
  

  Note: If you have already installed the required system libraries, then –deps is not required.

• Run the following commands to start the Media Analytics server:

    cd Release-<Version>/
    bin/start-all.sh
  

• To verify whether the server has started successfully, launch your browser and connect to the Media Analytics Server at https://XXXXX:9904. Replace XXXXX with the IP address or machine name of your MCU server.

Note: At present we require MCU portal's SSL setting in portal/portal.toml set to "false" for the analytics server to connect to MCU.

Stop the Media Analytics server

Run the following commands to stop the Media Analytics server:

cd Release-<Version>/
bin/stop-all.sh

Deploy the Media Analytics Server as cluster

The Media Analytics server supports to be distributed as cluster. There can be only one analytics sample service app and multiple analytics agents in the system.

• The analytics sample servers must be first started on one of the hosts using following command:

    bin/daemon.sh start app

   

• The analytics agents can be started following the app with below command, and you can do this on the same host that runs app, or other hosts multiple times.

  bin/daemon.sh start analytics

   

• When stopping the service, run following command to stop analytics agents first:

  bin/daemon.sh stop analytics

   

• After all analytics agents have been stopped, run following command to stop anlytics sample service app on the host that started it up.

  bin/daemon.sh stop app 

   

Perform Media Analytics

By default Media Analytics Server listens on non-secure TCP port 9901 and secure TCP port 9904 to receive RESTful requests from clients.

Requesting analytics for a stream

Client is required to send below request via analytics server URL to start analytics for existing streams on MCU.

Request:
   method: ‘POST’
   path: /startAnalyticsForStream
   body: {"access_token": string(Join_Token),
              "stream": string(StreamId),
              "plugin": string(PluginGuid),
              "params": string(Params)}

Response:
   body: object(Analytics_Token)

Note: access_token is required to be fetched by analytics client by sending RESTful request to MCU. The sampleAnalyticsServer in the package works as the proxy for token request, you can use it if there is no MCU basic example app deployed on server side. params is optional and could be used to pass extra parameters to the analytics plugin.

Analytics_Token will be used by the stopAnalyticsForStream request to stop analytics on a stream.

If the request with same parameters has been started, Media Analytics Server will not start a new analytics process but return the same token to client.

Once analytics on the stream is started, a stream with attribute “origin: StreamId” will be added to the room uniquely identified by {token, StreamId, pluginGuid}. Please be noted the analytics stream will not be directly mixed into any views. To view it, you'll need to subscribe the forward stream.

Stopping analytics for a stream

Clients need to send below request to video analytics server to stop analytics on a stream:

Request:
  method: ‘POST’
  path: /stopAnalyticsForStream
  body: {token: object(Analtyics_Token)}

Response:
   body: empty

Once analytics is stopped, the associated analytics stream will be removed from MCU.

Using the Pre-built Plugins

The Media Analytics Server is shipped with four prebuilt plugins. To verify them, first modify the analytics/public/script/index.js, search the line that contains plugin:, and update the plugin GUID with the one you would like to use.

Face detection plugin

Identified by GUID b849f44bee074b08bf3e627f3fc927c. This plugin provides the capability of finding faces in current streams and annotates it with a rectangle boarder on the face.

Face recognition plugin

Identified by GUID 3f932ff2a80341faa0a73ebb3bcfb85d. This plugin provides the capability of identify the people's name in current stream and annotates them with a rectangle on the face, and also list the name and the confidence of the recognition result. To add new people for recognition, here are the steps:

1. Run the photo_taker tool to take at least three pictures of one person and place them under the raw_photos directory with sub-directory name that identifies that person's name(no space in the directory name).
2. Run the pre-process tool to process the raw photos. Put the output vectors.txt under the plugin's configpath.

Body face style recognition plugin

Identified by GUID 7e399f65545b482194d6f6ae1d46df05. This plugin provides the capability of recognizing the body and face of a person and annotates accordingly. The procedure of adding new people for recognition is the same as face recognition plugin.

Dummy plugin

Identified by GUID dc51138a8284436f873418a21ba8cfa7. This plugin simply modifies part of the stream to demonstrate the working process of plugins.

Create and Deploy Media Analytics Plugin

The Media Analytics Server package provides analytics plugin sdk under “plugin” directory. Refer to the plugin.h for more detailed API description.

Create plugin

Your plugin class implementation must inherit from rvaPlugin interface. Following the plugin class implementation, it is required to include the DECLARE_PLUGIN(ClassName) macro to export your plugin implementation.

Refer to plugin/samples/dummy_plugin/myplugin.cc for a reference dummy implementation of plugin.

Build plugin

Plugin must be compiled into a shared .so file. At linking time, it is require that –fPIC and –shared –std=gnu++11 to be specified to be compatible with Media Analytics Server worker node.

Deploy plugin

To deploy a plugin to Media Analytics Server, you will need to generate a new GUID for your plugin. After that, copy your plugin .so file to server:

cp lib<yourplugin>.so Release-<Version>/lib/<yourPluginGuid>

And add a section in Release-<Version/analytics/plugin.cfg> file, for example:

[b342f49baa093c27cf1e328f2fc987b6]
description = 'dummy plugin without modifying original video'
pluginversion = 1
apiversion =400
name = 'libmyplugin.so'
libpath = 'lib/b342f49baa093c27cf1e328f2fc987b6/'
configpath = 'lib/b342f49baa093c27cf1e328f2fc987b6/'
messaging = true       # set to false if your plugin does not send notification  
inputfourcc = 'I420'   # must be I420 for current version
outputfourcc = 'I420'  # set to "" if your plugin will not republish analyzed stream to MCU.