Introduction
Intel® Deep Link technology harnesses the power of Intel® processors and graphics processing units in end-user devices to accelerate content creation, gaming, and productivity with intelligently boosted performance. Through dynamic power sharing, improved artificial intelligence (AI) performance, and powerful encode capabilities, Intel® Deep Link technology can help today’s devices to keep up with every workflow.
This article provides an overview of the latest multimedia processing capabilities accelerated by Intel® Deep Link technology and examines how Intel technology partner Fluendo applied these solutions to enable its R&D staff to stay productive remotely during the pandemic.
Intel® Deep Link Technology Overview
Introduced with 11th generation Intel® Core™ processors, Intel® Deep Link technology harnesses the compute power of the CPU and the GPUs in those processors: Intel® Iris Xe MAX and Intel® Iris Xe Graphics, as well as special instructions for increasing the performance of AI applications for generations to come. Features include:
- Dynamic power sharing intelligently routes power between the processor and GPU to boost performance for content creation and other tasks.
- AI-enabled editing as Intel® Deep Link technology helps keep the creative flow going, freeing the user device to handle AI video and image workloads.
- Video encoding to speed render times as Intel® Deep Link technology and Intel® Iris Xe graphics provide hyper-fast encoding speeds to accelerate uploads and shorten finish times.
Fluendo and the GStreamer* Framework
One of Intel’s key industry partners developing applications for Intel® Deep Link technology is Fluendo, a leading provider of multimedia software solutions for the open-source GStreamer* framework. GStreamer* simplifies audio/video write applications, enabling developers to process any kind of flow. By dramatically enhancing the multimedia experience, GStreamer* helps audio and video files run smoother while minimizing memory consumption.
Based in Barcelona, Spain, Fluendo’s multimedia technology drives a range of devices, from system-on-chip (SoC) and set-top boxes to smart TVs, thin clients, in-vehicle applications and personal computers. Since 2004, Fluendo has developed innovative video encoding/decoding (codec) solutions for GStreamer* to create media players, video editors, streaming media broadcasters, and similar applications, as well as software components such as codecs, filters, muxers and demuxers. Fluendo has taken a leading role in building legal multimedia products developed under this versatile framework, actively contributing to the open-source GStreamer* community growth.
Offered free of charge, GStreamer* is released under the LGPL V2 license, allowing developers to use it in commercial or closed applications. This important collaborative achievement has established GStreamer* as the de-facto standard multimedia framework for GNU*/Linux* and Unix* systems.
Fluendo Codec Pack for OEMs
Fluendo secured formal agreements with industry-leading patent holders, including MPEG LA, Microsoft*, VIA Licensing*, Technicolor* and Dolby*. It is the only multimedia company globally to provide a comprehensive set of codecs for audio and video, delivered to the client with its corresponding patent licenses for distribution. Fluendo Codec Pack, a custom package for enterprise purposes, can be easily embedded by OEMs into their platforms and devices to enhance their multimedia content playback.
In 2020, Fluendo established its R&D+Innovation department to diversify business lines, analyze new-product ideas, match existing technologies to new market solutions, and position itself as innovation leaders in the multimedia landscape. The main research areas of this new department include artificial intelligence applied to audio/video and next-generation audio/video codecs.
Fluendo is partnering with Intel to develop GStreamer* platform components to support hardware acceleration decoding for both audio and video, helping enhance the performance of embedded devices under Linux*, especially for set-top boxes. This successful project was released as open-source to ease development of multimedia applications in IPTV and digital TV markets.
“We collaborated with Intel® Software engineers to realize the full potential of new Intel® hardware”, explains Fluendo Chief Sales Officer Gregori Martinez. “This collaboration between Fluendo and Intel experts in hardware was focused on delivery of performance-oriented and cost-effective solutions for Fluendo customers.”
Support for Linux*
Still, support for hardware acceleration decoding was something new in Linux* and no API was yet available. That changed after mainstream GPU manufacturers for desktop devices began to support hardware acceleration in their graphics cards and no common API was available in Linux*. Fluendo stepped up with the first GStreamer*-based decoder to support VA-API to leverage the decoding to the GPU. Intel’s target graphics device, Intel® Graphics Media Accelerator 500 (Intel® GMA 500), focused mainly on the netbook market. Hardware acceleration support became a “must,” considering the low consumption of the device despite high market demand for multimedia content. The decoding capabilities included MPEG-2, MPEG-4, H.264 and VC-1.
Intel and Fluendo achieved a major milestone with the first dedicated graphics card, as Fluendo offered support for multiple GPUs, as well as Intel® Xe architecture under GStreamer*, adding it to its codec pack.
Fluendo chose a producer-consumer application approach: the “producer” side optimized the Fluendo Codec Pack for Intel® Xe architecture with multiple GPUs, to simultaneously encode multiple 4K video sources in H.264 and transmit them via WebRTC. On the “consumer” side, the web application developed in Node.js* is using WebRTC as the protocol to receive video, audio and control commands from the producer.
In its Barcelona headquarters, Fluendo built a laboratory composed of Hybrid broadcast broadband TV (HbbTV*) modulators and receptors using modern software technology. Thanks to this installation, developers and testers can immediately validate the result of their work on the same hardware as an end-user. “This initiative integrates TV broadcasts with Internet content on connected TVs providing higher quality playback and lower bandwidth,” according to Martinez.
Staying Productive During the Pandemic
In 2020, COVID-19 emergency measures forced Fluendo, like countless other businesses, to start working remotely. To enable developers and testers to remain productive, the company needed a very low latency streaming solution with simple client access through a web browser and parallel encoding of multiple 4K videos with audio. As experts in multimedia streaming, Fluendo opted to build its own solution.
A key feature of Intel® Deep Link technology is its ability to increase performance by balancing the load of the decoding GPUs, whether dedicated or integrated. During the pandemic, Fluendo leveraged that capability in a project that became critically important to maintaining product testability even under remote work conditions.
Using its codecs and technology expertise, Fluendo built a streaming solution around WebRTC protocol to comply with very low latency and easy access. Still, with common PC hardware, the company quickly encountered limitations to encode more than a few 4K streams in parallel. Intel and Fluendo collaborated on a new solution using the new Intel® Deep Link technology and Intel® Iris Xe graphics. As a result, Fluendo can now encode more than 12 simultaneous 4K streams in parallel. And its web application can connect directly to those streams so that users can remotely manage them, associating the laboratory audio and video sources to any stream.
“The new solution uses capabilities of Intel hardware, enabled with Intel software tools, so we are now able to encode up to 12 livestreams in 4K using one single PC using Intel® Deep Link technology and powerful Intel® Iris® Xe graphics together with 11th Gen Intel® Core™ processors”, says Martinez. “Thanks to our codec technology and Intel hardware, we are now able to offer a better, faster, more balanced solution.
Fluendo developers and testers can now view the results of their work in real-time as if they were physically present onsite. And, they can consult encoding statistics to monitor the streaming server load from Linux*, Mac*, or Windows*. Thanks to its codecs and Intel® Deep Link technology, Fluendo was able to adapt to the new realities of a remote workforce and maintain its same efficiencies as before.
Resources
Effective Enabling for Intel® Deep Link Technology