Co-author: Henrik Levring, Miguel Molina and Louis Vigil
Executive Summary
XSplit VCam was developed by the research and development division of SplitmediaLabs to bring advanced-quality video background removal solution to the game streaming market. The technology uses a deep learning AI framework coupled with a custom-built, high-performance inference engine.
The unique inferencing engine in XSplit VCam is based on OpenCL™, a standard framework for execution on heterogeneous platforms. It provides the flexibility to execute on the CPU, discrete and integrated Graphics Processing Units (GPUs), digital signal processors (DSPs), and field programmable gate arrays (FPGAs), as well as custom silicon designed for machine learning, such as the Intel® Movidius™ Neural Compute Stick (NCS).
This white paper demonstrates how XSplit VCam provides streaming gamers and gaming content creators with a reliable solution for intelligent, real-time video background removal.
Background
Performance remains a key issue for gamers, and the challenge is to ensure that video background removal does not impact their gaming experience. With AI, video encoding, and in-game physics – such as dynamic explosions, crumbling buildings, and so on – competing for resources, the discrete GPU on gaming computers is coming under increasing pressure for streamers. Assigning XSplit VCam processing to execute on Intel® Graphics Technology (integrated GPUs) or multicore CPUs, helps to ensure that a game running on a platform with a discrete GPU is not impacted.
Solution Overview
XSplit VCam Features for Game Streaming
One of the most important aspects of game streaming is the interaction between the streamer and the audience. Showing the streamer’s reactions through the use of a webcam is a common way to achieve interaction. XSplit estimates that up to 50 percent of game streams utilize a webcam as part of their broadcast presentation1 . XSplit VCam can further enhance this experience by adding a range of customization options to the streamer’s toolkit, allowing them to achieve a perceived high production value.
The following use cases demonstrate how the features in XSplit VCam handle common broadcast presentation problems that occur when streaming games, such as background removal, background replacement, and background blurring.
Background Removal
The majority of gaming broadcasters that feature a webcam in their presentation tend to prefer complete background removal, immersing themselves fully in the game. Previously, achieving these results would require a full green screen setup in the background in combination with studio lighting. However, XSplit VCam allows users to achieve this effect using only a standard webcam without a green screen. When used in conjunction with XSplit Broadcaster or other popular streaming software, the user is only limited by their creativity to use this feature in a variety of ways – see figure 1.
Figure 1. Full background removal with XSplit VCam and XSplit Broadcaster
Background Replacement
Often when streaming games, a user mayopttoreplace the background oftheir webcam feed. XSplitVCam allows forthis by letting users insert any image or website as their background. This option lets users place themselves into unique environments or take advantage of the ability to load a website toinsert motionbackgrounds. For example, if they’re playing the latest Legend of Zelda* game, users can set the kingdom of Hyrule as their background – see figure 2.
Figure 2. Background replacement with XSplit VCam
Background Blur
A newer trend among streamers is to display their original background, which is usually composed and designed to show on camera. Frequently users will ensure that their background is made up of branding, memorabilia, and an array of colored lighting to achieve a visually-pleasing presentation. Background blur can be useful in these cases by allowing a user to incrementally soften the background simulating the depth of field effect produced by high-end cameras.
This effect allows for a user to, again, increase the perceived production value of their broadcast with just a simple desktop webcam, as opposed to a full tripod mounted camera. Additionally, blur can be increased more dramatically to completely obfuscate the background which can be useful when wanting to provide privacy, hide unsightly details (e.g. a messy room) or reduce distractions that come with being in crowded or public areas.
Background blur can be used in conjunction with the aforementioned background replacement feature to add a soft or dramatic blur to whatever backdrop a user may choose to load, achieving an effect similar to those used in news broadcasts – see figure 3.
Figure 3. Background blur and background replacement with XSplit VCam
Benefits of Intel® Graphics Technology
Most laptop and desktop computers powered by an Intel CPU have an integrated Intel GPU embedded on the same die as the processor. The primary benefit of Intel integrated GPUs (iGPUs) have traditionally been their affordability as a single CPU and GPU system, combined with prolonged battery life in gaming laptops with both an iGPU and a discrete GPU (dGPU).
With 8th generation Intel® Core™ processors this no longer remains the main argument. The Intel iGPU now packs extra power, allowing applications to run alongside games, or the game itself, to use the iGPU to offload process-intensive tasks which can lead to an improvement in overall user experience.
On gaming computers with 8th generation Intel Core processors and integrated graphics, XSplit VCam can process camera frames in real-time and perform background removal, replacement, or blur exclusively on the Intel iGPU. Allowing games to run virtually unaffected on the computer’s dGPU and CPU.
Note: On some desktop gaming motherboards or prebuilt desktop PC systems with a discrete GPU, the iGPU is disabled in the BIOS by some manufacturers, but users generally can enable the iGPU themselves.
Performance Assessment
This resulting performance assessment is based on SplitMediaLabs simulations to gauge the impact on a game’s frames per second (FPS) rate when using the background removal function in XSplit VCam. The setup involves both a desktop and a laptop device considered mid-range by SplitmediaLabs at the time of publication.
The tests explore the effects on the average FPS of two different game applications, namely UNIGINE Valley, a game engine benchmarking tool, and Fortnite*, a wildly popular game. A baseline was established by measuring the average FPS of the game applications with no direct impact from other running applications and using the standard quality presets.
Next, Open Broadcaster Software (OBS) Studio, a widely used broadcasting solution, was added to the workflow with both a game capture and a standard webcam in the default scene. Representing the most common use case for live streaming, and the impact on the average FPS in this scenario is what is considered normal. Using a camera output of 360p at 30 FPS. It was being recorded with NVENC H.264, CQP 23 - a form of quality-based encoding where the bitrate adjusts dynamically to produce a consistent quality determined by the Constant Quantization Parameter (CQP). To illustrate how much XSplit VCam would affect the game applications the SplitMediaLabs tests were conducted on either the iGPU or the dGPU of the host machine.
In the interest of introducing less bias, OBS is used as the streaming/recording tool instead of XSplit Broadcaster or XSplit Gamecaster, the sister products of XSplit VCam.
Results
With the desktop setup, XSplit VCam running on the iGPU nets a 0.5 percent hit to the overall average FPS of UNIGINE VALLEY while in contrast, running on the dGPU causes over a 7.7 percent dip – see figure 4. The ‘performance %’ in This graph, and each of the subsequent graphs compare the FPS of the particular test with the baseline FPS of the control. The control is the instance where only the benchmark was running which is why it appears as 100 percent.
Figure 4. Desktop performance for UNIGINE Valley benchmark across three runs
See below for further notes and disclaimers.2
Similarly, Fortnite with XSplit VCam on the iGPU only affects the game’s FPS by 1.2 percent while on the dGPU, the performance drop is 8.8 percent – see figure 5.
Figure 5. Desktop performance for Fortnite* across three runs
See below for further notes and disclaimers.2
The laptop results show a similar pattern in performance with XSplit VCam on the iGPU only affecting UNIGINE Valley’s FPS by one percent, whereas running on the dGPU causes over an 8.5 percent drop – see figure 6.
Figure 6. Laptop performance for UNIGINE Valley benchmark across three runs
See below for further notes and disclaimers.2
The trend continues in Fortnite, with XSplit VCam on the iGPU impacting the game’s FPS by 1.1 percent versus having it on the dGPU, where the performance hit is 8.4 percent – see figure 7.
Figure 7. Laptop performance for Fortnite* across three runs
See below for further notes and disclaimers.2
Conclusion
XSplit VCam uses a convolutional neural network model to perform background removal processing in real time. The neural network processing (inferencing) is a significant workload which requires millions of calculations for each processed frame. However, results show that by performing this workload entirely on an Intel iGPU, users looking to stream or record their gameplay can use XSplit VCam to improve production quality of their stream with minimized FPS performance impact.
Ultimately, this means that XSplit VCam, when combined with an Intel iGPU, can provide gamers the benefit of high-quality simulated background removal with minimal performance loss while gaming.
To give XSplit VCam a try, download it for free.
Learn More
You may find the following resources useful:
References
1. Based on XSplit usage statistics.
2. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. For more complete information visit www.intel.com/benchmarks.
Performance results are based on testing as of December 18th, 2018 and may not reflect all publicly available security updates. See configuration disclosure for details. No product can be absolutely secure.
Configurations: HARDWARE: Desktop: Motherboard: ASUS TUF Z370-PLUS GAMING*; BIOS Version: American Megatrends Inc. 0606, 12/12/2017; CPU / iGPU: Intel® Core™ i7-8700K processor @ 3.7 - 4.7GHz / Intel® UHD Graphics 630 @ 350 MHz - 1.2 GHz (Driver 24.20.100.6136); RAM: 16 GB of DDR4 SDRAM @ 2400 MHz; dGPU: NVIDIA GeForce* GTX 1050 Ti 4 GB (Driver 417.22) @ 1291 - 1392 MHz; OS: Windows* 10 Enterprise (Build 17134). Laptop: Model: ASUS TUF GAMING* FX504G -FX80GE; BIOS Version: American Megatrends Inc. FX504GE.312, 13/07/2018; CPU / iGPU: Intel® Core™ i5-8300H processor @ 2.3 - 4.0GHz / Intel® UHD Graphics 630 @ 350 MHz - 1.0 GHz; RAM: 16 GB of DDR4 SDRAM @ 2400 MHz; dGPU: NVIDIA GeForce* GTX 1050 Ti 4 GB (Driver 417.22) @ 1493 - 1620 MHz; OS: Windows* 10 Home Single Language (Build 17763). Camera: Logitech C922 Pro Stream* HD Webcam. Secondary Monitor: BenQ XL2311P @ 1080p 60 Hz via HDMI. SOFTWARE: Broadcasting application: OBS 21.1.2 on Secondary Monitor; 1080p, 30 FPS Base and Output Resolution; NVENC H.264 Recording, CQP 23; Game Capture + Video Capture Device. Benchmark tool: UNIGINE Valley 1.0; Custom Preset, DirectX* 11, Ultra Quality, Anti-aliasing x8, 1080p Full Screen; Average FPS shown after running the benchmark. Game: Fortnite*; Full Screen 1080p, High Preset, Unlimited FPS, Motion Blur On; 3-Minute Replay Footage. FPS Tool: FRAPS 3.5.99; Used to measure Average FPS of Fortnite. TESTING: Carried out by SplitmediaLabs. December 18th, 2018.