Download Article
Download Using Unsupported Color Formats with the Intel® Media Software Development Kit [PDF 160KB]
Introduction
The Intel® Media Software Development Kit (Intel® Media SDK) is a software development library that exposes the media acceleration capabilities of Intel platforms for encoding, decoding and video preprocessing (VPP). VPP takes raw frames as input, performs some functions like color conversion, cropping, resizing, de-noising, de-interlacing and provides raw frames as output. However, VPP only supports certain type of input color formats of raw video frames like NV12, YV12 and YUY2. What happen if user has different type of input color formats and they want to take advantage of the Intel® Media SDK to enhance the VPP phase? In this article we will discuss how to use the Intel® Integrated Performance Primitives (Intel® IPP) to convert some common type of color formats like RGB, RGBA, BGR and BGRA into those that can be recognized by the Intel® Media SDK.
The Intel® Media Software Development Kit
As of this writing, the current version of the Intel® Media SDK is version 1.5. The version 1.5 of the Intel® Media SDK provides developers with a standard application programming interface (API) to create high-performance video solutions for consumer and professional uses. The API functions streamline and simplify video encoding, decoding, and preprocessing operations. Support is provided for encoding (including H.264 and MPEG-2 formats) and decoding (including H.264, MPEG-2, and VC-1 formats). The nice thing about the Intel® Media SDK is that developers can gain access to the hardware acceleration capabilities of the target platforms. Platforms that lack dedicated graphics hardware acceleration features still gain the benefit of tuned, optimized, and threaded software-based video encoding and decoding, tailoring performance to the capabilities of each system.
This paper will show how to convert RGB, RGBA, BGR and BGRA raw video of into YUY2 and YV12 before feeding them to VPP.
The Intel® Integrated Performance Primitives
The Intel® Integrated Performance Primitives (Intel IPP) library provides a wide range of functionality including general signal, image, speech, graphics, data compression, cryptography, text string processing, audio processing, vector manipulation, and matrix math; as well as more sophisticated primitives for the construction of audio, video and speech codecs including MP3 (MPEG-1 Audio, Layer 3), MPEG-4, H.264, VC-1,H.263, JPEG, JPEG2000, GSM-AMR* and G.729, plus computer vision. The Intel IPP makes use of new Intel architecture enhancements such as the Intel® 64 architecture (aka Intel® EM64T), Advanced Vector Extensions (Intel® AVX), Streaming SIMD Extensions (Intel® SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2), and MMX™ technology. Intel® IPP is optimized for a broad range of Intel microprocessors, including: Intel® Core™ i7 processors, Intel® Atom™ processors, Intel® Core™2 Quad processors, Intel® Core™2 Duo processors, Intel® Xeon® processors, Intel® Pentium® 4 processor and Intel® Itanium® 2 processors. This paper will talk only about how to use the image processing portion of the Intel IPP. The version 6.1 build 5 of the Intel IPP only supports converting other color formats to YUY2 and YV12. The new version, version 7.0 which is still in beta will support NV12.
Color Formats
Below are the descriptions of the three color formats that are supported by the Intel® Media SDK VPP. They are YUY2, YV12 and NV12. There are two ways for a YUV image to be stored. The first one is called packed format and the second one is planar format. In the packed format, Y, U and V are stored next to each other. In the planar format, all the Y values are put together as wells as U and V. For more information about YUV color formats, Please refer to the web site [3].
Packed Format:
Planar format:
YUY2 is the packed format with 4:2:2 sampling. In YUY2 format the data can be treated as an array of unsigned char values. This can be illustrated as follows:
YV12 is the planar format with 4:2:0 sampling. In YV12 format all the Y samples appear first in an array of unsigned char values. The V samples will be stored after the Y samples and the U samples will be stored last. This can be illustrated as follows:
NV12 is the planar format YUV with 4:2:0 sampling in which U and V are interleave. In NV12 format the data can be treated as an array of unsigned char values. This can be illustrated as follows:
Color Conversions
Developers can write their own color conversion functions to convert from RGB, RGBA, BGR and BGRA to YUY2, YV12 or NV12. However, it is highly recommend that developers use the conversion functions provided by the Intel IPP since IPP functions are highly optimized across many Intel platforms. Below are some of the Intel IPP color conversions:
ippiRGBToYCbCr422_8u_P3C2R - RGBP3 to YUY2
ippiRGBToYCbCr422_8u_C3C2R - RGB24 to YUY2
ippiBGRToYCbCr422_8u_C3C2R - BGR24 to YUY2
ippiBGRToYCbCr422_8u_AC4C2R - BGRA to YUY2
ippiBGRToYCrCb420_8u_C3P3R - BGR24 to YV12
ippiBGRToYCrCb420_8u_AC4P3R - BGRA to YV12
ippiRGBToYCrCb420_8u_AC4P3R - RGBA to YV12
The following functions are only available in the Intel IPP version 7.0
ippiRGBToYCbCr420_8u_C3P2R - RGBA to NV12
ippiRGBToYCbCr420_8u_C4P2R - RGB32 to NV12
The following sample code shows how to call the Intel IPP color conversion functions:
To use the conversion function in the above class, developers need to do something like this:
For the description of the variables please refer to the comments in the sample code since they are different depending on which format (YUY2, YV12 or NV12) to convert to.
Conclusion
Using the Intel IPP color conversion functions to convert raw video format unsupported by the Intel® Media SDK into those (YUY2, YV12) recognized by the Intel® Media SDK. The advantage of using the Intel IPP functions is that they are highly optimized for many Intel platforms.
References
[1] Intel® Media Software Development Kit, http://www.intel.com/software/mediasdk/
[2] Intel® Integrated Performance Primitives, Reference Manual, November 2009. Refer to Intel® IPP /en-us/intel-ipp/
[3] http://www.fourcc.org
About the Author
Khang Nguyen is a Software engineer with the Core Scale Engineering group at Intel. He is currently working on developing tools relating to parallelism and power matters in applications. He has been with Intel since 2000.