Contents-
Set VPP filter for Frame rate conversion
In this article, we will talk about frame rate conversion algorithm of Media SDK (part of Intel® Media Server Studio). Motivation behind this is different frame rates being adopted in different countries and to convert the frame rate of the video captured using HD equipment's which records at high fps. Frame rate is basically the rate at which frames process in a second, usually expressed in fps.
We will use simple_vpp.sln from latest tutorial package(version 0.0.3) which can be downloaded from the Training web page. Frame Rate conversion is one of the commonly used Video Processing Procedure(VPP), also known as mandatory filters. Please note that the Intel Media SDK implementation of frame rate conversion assumes constant frame rate. Do not use frame rate conversion for variable frame rate (VFR) inputs.
There are three implementation for FRC in the Intel Media SDK-
MFX_FRCALGM_PRESERVE_TIMESTAMP
MFX_FRCALGM_DISTRIBUTED_TIMESTAMP
MFX_FRCALGM_FRAME_INTERPOLATION
The application must specify the input and output frame rates using the FrameRateExtN and FrameRateExtD parameters in the mfxFrameInfo structure.The frame rate is defined as : Frame rate = FrameRateExtN / FrameRateExtD. We support both conversion i.e. high to low fps and low to high fps. Commonly supported conversion ratios are 1:2 and 2:5 (useful for creating 60Hz content from 30Hz and 24Hz, respectfully).
Set VPP filter for Frame rate conversion :
Here we are configuring the DoUse hint and initialize the mfxExtVPPFrameRateConversion structure. The mfxExtVPPFrameRateConversion structure configures the VPP frame rate conversion filter. The user can attach this structure to the mfxVideoParam structure when initializing video processing, resetting it or query its capability.
mfxExtVPPDoUse extDoUse;
memset(&extDoUse, 0, sizeof(extDoUse));
mfxU32 tabDoUseAlg[1];
extDoUse.Header.BufferId = MFX_EXTBUFF_VPP_DOUSE;
extDoUse.Header.BufferSz = sizeof(mfxExtVPPDoUse);
extDoUse.NumAlg = 1;
extDoUse.AlgList = tabDoUseAlg;
tabDoUseAlg[0] = MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION;
mfxExtVPPFrameRateConversion m_FrameRateConversion;
memset(&m_FrameRateConversion, 0, sizeof(m_FrameRateConversion));
m_FrameRateConversion.Header.BufferId = MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION;
m_FrameRateConversion.Header.BufferSz = sizeof(mfxExtVPPFrameRateConversion);
m_FrameRateConversion.Algorithm = 2;
mfxExtBuffer* ExtBuffer[2];
ExtBuffer[0] = (mfxExtBuffer*)&extDoUse;
ExtBuffer[1] = (mfxExtBuffer*)&m_FrameRateConversion;
VPPParams.NumExtParam = 2;
VPPParams.ExtParam = (mfxExtBuffer**)&ExtBuffer[0];- If the number of frames consumed at input is equal to the number of frames generated at output, VPP returns MFX_ERR_NONE when an output is ready.
- If the number of frames consumed at input is more than the number of frames generated at output, VPP returns MFX_ERR_MORE_DATA for additional input until an output is ready. When the output is ready, VPP returns MFX_ERR_NONE. This will happen in case of decreasing frame rate, where more data is needed to process.
- If the number of frames consumed at input is less than the number of frames generated at output, VPP returns either MFX_ERR_MORE_SURFACE (when more than one output is ready). when one output is ready and VPP expects new input, it returns MFX_ERR_NONE. This will happen in case of increasing frame rate, where more surfaces are needed.
The application must process the output frame after synchronization, as the MFXVideoVPP_RunFrameVPPAsync function is asynchronous, which means call processes immediately and doesn't wait for operation to finish. At the end of a sequence, the application must provide a NULL input to drain any remaining frames.
unsigned long int InputTimeStamp;
int factor = 90000/VPPParams.vpp.In.FrameRateExtN;
while (MFX_ERR_NONE <= sts || MFX_ERR_MORE_DATA == sts) {
if (!isMultipleOut) //to handle MFX_ERR_MORE_SURFACE
{
nSurfIdxIn = GetFreeSurfaceIndex(pVPPSurfacesIn, nVPPSurfNumIn); // Find free input frame surface
MSDK_CHECK_ERROR(MFX_ERR_NOT_FOUND, nSurfIdxIn, MFX_ERR_MEMORY_ALLOC);
sts = LoadRawFrame(pVPPSurfacesIn[nSurfIdxIn], fSource); // Load frame from file into surface
MSDK_BREAK_ON_ERROR(sts);
pVPPSurfacesIn[nSurfIdxIn]->Data.TimeStamp = InputTimeStamp;
InputTimeStamp += factor; // factor added to ditribute timestamps
}
else isMultipleOut = false;
nSurfIdxOut = GetFreeSurfaceIndex(pVPPSurfacesOut, nVPPSurfNumOut); // Find free output frame surface
MSDK_CHECK_ERROR(MFX_ERR_NOT_FOUND, nSurfIdxOut, MFX_ERR_MEMORY_ALLOC);
for (;;) {
// Process a frame asychronously (returns immediately)
sts = mfxVPP.RunFrameVPPAsync(pVPPSurfacesIn[nSurfIdxIn], pVPPSurfacesOut[nSurfIdxOut], NULL, &syncp);
if (MFX_WRN_DEVICE_BUSY == sts) {
MSDK_SLEEP(1); // Wait if device is busy, then repeat the same call
}
else
break;
}
if (MFX_ERR_MORE_DATA == sts) // Fetch more input surfaces for VPP
continue;
// MFX_ERR_MORE_SURFACE means output is ready but need more surface (example: Frame Rate Conversion 30->60)
if (MFX_ERR_MORE_SURFACE == sts)
{
isMultipleOut = true;
sts = MFX_ERR_NONE;
}
MSDK_BREAK_ON_ERROR(sts);
sts = session.SyncOperation(syncp, 60000); // Synchronize. Wait until frame processing is ready
MSDK_CHECK_RESULT(sts, MFX_ERR_NONE, sts);
++nFrame;
if (bEnableOutput) {
sts = WriteRawFrame(pVPPSurfacesOut[nSurfIdxOut], fSink);
MSDK_BREAK_ON_ERROR(sts);
printf("Frame number: %d\t", nFrame);
printf("Output Time Stamp: %d\n", pVPPSurfacesOut[nSurfIdxOut]->Data.TimeStamp);
fflush(stdout);
}
}This algorithm is based on the frame drop/repetition frame rate conversion algorithm with preserved original time stamps. The algorithm handles any inserted frames by giving MFX_TIMESTAMP_UNKNOWN or -1 stamp and the time stamp for the dropped frame will be skipped. This algorithm can handle arbitrary input/output frame rate conversion.
This algorithm is based on the frame drop/repetition frame rate conversion algorithm with distributed time stamps. This algorithm distributes the time stamps according to the samples per frame which is calculated by dividing the clock rate with the input frame rate. The number of samples per frame is included to calculate the time stamps so that it can distribute according to the increase in the change of the frame rate. The clock rate for the video is 90,000Hz. For example if the input frame rate is 30fps and the output frame rate is 60fps, then it will be 3000 samples per frame and the time stamps would look like 0,1500,300 and so on.Therefore, it will generate double the number of frames. The time stamp for the dropped frame will be skipped when decreasing the frame rate.
This algorithm is based on the frame interpolation which means it will interpolate the new frames in case of increasing the frames rate. It depends upon the driver implementation, it will choose best option out of interpolated vs. duplicated frame (quality based). So, in interpolated stream you may notice mix of interpolated and duplicated frames.
More complex frame rate conversion approaches can be implemented as user plugins. An Important point to note that the frame rate conversion algorithm does not take into account input time stamps. Therefore, the number of output frames does not depend on any inconsistencies in the input time stamps. For more details on Frame Rate Conversion, please refer to manual which comes in the package or developer guide from the documentation section of support webpage. With this we conclude the article on FRC in Intel Media SDK. We would include relevant information as and when we have about the algorithms and their implementation