More and more Chinese engineers joined FFmpeg development these days, while FFmpeg is broadly adopted in Cloud video services. The Media Lab of Shanghai Jiao Tong University's Image Institute and Intel Asia-Pacific R & D Co., Ltd. Video Frame Group jointly organized a technical salon called "FFmpeg China Day" to involve more developers to open source their technologies and share between ffmpeg developers/users. The salon was held on the afternoon of December 21, 2019 at Minhang Campus of Shanghai Jiao Tong University. Local core contributors of FFmpeg open source community were invited to exchange ideas. We are expecting to invite more global FFmpeg core contributors to share more.
The agenda and material is as below:
- Opening
- Looking back and forward of FFmpeg
- My learnings and thoughts about FFmpeg
- FFmpeg and AVS
- FFmpeg HW acceleration through VAAPI and QSV
- Application of vector instructions in video codec
- 360-degree video stitching for immersive media
- Solution of 8K Spherical Video Playing Online
- Round table discussion: AV1, DNN in FFmpeg
Slices download link
Extraction code: 2e5a
Details:
Opening Remarks
The meeting started at 1:30 pm on the 21st in Room 406, Building 5, School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University. Ms. Zhao Juan, R & D Manager of Intel Asia-Pacific R & D Co., Ltd., made the opening remarks, and talked about the goal and purpose of the event.
The first goal of the event is that developers of Chinese can sit down, talk about common requirements, and do some technical discussions, so that developers can improve themselves and change previous close development habits which may bring duplicate works. The needs of various companies are so similar. Hopefully, through open source, common problems of companies can be more easily addressed. The second goal is to provide more opportunities for colleges and students, allow them to participate more in the open source community. Cooperation between the two sides will not only exercise the students' own ability, but also give the community more fresh blood.
Professor Song Li from Shanghai Jiaotong University also made an opening speech, talking about the original intention of the event which is talking with the experts who have been in the industry for a long time, and pulling multimedia ecosystem towards the direction of open source, so many people can benefit. At the same time, some students from SJTU was invited to participate in the event. He hoped that students could participate more in the activities of the open source community. While improving themselves, they can also contribute to the community.
Next, each of the guests introduced themselves, their respective projects, the original intention of participating in this event, and their feelings about the FFmpeg community. The guests included a lot of FFmpeg Maintainers and development experts, including the core contributor of x265, Chen Min, SRS author Yang Chengli, and other experts of open source community, as well as Professor Li Zhu (University of Missouri, Kansas City) from afar. Developers who may be contacted through the FFmpeg mail list can communicate face to face, and the atmosphere of the venue is very active.
Keynote Speech
The next agenda of the conference was keynote speech where the first speaker is Mr. Steven Liu whose topic was “FFmpeg 3 years’ history and 2 years’ plan”.
He first introduced the overview of patches submitted by developers in FFmpeg in the past three years, and user comments on FFmpeg in the past few years, including too few examples, too complex output logs, and not being able to passthrough options to child format. Then he introduces his work plan as the FFmpeg Maintainer in the next two years, which to some extent also represents the future development direction of the FFmpeg community.
The second speaker is Mr. Zhao Jun, whose topic was “Some Opinions on FFmpeg”.
He talked about some issues of FFmpeg from the perspective of users and developers. The first was the complexity of FFmpeg and audio, video field. There were many types of formats, containers, transmissions, and DRMs, and there were possibilities for bugs. The second was that the API is not easy to use and difficult to perform detailed control which lead to poor user experience. The third is performance problem, because FFmpeg needed to cross the operating system, and fit all kinds of hardware, and its optimization on various platforms was inconsistent, which would lead to different speeds. Finally there are some open issues Including whether the deep neural network needed to be put into FFmpeg, whether to put a specific single algorithm or the whole framework, and the cooperation between CPU and GPU. The various questions he raised hit the pain points of FFmpeg, which is the direction that developers should work hard.
The third speaker is Ren Huiwen from Beijing University, whose topic was “FFmpeg x AVS”. It is worth mentioning that Huiwen is the first student committer of FFmpeg.
He first introduced the situation of each version of AVS, and introduced the development progress of AVS3. Then he said that the strategy at high resolution might not necessarily be the pursuit of lower bit rate, but it can be more flexible. Next, he introduced the SVT-AVS3 project, including its high degree of parallelism, more selectable quality levels, and so on. And then the process of AVS Internet promotion, including the establishment of an Internet ecology within two years, getting support from browser, mobile phone and other hardware to promote AVS. In the end, Huiwen and developers exchanged some problems encountered when AVS was integrated into FFmpeg.
The next speaker was Mr. Fu Linjie, whose topic was “An Insight of VA-API and QSV in FFmpeg”.
He first introduced hardware acceleration which is transferring the operations originally performed by the CPU to the GPU, so that a high degree of parallelism can be performed, and the CPU can make other operations. Intel's hardware acceleration includes VA-API and QSV, which exist as plug-ins in FFmpeg. He then introduced the position and processing flow of the two in FFmpeg, as well as the content and applicable platforms of the two.
The next speaker was Mr. Zong Wei, whose topic was “360-degree video stitching for immersive media”
He first introduced how to make a 360° video from beginning to end, including acquisition, stitching, encoding, cdn, player and equipment, etc. Then he introduced several methods of spherical projection, and their advantages and disadvantages, followed by some 360° VR devices, such as multi-camera cameras. Below he introduces the steps of the stitching algorithm, the difficulties encountered and the solutions. In the end, he raised the challenge of 360° VR to the entire process of audio and video processing. 8K 360° video has encountered many difficulties although its effect can only be equivalent to UHD flat video, which revealed the difficulty of getting a higher experience.
The next speaker was Mr. Yin Shiyou, whose topic was “Application of vector instructions in video codec”.
He first gave some data, indicating that vector optimization will bring 3-4 times performance improvement to the video codecs. However, the vector optimization method is different on each platform. If low-level implementation can be shielded, the optimization of each platform is the same. Therefore, he proposed a unified optimization method, that is, each compiler implements unified intrinsics, based on which we can implement more complex codec optimazation APIs, and then each platform can share a single optimized code. Finally, he put forward his own suggestions for developers, including data alignment, reducing jumps, using local variables and algorithm equivalent conversions.
The last speaker was Mr. Zhang Tao, whose topic was “Solution of 8K Spherical Video Playing Online”.
He first introduced the technical challenges of this project, including high bit rate and limitation of terminal chip decoding capabilities. Next, he proposed specific solutions to the challenges of each step. The first is the use of block streaming for spherical projection. This is a dual-stream solution, including a basic stream and an enhanced stream. In the following, he introduced the specific technical solutions of code stream splicing, encapsulation, delay, effective coverage, live broadcast synchronization, and viewpoint prediction.
Roundtable Forum
After the keynote speeches, two round-table forums were arranged, with the topics "AV1" and "NN in FFmpeg". Experts such as Xu Guangxin, Li Zhong , Zhao Jun, and Xing Huaifei commented on the technological progress and commercial deployment of AV1. Guo Yejun (main contributor to ffmpeg NN module) made thoughts and suggestions on the combination of deep learning framework and ffmpeg and got positive feedback from many experts.
At the end of the meeting, all participants took a group photo. The developers went to the Shanghai Jiao Tong University Academic Activity Center to attend a dinner and had further exchanges.
Event organizer Intel prepared a birthday cake for the first successful event. It is hoped that the event will be normalized, so as to promote the healthy development of the FFmpeg community and the progress of the school and the industry.