| RFC 0000 | Video Codec Requirements and Evaluation | February 2020 |
| Filippov, et al. | Informational | [Page] |
This document provides requirements for a video codec designed mainly for use over the Internet. In addition, this document describes an evaluation methodology needed for measuring the compression efficiency to ensure whether the stated requirements are fulfilled or not.¶
This document is not an Internet Standards Track specification; it is published for informational purposes.¶
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Not all documents approved by the IESG are candidates for any level of Internet Standard; see Section 2 of RFC 7841.¶
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained at https://www.rfc-editor.org/info/rfc0000.¶
Copyright (c) 2020 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Simplified BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Simplified BSD License.¶
This document presents the requirements for a video codec designed mainly for use over the Internet. The requirements encompass a wide range of applications that use data transmission over the Internet, including Internet video streaming, IPTV, peer-to-peer video conferencing, video sharing, screencasting, game streaming, and video monitoring / surveillance. For each application, typical resolutions, frame rates, and picture-access modes are presented. Specific requirements related to data transmission over packet-loss networks are considered, as well. In this document, when we discuss data-protection techniques, we only refer to methods designed and implemented to protect data inside the video codec, since there are many existing techniques that protect generic data transmitted over networks with packet losses. From the theoretical point of view, both packet-loss and bit-error robustness can be beneficial for video codecs. In practice, packet losses are a more significant problem than bit corruption in IP networks. It is worth noting that there is an evident interdependence between possible amount of delay and the necessity of error-robust video streams:¶
o If an amount of delay is not crucial for an application, then reliable transport protocols such as TCP that retransmit undelivered packets can be used to guarantee correct decoding of transmitted data.¶
o If the amount of delay must be kept low, then either data transmission should be error free (e.g., by using managed networks), or the compressed video stream should be error resilient.¶
Thus, error resilience can be useful for delay-critical applications to provide low delay in a packet-loss environment.¶
| Abbreviation | Meaning |
|---|---|
| AI | All-Intra (each picture is intra-coded) |
| BD-Rate | Bjontegaard Delta Rate |
| FIZD | just the First picture is Intra-coded, Zero structural Delay |
| GOP | Group of Picture |
| HBR | High Bitrate Range |
| HDR | High Dynamic Range |
| HRD | Hypothetical Reference Decoder |
| IPTV | Internet Protocol Television |
| LBR | Low Bitrate Range |
| MBR | Medium Bitrate Range |
| MOS | Mean Opinion Score |
| MS-SSIM | Multi-Scale Structural Similarity quality index |
| PAM | Picture Access Mode |
| PSNR | Peak Signal-to-Noise Ratio |
| QoS | Quality of Service |
| QP | Quantization Parameter |
| RA | Random Access |
| RAP | Random Access Period |
| RD | Rate-Distortion |
| SEI | Supplemental Enhancement Information |
| SNR | Signal-to-Noise Ratio |
| UGC | User-Generated Content |
| VDI | Virtual Desktop Infrastructure |
| VUI | Video Usability Information |
| WCG | Wide Color Gamut |
In this section, an overview of video codec applications that are currently available on the Internet market is presented. It is worth noting that there are different use cases for each application that define a target platform, and hence there are different types of communication channels involved (e.g., wired or wireless channels), which are characterized by different qualities of service as well as bandwidth; for instance, wired channels are considerably more free from error than wireless channels and therefore require different QoS approaches. The target platform, the channel bandwidth, and the channel quality determine resolutions, frame rates, and quality (bitrates) for encoding or decoding video streams. By default, color format YCbCr 4:2:0 is assumed for the application scenarios listed below.¶
Typical content for this application is movies, TV shows, and animation. Internet video streaming uses a variety of client devices and has to operate under changing network conditions. For this reason, an adaptive streaming model has been widely adopted. Video material is encoded at different quality levels and different resolutions, which are then chosen by a client depending on its capabilities and current network bandwidth. An example combination of resolutions and bitrates is shown in Table 2.¶
A video encoding pipeline in on-demand Internet video streaming typically operates as follows:¶
| Resolution * | Frame rate, fps | PAM |
|---|---|---|
| 4K, 3840x2160 | 24/1.001, 24, 25, 30/1.001, 30, 50, 60/1.001, 60, 100, 120/1.001, 120. The set of frame rates presented in this table is taken from Table 1 in [PARAMETER] | RA |
| 2K (1080p), 1920x1080 | RA | |
| 1080i, 1920x1080* | RA | |
| 720p, 1280x720 | RA | |
| 576p (EDTV), 720x576 | RA | |
| 576i (SDTV), 720x576* | RA | |
| 480p (EDTV), 720x480 | RA | |
| 480i (SDTV), 720x480* | RA | |
| 512x384 | RA | |
| QVGA, 320x240 | RA |
*NB: Interlaced content can be handled at the higher system level and not necessarily by using specialized video coding tools. It is included in this table only for the sake of completeness, as most video content today is in the progressive format.¶
The characteristics and requirements of this application scenario are as follows:¶
Support and efficient encoding of a wide range of content types and formats is required:¶
This is a service for delivering television content over IP-based networks. IPTV may be classified into two main groups based on the type of delivery, as follows:¶
In the IPTV scenario, traffic is transmitted over managed (QoS- based) networks. Typical content used in this application is news, movies, cartoons, series, TV shows, etc. One important requirement for both groups is random access to pictures -- i.e., random access period (RAP) -- should be kept small enough (approximately, 1-5 seconds). Optional requirements are as follows:¶
For this application, typical values of resolutions, frame rates, and RAPs are presented in Table 3.¶
| Resolution * | Frame rate, fps | PAM |
|---|---|---|
| 2160p (4K),3840x2160 | 24/1.001, 24, 25, 30/1.001, 30, 50, 60/1.001, 60, 100, 120/1.001, 120. The set of frame rates presented in this table is taken from Table 2 in [PARAMETER] | RA |
| 1080p, 1920x1080 | RA | |
| 1080i, 1920x1080* | RA | |
| 720p, 1280x720 | RA | |
| 576p (EDTV), 720x576 | RA | |
| 576i (SDTV), 720x576* | RA | |
| 480p (EDTV), 720x480 | RA | |
| 480i (SDTV), 720x480* | RA |
*NB: Interlaced content can be handled at the higher system level and not necessarily by using specialized video coding tools. It is included in this table only for the sake of completeness, as most video content today is in a progressive format.¶
This is a form of video connection over the Internet. This form allows users to establish connections to two or more people by two- way video and audio transmission for communication in real time. For this application, both stationary and mobile devices can be used. The main requirements are as follows:¶
Support of resolution and quality (SNR) scalability is highly desirable. For this application, typical values of resolutions, frame rates, and RAPs are presented in Table 4.¶
| Resolution | Frame rate, fps | PAM |
|---|---|---|
| 1080p, 1920x1080 | 15, 30 | FIZD |
| 720p, 1280x720 | 30, 60 | FIZD |
| 4CIF, 704x576 | 30, 60 | FIZD |
| 4SIF, 704x480 | 30, 60 | FIZD |
| VGA, 640x480 | 30, 60 | FIZD |
| 360p, 640x360 | 30, 60 | FIZD |
This is a service that allows people to upload and share video data (using live streaming or not) and watch those videos. It is also known as video hosting. A typical User-Generated Content (UGC) scenario for this application is to capture video using mobile cameras such as GoPros or cameras integrated into smartphones (amateur video). The main requirements are as follows:¶
Support of resolution and quality (SNR) scalability is highly desirable. For this application, typical values of resolutions, frame rates, and RAPs are presented in Table 5.¶
This is a service that allows users to record and distribute computer-desktop screen output. This service requires efficient compression of computer-generated content with high visual quality up to visually and mathematically (numerically) lossless [HEVC-EXT]. Currently, this application includes business presentations (PowerPoint, Word documents, email messages, etc.), animation (cartoons), gaming content, and data visualization -- i.e., the type of content that is characterized by fast motion, rotation, smooth shade, 3D effect, highly saturated colors with full resolution, clear textures, sharp edges with distinct colors [HEVC-EXT]), virtual desktop infrastructure (VDI), screen/desktop sharing and collaboration, supervisory control and data acquisition (SCADA) display, automotive/navigation display, cloud gaming, factory automation display, wireless display, display wall, digital operating room (DiOR), etc. For this application, an important requirement is the support of low-delay configurations with zero structural delay for a wide range of video formats (e.g., RGB) in addition to YCbCr 4:2:0 and YCbCr 4:4:4 [HEVC-EXT]. For this application, typical values of resolutions, frame rates, and RAPs are presented in Table 6.¶
| Resolution | Frame rate, fps | PAM |
|---|---|---|
| Input color format: RGB 4:4:4 | ||
| 5k, 5120x2880 | 15, 30, 60 | AI, RA, FIZD |
| 4k, 3840x2160 | 15, 30, 60 | AI, RA, FIZD |
| WQXGA, 2560x1600 | 15, 30, 60 | AI, RA, FIZD |
| WUXGA, 1920x1200 | 15, 30, 60 | AI, RA, FIZD |
| WSXGA+, 1680x1050 | 15, 30, 60 | AI, RA, FIZD |
| WXGA, 1280x800 | 15, 30, 60 | AI, RA, FIZD |
| XGA, 1024x768 | 15, 30, 60 | AI, RA, FIZD |
| SVGA, 800x600 | 15, 30, 60 | AI, RA, FIZD |
| VGA, 640x480 | 15, 30, 60 | AI, RA, FIZD |
| Input color format: YCbCr 4:4:4 | ||
| 5k, 5120x2880 | 15, 30, 60 | AI, RA, FIZD |
| 4k, 3840x2160 | 15, 30, 60 | AI, RA, FIZD |
| 1440p (2K), 2560x1440 | 15, 30, 60 | AI, RA, FIZD |
| 1080p, 1920x1080 | 15, 30, 60 | AI, RA, FIZD |
| 720p, 1280x720 | 15, 30, 60 | AI, RA, FIZD |
This is a service that provides game content over the Internet to different local devices such as notebooks and gaming tablets. In this category of applications, the server renders 3D games in a cloud server and streams the game to any device with a wired or wireless broadband connection [GAME]. There are low-latency requirements for transmitting user interactions and receiving game data with a turnaround delay of less than 100 ms. This allows anyone to play (or resume) full-featured games from anywhere in the Internet [GAME]. An example of this application is Nvidia Grid [GAME]. Another category application is broadcast of video games played by people over the Internet in real time or for later viewing [GAME]. There are many companies, such as Twitch and YY in China, that enable game broadcasting [GAME]. Games typically contain a lot of sharp edges and large motion [GAME]. The main requirements are as follows:¶
Support of resolution and quality (SNR) scalability is highly desirable. For this application, typical values of resolutions, frame rates, and RAPs are similar to ones presented in Table 4.¶
This is a type of live broadcasting over IP-based networks. Video streams are sent to many receivers at the same time. A new receiver may connect to the stream at an arbitrary moment, so the random access period should be kept small enough (approximately, ~1-5 seconds). Data are transmitted publicly in the case of video monitoring and privately in the case of video surveillance, respectively. For IP cameras that have to capture, process, and encode video data, complexity -- including computational and hardware complexity, as well as memory bandwidth -- should be kept low to allow real-time processing. In addition, support of a high dynamic range and a monochrome mode (e.g., for infrared cameras) as well as resolution and quality (SNR) scalability is an essential requirement for video surveillance. In some use cases, high video-signal fidelity is required even after lossy compression. Typical values of resolutions, frame rates, and RAPs for video monitoring / surveillance applications are presented in Table 7.¶
| Resolution | Frame rate, fps | PAM |
|---|---|---|
| 2160p (4K),3840x2160 | 12, 25, 30 | RA, FIZD |
| 5Mpixels, 2560x1920 | 12, 25, 30 | RA, FIZD |
| 1080p, 1920x1080 | 25, 30 | RA, FIZD |
| 1.3Mpixels, 1280x960 | 25, 30 | RA, FIZD |
| 720p, 1280x720 | 25, 30 | RA, FIZD |
| SVGA, 800x600 | 25, 30 | RA, FIZD |
Taking the requirements discussed above for specific video applications, this section proposes requirements for an Internet video codec.¶
The most basic requirement is coding efficiency -- i.e., compression performance on both "easy" and "difficult" content for applications and use cases in Section 2. The codec should provide higher coding efficiency over state-of-the-art video codecs such as HEVC/H.265 and VP9 by at least 25%, in accordance with the methodology described in Section 4.1 of this document. For higher resolutions, the coding efficiency improvements are expected to be higher than for lower resolutions.¶
Good-quality specification and well-defined profiles and levels are required to enable device interoperability and facilitate decoder implementations. A profile consists of a subset of entire bitstream syntax elements, and consequently it also defines the necessary tools for decoding a conforming bitstream of that profile. A level imposes a set of numerical limits to the values of some syntax elements. An example of codec levels to be supported is presented in Table 8. An actual level definition should include constraints on features that impact the decoder complexity. For example, these features might be as follows: maximum bitrate, line buffer size, memory usage, etc.¶
| Level | Example picture resolution at highest frame rate |
|---|---|
| 1 | 128x96(12,288*)@30.0 176x144(25,344*)@15.0 |
| 2 | 352x288(101,376*)@30.0 |
| 3 | 352x288(101,376*)@60.0 640x360(230,400*)@30.0 |
| 4 | 640x360(230,400*)@60.0 960x540(518,400*)@30.0 |
| 5 | 720x576(414,720*)@75.0 960x540(518,400*)@60.0 1280x720(921,600*)@30.0 |
| 6 | 1,280x720(921,600*)@68.0 2,048x1,080(2,211,840*)@30.0 |
| 7 | 1,280x720(921,600*)@120.0 |
| 8 | 1,920x1,080(2,073,600*)@120.0 3,840x2,160(8,294,400*)@30.0 4,096x2,160(8,847,360*)@30.0 |
| 9 | 1,920x1,080(2,073,600*)@250.0 4,096x2,160(8,847,360*)@60.0 |
| 10 | 1,920x1,080(2,073,600*)@300.0 4,096x2,160(8,847,360*)@120.0 |
| 11 | 3,840x2,160(8,294,400*)@120.0 8,192x4,320(35,389,440*)@30.0 |
| 12 | 3,840x2,160(8,294,400*)@250.0 8,192x4,320(35,389,440*)@60.0 |
| 13 | 3,840x2,160(8,294,400*)@300.0 8,192x4,320(35,389,440*)@120.0 |
*NB: The quantities of pixels are presented for applications in which a picture can have an arbitrary size (e.g., screencasting)¶
Bitstream syntax should allow extensibility and backward compatibility. New features can be supported easily by using metadata (such as SEI messages, VUI, and headers) without affecting the bitstream compatibility with legacy decoders. A newer version of the decoder shall be able to play bitstreams of an older version of the same or lower profile and level.¶
A bitstream should have a model that allows easy parsing and identification of the sample components (such as Annex B of [ADVANCED] or [NAL]). In particular, information needed for packet handling (e.g., frame type) should not require parsing anything below the header level.¶
Perceptual-quality tools (such as adaptive QP and quantization matrices) should be supported by the codec bitstream.¶
The codec specification shall define a buffer model such as hypothetical reference decoder (HRD).¶
Specifications providing integration with system and delivery layers should be developed.¶
Color sampling formats:¶
| Profile | Bit depths per color component | Color sampling formats |
|---|---|---|
| 1 | 8 and 10 | 4:0:0 and 4:2:0 |
| 2 | 8 and 10 | 4:0:0, 4:2:0 and 4:4:4 |
| 3 | 8, 10 and 12 | 4:0:0, 4:2:0, 4:2:2 and 4:4:4 |
Support of configurations with zero structural delay, also referred to as "low-delay" configurations.¶
Tools that enable parallel processing (e.g., slices, tiles, and wave-front propagation processing) at both encoder and decoder sides are highly desirable for many applications.¶
Compression efficiency on noisy content, content with film grain, computer generated content, and low resolution materials is desirable.¶
As shown in Figure 1, compression-performance testing is performed in 3 overlapped ranges that encompass 10 different bitrate values:¶
Initially, for the codec selected as a reference one (e.g., HEVC or VP9), a set of 10 QP (quantization parameter) values should be specified as in [TESTING], and corresponding quality values should be calculated. In Figure 1, QP and quality values are denoted as QP0, QP1, QP2,..., QP8, QP9 and Q0, Q1, Q2,..., Q8, and Q9, respectively. To guarantee the overlaps of quality levels between the bitrate ranges of the reference and tested codecs, a quality-alignment procedure should be performed for each range's outermost (left- and rightmost) quality levels Qk of the reference codec (i.e., for Q0, Q3, Q6, and Q9) and the quality levels Q'k (i.e., Q'0, Q'3, Q'6, and Q'9) of the tested codec. Thus, these quality levels Q'k and, hence, the corresponding QP value QP'k (i.e., QP'0, QP'3, QP'6, and QP'9) of the tested codec should be selected using the following formulas:¶
Q'k = min { abs(Q'i - Qk) },
i in R
QP'k = argmin { abs(Q'i(QP'i) - Qk(QPk)) },
i in R
¶
where R is the range of the QP indexes of the tested codec -- i.e., the candidate Internet video codec. The inner quality levels (i.e., Q'1, Q'2, Q'4, Q'5, Q'7, and Q'8), as well as their corresponding QP values of each range (i.e., QP'1, QP'2, QP'4, QP'5, QP'7, and QP'8), should be as equidistantly spaced as possible between the left- and rightmost quality levels without explicitly mapping their values using the procedure described above.¶
QP'9 QP'8 QP'7 QP'6 QP'5 QP'4 QP'3 QP'2 QP'1 QP'0 <+-----
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ | Tested
| | | | | | | | | | | codec
Q'0 Q'1 Q'2 Q'3 Q'4 Q'5 Q'6 Q'7 Q'8 Q'9 <+-----
^ ^ ^ ^
| | | |
Q0 Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 <+-----
^ ^ ^ ^ ^ ^ ^ ^ ^ ^ | Reference
| | | | | | | | | | | codec
QP9 QP8 QP7 QP6 QP5 QP4 QP3 QP2 QP1 QP0 <+-----
+----------------+--------------+--------------+--------->
^ ^ ^ ^ Bitrate
|-------LBR------| |-----HBR------|
^ ^
|------MBR-----|
Since the QP mapping results may vary for different sequences, this quality alignment procedure eventually needs to be performed separately for each quality assessment index and each sequence used for codec-performance evaluation to fulfill the requirements described above.¶
To assess the quality of output (decoded) sequences, two indexes are separately computed, PSNR [IMAGE] and MS-SSIM [IMAGE][MULTI-SCALE]. In the case of the YCbCr color format, PSNR should be calculated for each color plane, whereas MS-SSIM is calculated for the luma channel only. In the case of the RGB color format, both metrics are computed for R, G, and B channels. Thus, for each sequence, 30 RD-points for PSNR (i.e., three RD-curves, one for each channel) and 10 RD-points for MS-SSIM (i.e., one RD-curve, for luma channel only) should be calculated in the case of YCbCr. If content is encoded as RGB, 60 RD-points (30 for PSNR and 30 for MS-SSIM) should be calculated -- i.e., three RD- curves (one for each channel) are computed for PSNR as well as three RD-curves (one for each channel) for MS-SSIM.¶
Finally, to obtain an integral estimation, BD-rate savings [PSNR] should be computed for each range and each quality index. In addition, average values over all 3 ranges should be provided for both PSNR and MS-SSIM. A list of video sequences that should be used for testing are defined in [TESTING], as are the 10 QP values for the reference codec. Testing processes should use the information on the codec applications presented in this document. As the reference for evaluation, state-of-the-art video codecs such as HEVC/H.265 [HETEROGENEOUS][CODING] or VP9 must be used. The reference source code of the HEVC/H.265 codec can be found at [HEVC]. The HEVC/H.265 codec must be configured according to [CONDITIONS] and Table 10.¶
| Intra-period, second | HEVC/H.265 encoding mode according to [CONDITIONS] |
|---|---|
| AI | Intra Main or Intra Main10 |
| RA | Random access Main or Random access Main10 |
| FIZD | Low delay Main or Low delay Main10 |
According to the coding efficiency requirement described in Section 3.1.1, BD-rate savings calculated for each color plane and averaged for all the video sequences used to test the NETVC codec should be, at least,¶
Since values of the two objective metrics (PSNR and MS-SSIM) are available for some color planes, each value should meet these coding efficiency requirements. That is, the final BD-rate saving denoted as S is calculated for a given color plane as follows:¶
S = min { S_psnr, S_ms-ssim },
¶
where S_psnr and S_ms-ssim are BD-rate savings calculated for the given color plane using PSNR and MS-SSIM metrics, respectively.¶
In addition to the objective quality measures defined above, subjective evaluation must also be performed for the final NETVC codec adoption. For subjective tests, the MOS-based evaluation procedure must be used as described in Section 2.1 of [IMAGE]. For perception-oriented tools that primarily impact subjective quality, additional tests may also be individually assigned even for intermediate evaluation, subject to a decision of the NETVC WG.¶
This document itself does not address any security considerations. However, it is worth noting that a codec implementation (for both an encoder and a decoder) should take into consideration the worst-case computational complexity, memory bandwidth, and physical memory size needed to process the potentially untrusted input (e.g., the decoded pictures used as references).¶
This document has no IANA actions.¶
The authors would like to thank Mr. Paul Coverdale, Mr. Vasily Rufitskiy, and Dr. Jianle Chen for many useful discussions on this document and their help while preparing it, as well as Mr. Mo Zanaty, Dr. Minhua Zhou, Dr. Ali Begen, Mr. Thomas Daede, Mr. Adam Roach, Dr. Thomas Davies, Mr. Jonathan Lennox, Dr. Timothy Terriberry, Mr. Peter Thatcher, Dr. Jean-Marc Valin, Mr. Roman Danyliw, Mr. Jack Moffitt, Mr. Greg Coppa, and Mr. Andrew Krupiczka for their valuable comments on different revisions of this document.¶