Transport over video links and IP networks

Video bandwidth requirements are growing continuously, as video resolutions, frame rates, bit depths and the amount of video streams are constantly increasing. Likewise, the capacities of video links and communication channels are also growing, yet at a slower pace than what is needed to address the huge video bandwidth growth. In addition, the investments to upgrade the capacity of links and channels are significant and need to be amortized over several years.

Moreover, both the broadcast and pro-AV markets are shifting towards AV-over-IP based infrastructure, with a preference going to 1 Gigabit Ethernet links for remote production or 10G Ethernet networks for in-house facilities. Both 1G, 2.5G and 10G Ethernet are cheap and ubiquitous, while 25G or better links are usually not yet affordable. Given the available bandwidth and infrastructure cost, relying on uncompressed video is therefore no longer an option, as 4K, 8K, increased bit depths (for HDR), and higher framerates need to be supported.

JPEG XS offers a light-weight compression that visually preserves the quality compared to an uncompressed stream, at a low cost, targeted at compression ratios of up to 10:1. With XS, it is for example possible to repurpose existing SDI cables to transport 4K60 over a single 3G-SDI (at 4:1), and even over a single HD-SDI (at 8:1). Similar scenarios can be used to transport 8K60 content over various SDI cable types (e.g. 6G-SDI and 12G-SDI). Alternatively, XS enables transporting 4K60 content over 1G Ethernet and 8K60 over 5G or 10G Ethernet, which would be impossible without compression. The following table shows some expected compression ranges for some typical use cases.

Real-time video storage and playout

Related to the transport of video streams is the storage and retrieval of high resolution streams where bandwidth limitations similarly apply. For instance, video cameras use internal storage like SSD drives or SD cards to hold large streams of images, yet the maximum data rates of such storage devices are limited and well below the uncompressed video throughput.

Sensor compression

As stated, JPEG XS offers built-in support for the direct compression of RAW Bayer/CFA images using the Star-Tetrix Color Transform. This transform takes a RAW Bayer pattern image and decorrelates the samples into a 4-component image with each component having only a quarter of the resolution.^[15] This means that the total amount of samples to further process and compress remains the same, yet the values are decorrerlated in a similar fashion to a classical Multiple Component Transform.

Avoiding such conversion prevents loss of information and allows that this processing step be done outside of the camera. This is advantageous because it allows to defer demosaicing the Bayer content from the moment of capturing to the production phase, where choices regarding artistic intent and various settings can be better made. Recall that the demosaicing process is irreversible and requires certain choices, like the choice of interpolation algorithm or the level of noise reduction, to be made up front. Moreover, the demosaicing process can be power-hungry and will also introduce extra latency and complexity. The ability to push this step out of the camera is possible with JPEG XS and allows to use more advanced algorithms resulting in better quality in the end.

JPEG XS (ISO/IEC 21122)

The JPEG XS coding system is an ISO/IEC suite of standards that consists of the following parts:

Part 1, formally designated as ISO/IEC 21122-1, describes the core coding system of JPEG XS. This standard defines the syntax and, similarly to other JPEG and MPEG image codecs, the decompression process to reconstruct a continuous-tone digital image from its encoded codestream. Part 1 does provide some guidelines of the inverse process that compresses a digital image into a compressed codestream, or more simply called the encoding process, but leaves implementation specific optimizations and choices to the implementers.

Part 2 (ISO/IEC 21122-2) builds on top of Part 1 to segregate different applications and uses of JPEG XS into reduced coding tool subsets with tighter constraints. The definition of profiles, levels and sublevels allows reducing the complexity of implementations in particular application use cases, while also safeguarding interoperability. Recall that lower complexity typically means less power consumption, lower production costs, easier constraints, etc. Profiles represent interoperability subsets of the codestream syntax specified in Part 1. In addition, levels and sublevels provide limits to the maximum throughput in respectively the encoded (codestream) and the decoded (spatial and pixels) image domains. Part 2 furthermore also specifies a buffer model, consisting of a decoder model and a transmission channel model, in order to enable guaranteeing low latency requirements to a fraction of the frame size.

Part 3 (ISO/IEC 21122-3) specifies transport and container formats for JPEG XS codestreams. It defines carriage of important metadata, like color spaces, mastering display metadata (MDM), and EXIF, in order to facilitate transport, editing and presentation. Furthermore, this part defines the XS-specific ISOBMFF boxes, an Internet Media Type registration and additional syntax to allow embedding XS in formats like MP4, MPEG-2 TS, or the HEIF image file format.

Part 4 (ISO/IEC 21122-4) is a supporting standard of JPEG XS that provides conformance testing and buffer model verification. This standard is crucial to implementers of XS and appliance conformance testing.

And finally, Part 5 (ISO/IEC 21122-5) represents a reference software implementation (written in ISO C11) of the JPEG XS Part 1 decoder, conforming to the Part 2 profiles, levels and sublevels, as well as an exemplary encoder implementation.

A second edition of all five parts is in the making and will be published at latest in the beginning of 2022. It provides additional coding tools, profiles and levels, and new reference software to add support for efficient compression of 4:2:0 content, RAW Bayer/CFA content, and mathematically lossless compression.

RFC9134 - RTP Payload Format for ISO/IEC 21122 (JPEG XS)

RFC 9134^[16] describes a payload format for the Real-Time Transport Protocol (RTP, RFC 3550^[17]) to carry JPEG XS encoded video. In addition, the recommendation also registers the official Media Type Registration for JPEG XS video as video/jxsv, along with its mapping of all parameters into the Session Description Protocol (SDP).

The RTP Payload Format for JPEG XS in turn enables using JPEG XS in SMPTE ST 2110 environments using SMPTE ST 2110-22 for CBR compressed video transport.

MPEG-TS for JPEG XS

ISO/IEC 13818-1:2022, known as MPEG-TS 8th edition, specifies carriage support for JPEG XS in MPEG Transport Streams.^[18] See also MPEG-2. Note that AMD1 (Carriage of LCEVC and other improvements) of ISO/IEC 13818-1:2022 contains some additional corrections, improvements and clarifications regarding embedding JPEG XS in MPEG-TS.^[19]

VSF TR-07 and TR-08

See VSF TR-07^[20] and TR-08,^[21] published by the Video Services Forum

NMOS with JPEG XS

A Networked Media Open Specifications that enables registration, discovery, and connection management of JPEG XS endpoints using the AMWA IS-04 and IS-05 NMOS Specifications. See AMWA BCP-006-01,^[22] published by Advanced Media Workflow Association.

JPEG XS in IPMX

Internet Protocol Media Experience (IPMX) is a proposed set of open standards and specifications to enable the carriage of compressed and uncompressed video, audio and data over IP networks for the pro AV market. JPEG XS is supported under IPMX via VSF TR-10-8^[23] and TR-10-11.^[24]

Core coding

The JPEG XS standard is a classical wavelet-based still-image codec without any frame buffer. While the standard^[26] defines JPEG XS on the basis of a hypothetical reference coder, JPEG XS is easier to explain through the steps a typical encoder performs:^[27]

Component up-scaling and optional component decorrelation: In the first step, the DC gain of the input data is removed and it is upscaled to a bit-precision of 20 bits. Optionally, a multi-component generation, identical to the JPEG 2000 RCT, is applied. This transformation is a lossless approximation of an RGB to YUV conversion, generating one luma and two chroma channels.

Wavelet transformation: Input data is spacially decorrelated by a 5/3 Daubechies wavelet filter. While a five-stage transformation is performed in horizontal direction, only 0 to 2 transformations are run in vertical direction. The reason for this asymmetrical filter is to minimize latency.

Prequantization: The output of the wavelet filter is converted to a sign-magnitude representation and pre-quantized by a deadzone Quantizer to 16 bit precision.

Rate control and quantization: The encoder determines by a non-normative process^[27] the rate of each possible quantization setting and then quantizes data by either a deadzone Quantizer or a data dependent uniform Quantizer.

Entropy coding: JPEG XS uses minimalistic Entropy encoding for the quantized data which proceeds in up to four passes over horizontal lines of quantized wavelet coefficients. The steps are:

• Significance coding: In the (optional) first pass, the significance of 32 consecutive wavelet coefficients is coded by a single bit.
• Bitplane count coding: In the second pass, the number of non-zero bitplanes of groups of four coefficients each, the so-called "bitplane count", is entropy coded through a Golomb type code. This step may optionally use the bitplane counts of the preceding line as source for prediction (Differential pulse-code modulation) and then encodes only the prediction difference.
• Data coding:The third pass inserts the raw bitplane values into the codestream without further coding.
• Sign coding: In the last optional coding pass, the sign bits of all non-zero coefficients is inserted into the codestream. If this coding pass is not present, sign bits are included in the data coding pass for all coefficients.

Codestream packing: All entropy coded data are packed into a linear stream of bits (grouped in byte multiples) along with all of the required image metadata. This sequence of bytes is called the codestream and its high-level syntax is based on the typical JPEG markers and marker segments syntax.^[28]

Profiles, levels and sublevels

JPEG XS defines profiles (in ISO/IEC 21122-2) that define subsets of coding tools which conforming decoders shall support, by limiting the permitted parameter values and allowed markers. The following table represents an overview of all the profiles along with their most important properties. Please refer to the standard for a complete specification of each profile.

In addition, JPEG XS defines levels to represent a lower bound on the required throughput that conforming decoders need to support in the decoded image domain (also called the spatial domain). The following table lists the levels as defined by JPEG XS. The maximums are given in the context of the sampling grid, so they refer to a per pixel value where each pixel represents one or more component values. However, in the context of Bayer data JPEG XS internally interprets the Bayer pattern as an interleaved grid of four components. This means that the number of sampling grid points required to represent a Bayer image is four times smaller than the total number of Bayer sample points. Each group of 2x2 (four) Bayer values gets interpreted as one sampling grid point with four components. Thus sensor resolutions should be divided by four to calculate the respective width, height and amount of sampling grid points. For this reason, all levels also bear double names. Please refer to the standard for a complete specification of each level.

Similarly to the concept of levels, JPEG XS defines sublevels to represent a lower bound on the required throughput that conforming decoders need to support in the encoded image domain. Each sublevel is defined by a nominal bits per pixel (Nbpp) value that indicates the maximum amount of bits per pixel for an encoded image of maximum permissible number of sampling grid points according to the selected conformance level. Thus, decoders conforming to a particular level and sublevel shall conform to the following constraints derived from Nbpp:

• The maximum codestream size in bytes (from SOC to EOC, including all markers) is ${\displaystyle S_{sl,max}={\bigg \lfloor }{\frac {L_{max}\times N_{bpp}}{8}}{\bigg \rfloor }}$.
• The maximum admissible encoded throughput in bits per second is ${\displaystyle R_{t,max}=R_{s,max}\times N_{bpp}}$.

The following table lists the existing sublevels and their respective nominal bpp values. Please refer to the standard for a complete specification of each level.