One-Click Upgrade from 2D to 3D: Sandwiched RGB-D Video Compression for Stereoscopic Teleconferencing (2024)

2.1 Multi-View Video Coding

Coding images and videos of one scene captured by multiple cameras has attracted broad interest in the past decades[21, 19, 25, 17].The core idea of recent achievements in multi-view video coding based on standard codecs[26, 24] is to utilize the established motion-compensation tools for cross-view predictive coding.This technique usually does not require significant change in the decoder structure. However, since the encoding process is complicated, there is a lack of an efficient implementation of a multi-view video encoder, and there are no practical solutions for low-delay encoding. Therefore, such techniques are still far from practical use in stereo video conferencing. Specifically, 3D-HEVC encodes both the depth map and color frames and shows better performance, but it requires major changes in the encoder and decoder pipeline to allow geometry-specific compensation, making it more challenging to deploy on consumer devices.

In our work, we propose a new scheme for cross-view and cross-modality coding that is more practical for real-world use than existing methods. Our scheme offloads the burden of cross-view and cross-modality coding to the pre- and post-processor, and adopts a general purpose video codec to handle temporal redundancy. This makes our scheme more efficient and easier to implement in hardware, while achieving a great improvement in rate-distortion performance.

2.2 Learned Stereo Image and Video Compression

With the proliferation of learned image and video codecs[1, 9, 15, 14, 18, 10], there has been a growing interest in using learning-based methods for stereo / volumetric image and video compression. In [13, 29, 16, 3, 27], neural network-based stereo image compression schemes are proposed. In [13, 16] the network architecture allows information sharing and compensation between the feature maps from two views, while in [29] bit-rate is reduced by coding a second view conditioned on the feature maps from a previously coded view. The idea of feature representation compensation has also been extended to stereo video coding[4], where the neural network conducts disparity and motion compensation simultaneously.

In direct contrast to these series of work, our work considers color and geometry joint coding and targets rendering quality. We show that it is beneficial to have the scheme consider color and geometry together to optimize rendering quality. Nevertheless, our scheme is further capable of achieving higher performance or higher efficiency given progress made to general video coding. For example, our pre- and post-processors can be combined with a state-of-the-art neural video codec in a plug-and-play fashion and achieve higher rate-distortion performance. Therefore, our scheme has more flexibility than existing learned stereo video compression works.

3.1 Sandwiched Video Compression Scheme

We aim to design a practical scheme to efficiently deliver stereo RGB-D video. The system should have the following desired characteristics:1) Simulcast support. This means that different views (i.e., left and right) and different modalities (i.e., RGB and depth) are transmitted simultaneously. This allows the system to reduce latency in a real-time video conferencing scenario.2) Smart bit-allocation. The system should should efficiently allocate bits between views and modalities to achieve an optimal rate-distortion tradeoff.3) Cross-view/cross-modality redundancy reduction. The system should be able to reduce cross-view/cross-modality redundancy for more efficient coding.4) Generalization to different camera configurations. The system should be able to generalize to different camera configurations, such as different baselines and different resolutions.

To achieve these goals, we build the stereo RGB-D video compression scheme by sandwiching a traditional video codec with preprocessing and post-processing neural networks, shown in Fig.2.In contrast to a video simulcast baseline that directly codes the RGB and depth frames independently, we employ a neural network preprocessor that takes the color and geometry as input and generates neural codes, in the format of video frames, to be coded by the video codecs. The transmitted neural codes are postprocessed by the neural postprocessor to reconstruct the color and geometry for client-side rendering. The pre- and post-processor are jointly trained neural networks. To make them generate neural codes suitable for conventional video coding, we employ an image codec proxy to enforce a rate-distortion constraint onto the neural codes. With the proxy, the preprocessor is trained to generate neural codes as color images that can be efficiently compressed by a real image codec. We show that such neural code generation generalize to other transform coding based hybrid video codecs (i.e. H264 / AVC and HEVC).

Our pre- and post-processor handle each frame in the video independently. Each stereo RGB-D frame includes RGB and depth maps in the left and right views. Since the left and right depth maps are defined in the left and right camera coordinate system, respectively, to make the preprocessor better explore cross-view redundancy, we apply a camera-to-world coordinate transformation on each of the depth maps, making the geometric inputs to the preprocessor lie in the canonical coordinate system (i.e. each pixel is endowed three coordinate values defined on a common 3D coordinate system). The preprocessor takes the 12-channel images as input, and produces $K$-channel bottleneck neural code images. After lossy encoding and decoding, either by the image proxy at the training time or a video codec in actual deployment, the post-processor converts the compressed bottleneck neural code images to the reconstructed RGB images and canonical space coordinates. The camera-specific world-to-camera transformations are applied to the coordinates to produce the depth maps, which together with the color images serve the novel view rendering.

We mainly choose $K=12$ in our experimental settings to provide enough flexibility for neural network optimization. Since 12 channels can be organized into 4 video streams, we are not coding extra motion vectors compared to a simulcast codec. Through end-to-end learning, the neural networks learn to occupy adequate number of channels and zero out certain channels to save bit-rates in a tight bandwidth budget. Nevertheless, one can set a smaller $K$ for a specific scenario that requires a lower number of video streams.

3.2 Preprocessor

The neural network-based pre- and post-processors are the key components in the proposed scheme to conduct smart bit-allocation and redundancy removal across views and modalities. We adopt the architecture design from other sandwiched codec schemes[6], which is composed of a U-Net and a multilayer perceptron (MLP) or equivalently multiple 1x1 2D convolutional layers. We specifically design the preprocessor architecture for multi-modality processing. The structure is illustrated in Fig.3.

It is discussed in [11] that the MLP usually does the per-pixel color space conversion, while the U-Nets add or remove modulation patterns. In our scenario, the color space transformation and the geometry transformation have different physical meanings. Therefore it is better to handle them separately. In our proposed design, two separate MLPs process the color and geometry maps, each in two views and 6 channels, respectively. Each of the MLP generates 6-channel neural code maps from the 6-channel input. To allow further cross-modality bit-allocation and rate reduction, a U-Net takes the concatenated RGB and coordinate maps as input, and generate 12-channel feature maps. These feature maps are split by half and added to the two groups of feature maps generated by the two MLP, respectively. The preprocessor produces feature maps (neural codes) in two groups, each with 6 channels. When using a video codec to encode the neural codes in each group, we organize them into 2 YUV 4:4:4 sequences. We choose the 2 channels that are highest in pixel variance as the luma component, and the other 4 channels are organized as the chroma channels. This is to best utilize motion estimation and compensation tools in modern video codecs.

3.3 Loss Function

We aim to find a distortion loss that well approximates the actual novel-view rendering distortion to enables effective rate-allocation between color and geometry, and at the same time is efficient for training. The distortion term in Eq.(1) is formulated as weighted sum of the color image distortion and depth distortion, where we specifically design the depth distortion, illustrated in Fig.4. We are given a ground truth depth map $Z$ and the reconstructed depth map $\hat{Z}$, both corresponding to the ground truth color image $I$. In the following, without loss of generality, we take the left view as an example. Terminology can be changed to the other view respectively.

4.1 Settings

Our method is designed for any 3D streaming system that transmits stereo RGB-D videos. In the following, we describe one of the design options.

The prospective teleconferencing system consists of an up-link and a down-link. In the up-link, the sender captures multi-view images of a person from multiple calibrated cameras, and send them to a server as video streams. With the calibrated multi-view images, the server conducts a 3D reconstruction process and build a high quality mesh. The server then renders stereo interpupillary distance (IPD) RGB-D images out of the mesh. With the down-link, the server streams the IPD stereo RGB-D video to the receiver. The color images will allow stereo perception at the receiver. The depth images further support more efficient interactive novel-view rendering and therefore provide more degrees of viewing freedom.

In this work, we particularly focus on the down-link compression problem. To make the system practical, the latency of the down-link compression system should be small. With this in mind, we specifically design our system to be standard hardware compatible. The decoder of our system can be practically implemented on a modern graphics card with hardware video decoder and GPU. The bit-stream produced by our system can be first decoded by an on-board standard decoder, and then directly processed by the GPU. Therefore, our system can be potentially practically deployed on existing commercial hardware and offers the upgrade from 2D to 3D video conferencing experience.

We simulate this VR video conferencing scenario to evaluate our compression scheme. In this scenario, we focus on the upper body of a human subject performing different actions. We assume that the background is static and in simple color. We evaluate the multi-novel-view rendering quality at the client side with different rate constraints.

4.2 Rate-Distortion Performance

4.2.1 Synthetic Testing Dataset

Sequence IDBD-Rate0-36.4 %1-39.9 %2-23.0 %3-37.4 %4-23.9 %5-33.7 %6-33.3 %7-7.6 %8-31.1 %9-26.1 %Avg-29.3 %
Sandwiched / H264

Sequence IDBD-Rate0-32.5 %1-40.0 %2-23.5 %3-35.2 %4-20.7 %5-28.3 %6-28.0 %7-10.2 %8-28.7 %9-24.3 %Avg-27.1 %
Sandwiched / HEVC

We first demonstrate the improvement in rendering rate-distortion performance on the synthetic testing dataset. It is shown in Fig.5 that our scheme, although trained with simple JPEG proxy, works well with more sophisticated video codecs and brings significant improvements in rate-distortion performance. We also include results from MV-HEVC in the comparison. However, MV-HEVC only supports YUV 4:2:0, making it less competitive at the higher bit-rate region.

Table1 shows the quantitative BD-Rate results, calculated w.r.t. each testing sequence in the PSNR range 32 to 50 dB. Each table presents the bit-rate reduction given by sandwiching the specified codec (i.e. H264 and HEVC) with our pre- and post-processor. As shown, our proposed method steadily achieves over 29.3% bit-rate saving on most of the sequences with H264. Similar gain is observed with HEVC.

4.2.2 Real Captured Data

To demonstrate that our models generalize well to real use cases, we evaluate our method on sequences obtained from The Relightables dataset. As shown by the R-D curves in Fig.6, although the models are trained on synthetic dataset, they generalize well on real captured scenes.

4.3 Ablation Study

4.3.1 Loss Function

We analyze the impact of distortion term in the loss function by setting different values for $\alpha$ and $\beta$ in Eq.4. Fig.8 shows the comparison between different loss function settings, where MSE means $\alpha=0$ and $\beta=1$. In all other settings when we vary $\beta$, we set $\alpha=1$. We can see that models trained with RGB-D MSE loss under-perform the H264 baseline. By adding the $L_{\text{warp}}$ term in the distortion loss, the rendering performance is overall improved. The results also show that it may be beneficial to reasonably reduce the weight for the depth MSE term when $L_{\text{warp}}$ is used, but the rate-distortion performance is not very sensitive to $\beta$. We use $\alpha=1$ and $\beta=0.1$ as the main setting.

4.3.2 Architecture

In the proposed scheme, we transform the depth values to the canonical coordinate space, and employ one U-Net with two MLPs to process the color and geometry, respectively.In this experiment, we study configurations with one U-Net but different numbers of MLPs for the preprocessor, i.e.,

• •

  Two MLPs. The main architecture shown in Fig.2.

• •

  Four MLPs. We employ four MLPs in the preprocessor besides the U-Net, for different views (left and right) and different modalities (color and geometries). Each of the MLPs takes 3 channels as input and produce 3 output channels.

• •

  Single MLP. One U-Net with one MLP, i.e. the default architecture in [6].

• •

  Geometry: Depth. The preprocessor takes depth maps as input and does not transform them to the canonical space.

• •

  Geometry: Camera Coordinates. Camera-space 3D coordinates constructed from the depth maps are used as the input. The coordinates from the two views are in different coordinate space (i.e. the left and right camera space, respectively).

• •

  Color Only. We only use the proposed scheme to encode and decode the left and right color maps. The depth maps are coded by H264.

To better analyze different settings in different bit-rate ranges, we divide the valid PSNR range into two sub-ranges, i.e. lower bit-rate (32 - 42 dB) and higher bit-rate (42 - 52 dB). We evaluate BD-Rate in different ranges for comparison. The results are shown in Table.2. We can see that having separate MLPs for the color and geometry greatly benefits the performance, especially for the higher bit-rates. Further splitting the MLP for different views brings a little more gains but also increase the complexity. Hence we basically adopt the two-MLP architecture. It is also shown that transforming coordinates to the canonical space greatly helps improve performance at the higher bit-rates. Finally, by comparing the color only scheme to the proposed one that jointly handles color and geometry, we show that in the scenario of stereo RGB-D compression for rendering, it is important to jointly process and compress the color and geometry. Existing methods that handle color and geometry separately may be suboptimal.

Bit-Rate RangeLowerHigherTwo MLPs-25.0 %-29.8 %Four MLPs-28.8 %-29.0 %Single MLP-22.1 %-12.7 %Geometry: Depth-27.4 %-20.6 %Geometry: Camera Coordinates-24.1 %-20.1 %Color Only:-9.5 %-19.3 %

4.4 Qualitative Results

As shown by the R-D curves, our coding scheme is capable of adapting to different bandwidth constraints. In a low bandwidth scenario, we are interested in the video quality from a nearby novel view point.With more bandwidth resources, we evaluate our methods in a novel view that is further different from the original ones.

6.1 Architecture Details

In this section, we present more details on the neural network architecture used in our method. As mentioned in the main paper, our method is based on the sandwiched codec, which has a neural preprocessor and a postprocessor. Both neural processors are constructed using the U-Net architecture[22] and the multi-layer perceptron (MLP) architecture. As shown in Fig.3 from the main paper, the preprocessor is composed of one U-Net (see architecture details in Table4) and two MLPs (see architecture details in Table5 with $C=6$ for each of them). The post-processor is composed of one U-Net (see architecture details in Table4) and one MLP (see architecture details in Table5 with $C=12$).

6.2 Visualization of Neural Codes

We visualize the learned neural code generated by the preprocessor, with a low-bit-rate model (in Fig.10) and a high-bit-rate model (in Fig.11), respectively. As described in the main paper, we organize the 12 latent channels into 4 groups, each group containing 3 channels, corresponding to the $Y^{\prime}$, $U^{\prime}$, and $V^{\prime}$ channels of the YUV color space. As shown, at higher bit-rates, the preprocessor learns to maintain more detailed information by fully utilizing the 12 neural code channels. In contrast, at lower bit-rates, the preprocessor learns to compress the information by compressing the input stereo RGB-D information into fewer channels and produce high frequency modulation in the compact latent channels. The resulting neural code channels are thus more sparse and different from the human perceptable RGB-D information.

6.3 Extension: Relightability

Method

Normal Map

Relighted

Method

Normal Map

Relighted

Ground
Truth

Ground
Truth

H264
0.78 mbps

H264
1.46 mbps

Ours
0.76 mbps

Ours
1.40 mbps

Since decoder side relightability is sometimes desired in a scene transmission pipeline. In this section, we demonstrate a simple extension of our method to support decoder-side relighting by transmitting normal maps together with the RGB-D signal in our sandwiched codec. We qualitatively evaluate the extension setting by relighting the transmitted 3D representation. Results shown in Fig.12 demonstrate that our method can be easily extended to a relighting rendering pipeline and maintains better performance.