A) The AMD using estimated rate

The techniques for mode decision as employed in our codec extend the method in [Reference Slowack13]. Let X denote the original WZ frame and Y denote the side information frame. The cost for WZ coding a coefficient X _k with index k in a particular coefficient band is defined as [Reference Slowack13]:

(1)$$C_{{WZ}}^{k}=H\lpar Q\lpar X_{k}\rpar \vert Y_{k}=y_{k}\rpar +\lambda {\rm E}[ \vert X_{k}-\widehat{X}_{k}\Vert Y_{k}=y_{k}]. $$

The first term in this sum denotes the conditional entropy of the quantized coefficient Q(X _k) given the side information. The second term consists of the Lagrange parameter λ multiplied by the mean absolute distortion between the original coefficient X _k and its reconstruction $\widehat{X}_{k}$, given the side information. Entropy and distortion are calculated as in [Reference Slowack13].

To calculate cost for skipping using (1) for the coefficient X _k [Reference Slowack13], we set the entropy, H() = 0, representing the variable contribution after coding the mode. This gives:

(2)$$C_{skip}^{k}=\lambda \displaystyle{{1} \over {\alpha}}\comma \;$$

where α is the noise parameter and 1/α gives the expected value, E[].

Often RD optimization in video coding is based on a Lagrangian expression J = D + λR, where D is the distortion and the rate. The expression we use (1) is, in these terms, based on the cost C = R + λD. One reason is that in skip mode R is small, thus by shifting lambda to the distortion term, the exact value of R is less important and we can even set the contribution of having coded the mode to 0 for skip.

If C _skip^k < C _WZ^k for all coefficients in a coefficient band, all bitplanes in the coefficient band are skipped and the side information is used as the result. Otherwise, bitplane-level mode decision is performed to decide between bitplane-level skip, intra, or WZ coding as described in [Reference Slowack13]. The coding mode for each bitplane is communicated to the encoder through the feedback channel. It can be remarked that the mode information for each band is coded by 1 bit, e.g. 0 for skipped and 1 for not skipped. Thereafter, for a band which is not skipped, the information for each bitplane mode is coded by two bits for skip, intra, and WZ modes. Thus the number of feedback instances is reduced especially for skip coding at band level, but also for skip and intra at the bitplane level. We shall include the mode decision feedback bits in the code length when reporting results. Depending on the number of bands and corresponding bitplanes which are used for each QP point, the contribution by mode decision to the rates is relatively small compared to the total coding rate. For example, the mode information for QP 8, which has 15 bands coded in 63 bitplanes, needs the highest bits with 141 bits at most for coding modes (1 × 15 bands + 2 × 63 bitplanes not skipped).

One of the contributions in this paper is to extend the method above. Instead of using a sequence-independent formula for λ as in [Reference Slowack13], we propose to vary the Lagrange parameter depending on the sequence characteristics.

As a first step, results are generated for a range of lambdas and WZ quantization points, using the sequences Foreman, Coastguard, Hall Monitor, and Soccer (QCIF, 15Hz, and GOP2), which are typical for DVC, for training. Wherever necessary, the intra quantization parameter (QP) of the key frames is adjusted, so that the qualities of WZ frames and intra frames are comparable (i.e., within a 0.3 dB difference) for each of the RD points. For each sequence and WZ quantization matrix, the optimal lambda(s) are identified by selecting the set providing the best RD curve. These points are then used to create a graph of (optimal) lambdas as a function of the intra QP, as in Fig. 2. For each test sequence, the points were fitted with a continuous exponential function, where it can be noted that four reasonable QP points are considered sufficient in this work. This results in an approximation of the optimal lambda as a function of the intra QP, for each test sequence, i.e.

Fig. 2. Experiments on optimal λ.

(3)$$\lambda=a\, e^{-b\cdot {\rm QP}}\comma \; $$

where QP denotes the intra QP of the key frames, and a and b are constants. The optimal λ is obtained by the work in [Reference Slowack13] with fixed a = 7.6 and b = 0.1 for all sequences.

As shown in Fig. 2, the optimal λ differs among the sequences. Typically, for sequences with less motion (such as Hall Monitor), the optimal λ is lower to give more weight to the rate term in (1) and consequently encourage skip mode. On the other hand, for sequences with complex motion such as Soccer, the distortion introduced in the case of skip mode is significant due to errors in the side information, so that higher values for λ give better RD results.

The results in Fig. 2 are exploited to estimate the optimal λ on a frame-by-frame basis during decoding. The approach taken is – relatively simple – to look at the rate. Apart from the graph (Fig. 2) we also store the average rate per WZ frame associated with each of the points. For sequences with simple motion characteristics (e.g., Hall Monitor, Coastguard), for the same intra QP, the WZ rate is typically lower than for more complex sequences such as Foreman and Soccer. Therefore, during decoding, we first estimate the WZ rate and compare this estimate with the results in Fig. 2 to estimate the optimal lambda. Specifically, the WZ rate r _i for the current frame is estimated as the median (med ) of the WZ rates r _i−3, r _i−2r _i−1 of the three previously decoded WZ frames (as in [Reference Slowack, Skorupa, Deligiannis, Lambert, Munteanu and Van de Walle18]):

(4)$$r_{i}= \hbox{med}\lpar r_{i-1}\comma \; r_{i-2}\comma \; r_{i-3}\rpar .$$

It can be noted that the first three WZ frames are coded using only intra and skip mode as in [Reference Slowack, Skorupa, Deligiannis, Lambert, Munteanu and Van de Walle18]. The estimated r _i (4) is compared with rate points from the training sequences, which are shown in Fig. 2. We then obtain an estimate of the optimal lambda parameter for the current WZ frame to be decoded through interpolation.

In the training step, it may be noted that the optimal λs (in Fig. 2) are obtained along with the corresponding rate points. It is assumed that we have found the two closest rate points r ₁, r ₂, r ₁ ≤ r _i ≤ r ₂, from the training sequences with the corresponding λ_{r 1}, λ_{r 2}, respectively. By means of a linear interpolation, the relations are expressed as:

(5)$$\displaystyle{{\lambda_{r_{i}}-\lambda_{r_{1}}} \over {r_{i}-r_{1}}} = \displaystyle{{\lambda_{r_{2}}-\lambda_{r_{i}}} \over {r_{2}-r_{i}}}.$$

As a result, we obtained λ_{r i} by

(6)$$\lambda_{r_{i}}= \displaystyle{{r_{i}-r_{1}} \over {r_{2}-r_{1}}} \lambda_{r_{2}}+ \displaystyle{{r_{2}-r_{i}} \over {r_{2}-r_{1}}} \lambda_{r_{1}}.$$

In summary, we can obtain λ_{r i} for each WZ frame with the estimated rate r _i given the optimal λ versus IntraQP (in Fig. 2) and rate points from the training sequences as follows:

• • Estimating the rate r _i of the WZ frame based on the three previously decoded WZ frames by (4);

• • Looking up the given rate points of the training sequences to get the two closest rate points r ₁, r ₂ with the corresponding λ_{r 1}, λ_{r 2} satisfying r ₁ ≤ r _i ≤ r ₂;

• • Obtaining λ_{r i} by interpolation given by equation (6).

B) The residual MC

Noise modeling is one of the main issues impacting the accuracy of mode decisions. Both the WZ and skip costs as in (1) and (2) depend on the α parameter of the noise modeling. To improve performance and the noise modeling, this paper integrates the AMD (Section IIIA) with a technique exploiting information from previously decoded frames based on the assumption of useful correlation between the previous and current residual frames [Reference Luong, Rakêt and Forchhammer7]. This correlation was initially experimentally observed. This correlation can be expressed using the motion between the previous residue and the current residue, which we may hope to be similar to the motion between the previous SI and the current SI. This technique generates residual frames by compensating the motion between the previous SI frames and the current SI frame to the current residual frame to generate a more accurate noise distribution for noise modeling.

For a GOP of size two, let $\widehat{X}_{2n-2\omega}$ and $\widehat{X}_{2n}$ denote two decoded WZ frames at time 2n − 2ω and 2n, where ω denotes the index of the previously decoded ωth WZ frame before the current WZ frame at time 2n. Their associated SI frames are denoted by Y _2n−2ω and Y _2n, respectively. For objects that appear in the previous and current WZ frames, we expect the quality of the estimated SI, expressed by the distribution parameter to be similar. We shall try to capture this correlation using MC from frame 2n − 2ω to frame 2n. The motion between two the SI frames provides a way to capture this correlation. Here, each frame is split into N non-overlapped 8 × 8 blocks indexed by k, where 1 ≤ k ≤ N. It makes sense to assume that the motion vector v _k of block k at position z _k between $\widehat{X}_{2n-2\omega}$ and $\widehat{X}_{2n}$ is the same as between Y _2n−2ω and Y _2n. This is represented as follows:

(7)$$Y_{2n}\lpar z_{k}\rpar \approx Y_{2n-2\omega}\lpar z_{k}+v_{k}\rpar .$$

A motion compensated estimate of $\widehat{X}_{2n}$ based on the motion v _k, $\widehat{X}_{2n}^{MC}$, can be obtained by

(8)$$\widehat{X}_{2n}^{MC}\lpar z_{k}\rpar = \widehat{X}_{2n-2\omega}\lpar z_{k}+v_{k}\rpar \comma \;$$

Based on the estimated SI frames Y _2n−2ω and Y _2n, the vectors v _k are calculated using (7) within a search range (Φ of [16 × 16] pixels) as

(9)$$v_{k}=\arg \mathop {\min }\limits_{v \in \Phi } \sum\nolimits_{\rm block}\lpar Y_{2n}\lpar z_{k}\rpar -Y_{2n-2\omega}\lpar z_{k}+v\rpar \rpar ^{2}\comma \;$$

where $\sum\nolimits_{\rm block}$ is the sum over all pixel positions z _k. Thereafter, $\widehat{X}_{2n}^{MC}$ is estimated by compensating $\widehat{X}_{2n-2\omega}$ (8) for the selected motion v (9). Let R _2n denote the current residue at time 2n, generated by OBMC, and let $\widehat{R}_{2n}^{MC}$ denote the motion compensated residue, where R _2n and $\widehat{R}_{2n}^{MC}$ are equivalent to R ₀ and R ₁ (Section II, Fig. 1). Other motion estimation techniques may also be applied, e.g. OF [Reference Luong, Rakêt, Huang and Forchhammer17]. In the tests (Section IV), we shall apply both OBMC and OF. $\widehat{R}_{2n}^{MC}$ can be estimated from $\widehat{X}_{2n}^{MC}$ and Y _2n as follows:

(10)$$\widehat{R}_{2n}^{MC}\lpar z_{k}\rpar =\widehat{X}_{2n}^{MC}\lpar z_{k}\rpar -Y_{2n}\lpar z_{k}\rpar.$$

Finally, the compensated residue is obtained by inserting (8) in (10)

(11)$$\widehat{R}_{2n}^{MC}\lpar z_{k}\rpar = \widehat{X}_{2n-2\omega}\lpar z_{k}+v_{k}\rpar -Y_{2n}\lpar z_{k}\rpar .$$

A motion compensated residue $\widehat{R}_{18}^{MC}$ (11) is predicted based on the decoded frame $\widehat{X}_{2n-2}$ and the motion v between the SI frames Y _2n and Y _2n−2. To show the efficiency of the proposed technique, we calculate a difference between the motion compensated residue and an ideal residue calculated by X _2n − Y _2n−2 and compared this with a difference between the OBMC residue and the ideal residue. Figure 3 illustrates the frame by frame mean-square error (MSE) for Soccer (key frames QP=26) in order to compare the MSE between the OBMC residue and the ideal residue with the MSE between the motion compensated residue, denoted Motion, and the ideal residue. The MSE for Motion in Fig. 3 is consistently smaller than the MSE of the OBMC, i.e. the Motion residue is closer to the ideal residue than the OBMC residue.

Fig. 3. MSE (denoted OBMC) between the OBMC residue and the ideal residue versus MSE (denoted Motion) between the motion compensated residue and the ideal residue (for Frame 18 of Soccer).

C) The AMD MORE2SI codec

In order to further enhance the RD performance and test AMD, we shall also integrate AMD into the state-of-the-art, but also more complex, MORE2SI codec [Reference Luong, Rakêt and Forchhammer7], which is based on the SING2SI scheme [Reference Luong, Rakêt, Huang and Forchhammer17] additionally employing motion and residual re-estimation and a generalized reconstruction (Fig. 4). The MORE2SI scheme is here enhanced by integrating the AMD using the (decoder side) estimated rate of WZ frames to obtain a Lagrange parameter (Section IIIA). Figure 4 depicts the Adaptive Mode Decision MORE architecture using 2SI, which combines the powers of the MORE2SI scheme [Reference Luong, Rakêt and Forchhammer7] and the AMD technique (Sections IIIA + B) determining the three modes skip, arithmetic, or WZ coding of each bitplane. Initial experiments are reported in Section IV. It may be noted (Fig. 4) that the MORE2SI scheme [Reference Luong, Rakêt and Forchhammer7] applies OF as well as OBMC in the SI generation.

Fig. 4. Adaptive mode decision MORE video architecture.