A) Intra block copy (intraBC)

Motion compensation is one of the key technologies in modern video coding. The correlation between adjacent pictures has been investigated in various efficient ways in the literature to reduce the bandwidth of representing the video signal. Similar concept has also been tried to allow block matching and copy within the same picture. It was not very successful when applying this concept to camera-captured video. Part of the reasons is that the textual pattern in a spatial neighboring area may be similar to the current coding block but usually with gradual changes over space. It is thus difficult for a block to find an exact match within the same picture; therefore the improvement in coding performance is limited. However, the spatial correlation among pixels within the same picture is different for screen content. For a typical video with text and graphics, there are usually repetitive patterns within the same picture. Hence, intra-picture block copy becomes possible and has been proved to be very effective. A new prediction mode, i.e. intraBC mode, is thus introduced to utilize this characteristic. In the intraBC mode, a prediction unit (PU) is predicted from a previously reconstructed block within the same picture. Similar to a PU in motion compensation, a displacement vector (called block vector or BV) is used to signal the relative displacement from the position of the current PU to that of the reference block. The prediction errors after compensation are then coded in the same fashion as how the inter residues are coded in HEVC version 1. An example of intraBC compensation is illustrated in Fig. 1.

Fig. 1. An example of intra block copy and compensation.

Despite the similarity, there are a few aspects that differentiate the intraBC mode from the inter mode in HEVC version 1:

2) BV prediction and coding

New methods for predicting and coding intraBC BVs are proposed, as opposed to the methods for predicting and coding motion vectors in HEVC version 1, for further improving the coding efficiency of HEVC SCC. In the new BV prediction scheme, one candidate from the left and one from the top neighbors are used as the two primary predictors for BV prediction [Reference Pang11,Reference Xu, Liu, Chuang and Lei12]. In the case when spatial neighbors are not available, e.g. neighboring blocks are coded in regular intra or inter mode instead of the intraBC mode, or hit the picture or slice boundaries, two last coded BVs are used to fill the candidate list. At the beginning of each coding tree unit (CTU), these two so-called last coded BVs are initialized using constant values. A 1-bit flag is used to signal one of the two candidates that is used to predict the current BV. As illustrated in Fig. 2(a), the two spatial neighboring positions a1 and b1 are used as the primary predictor candidates. Figure 2(b) shows an example of intraBC BV predictor list construction, where the spatial neighbors for the current block are block 9 and block 4. If only blocks 2 and 8 are intraBC coded while all others are not, then the BVs of these two blocks are considered as two last coded BVs and used as the predictor candidates for the current block. If in addition block 4 is intraBC coded, then the BVs of block 4 (spatial neighbor) and block 8 (last coded) are used as the two BV predictor candidates.

Fig. 2. An example of intra block copy, block vector prediction. (a) Spatial neighboring positions, where a1 and b1 positions are selected as the spatial candidates for block vector prediction. (b) An example of BV predictor candidate list construction.

The BV prediction difference needs to be further entropy coded. A new binarization method was proposed in [Reference Pang, Rapaka, Sole and Karczewicz13], as opposed to the binarization method for coding the motion vector prediction difference as in HEVC version 1. In this method, a 1-bit flag is context coded to signal whether the BV component being coded, either x or y, is 0. If it is not zero, the magnitude of this component is bypass coded using Exponential-Golomb code order 3, followed by its sign.

3) Search range consideration

Intra picture block copy is a causal process, that is, an intraBC block can only use the previously reconstructed pixels in the same picture as its predictor. When intraBC was first proposed, the search range was kept local. That is, only pixels from the CTU that the current block belongs to and its left neighboring CTU can be used as intraBC predictor. Later on, it was decided to extend the search area to the full picture [Reference Pang, Sole, Hsieh and Karczewicz14]. That is, all previously reconstructed pixels in the current picture can be used as predictor. In order for a BV to be valid, at least one of the following conditions shall be true:

(1)$${\rm BV}\_x + n{\rm PbSw} + x{\rm PbS} - x{\rm Cbs} \lt = 0, $$

(2)$${\rm BV}\_y + n{\rm PbSh} + y{\rm PbS} - y{\rm Cbs} \lt = 0, $$

where (BV_x, BV_y) is the BV for the current PU; nPbSw and nPbSh are the width and height of the current PU; (xPbS, yPbS) is the location of the top-left pixel of the current PU relative to the current picture; and (xCbs, yCbs) is the location of the top-left pixel of the current CU relative to the current picture.

Furthermore, when wavefront parallel processing (WPP) is enabled, some regions top-right to the current block cannot be used for prediction as the pixels in these regions may not yet be decoded and reconstructed while the current block is being processed. Therefore, one additional constraint is applied to regulate block vectors as proposed in [Reference Li and Xu15,Reference Lai, Xu, Liu, Chuang and Lei16], and described as follows:

(3)$$\eqalign{&(x{\rm Pb} + {\rm BV}\_x + n{\rm PbSw} - 1)/{\rm CtbSizeY} \cr & - x{\rm Cbs}/{\rm CtbSizeY} \lt = y{\rm Cbs}/{\rm CtbSizeY} \cr & - (y{\rm Pb} + {\rm BV}\_y + n{\rm PbSh} - 1)/{\rm CtbSizeY}},$$

where CtbSizeY is the size of the CTU. In fact, constraint (3) only causes very small coding performance decrease when WPP is not used, thus it is decided that constraint (3) is always enforced regardless of whether WPP is enabled. Figure 3 demonstrates when constraint (3) is invoked, in which the shaded CTUs top-left to the current CTU can be used for intraBC prediction of any block belongs to the current CTU, while the rest CTUs which are bottom-right to the current CTU are unavailable for the same prediction.

Fig. 3. Demonstration of the search range constraint in (3) for intra block copy.

It is also asserted that intra picture prediction should not cross-slice or tile boundaries. Therefore, a valid BV shall not point to areas in other slices or tiles than the current one.

B) Palette coding

Besides the repetition of textual patterns inside one picture, another unique characteristic of screen content is the statistically fewer number of colors used for representing an image block, in comparison with an image block of the same size in camera-captured content. For example, in a screen content image with text, typically a coding block contains only the foreground text color and the background color. Sometimes, the random patterns of text characters and letters make it challenging for the current coding block to find a matching block in the same or previously coded pictures. It may also be challenging to utilize the directional local intra prediction for efficient compression in this circumstance. A new coding tool, a.k.a. palette coding was proposed and proven to be effective for handling this type of source.

Briefly speaking, palette coding is a major color-based prediction method [Reference Guo, Pu, Zou, Sole, Karczewicz and Joshi17,Reference Onno, Xiu, Huang and Joshi18]. All pixels (their values of all components, Y/Cb/Cr or R/G/B) of a coding block are classified into a list of major colors. A major color is a representative color which has high frequency of occurrence in the block. For each palette coded CU, a color index table, i.e. palette is formed with each index entry associated with three (Y/Cb/Cr or R/G/B) sample values. All the pixels in the CU are converted into corresponding indices, except some rarely used pixels, which are isolated in the color index map and cannot be quantized to any of the major colors. These pixels are called escape pixels and an ESCAPE symbol is used to mark each of them. The actual pixel values of these escape pixels are signaled explicitly. The indices, including ESCAPE, are run-length coded using the index of either above or left neighbor as predictor. The following paragraphs describe how palette coding is performed with more detail.

1) Coding of palette entries

The coding of palette entries for a CU is based on a predictor buffer called palette predictor. For each entry in the palette predictor, a reuse flag is sent to signal whether or not this entry will be used in the current palette. If yes, this entry will be put in front of the current palette. For those entries in the current palette but not in the palette predictor, the number of them and their pixel values are signaled. These signaled new entries are put at the bottom of the current palette. The current palette size is then calculated as the number of reused palette entries plus the number of signaled palette entries.

After decoding a palette-coded CU, the palette predictor is updated for the next palette-coded CU. This is done using the information of the current palette. The entries of the current palette are put in front of the new palette predictor, followed by those unused entries from the previous palette predictor. This process is called “palette predictor stuffing”. The new predictor size is then calculated as the size of the current palette plus the number of unused palette entries. In Fig. 4, an example of the current palette derivation and the palette predictor update is shown. In this example, the palette predictor for the current CU has a size of 4, two of which are reused. The current palette has a size of 4, while the updated predictor (for the next CU) has a size of 7.

Fig. 4. Use of palette predictor to signal palette entries.

For a special case, when the current palette is exactly the same as the one previously coded (the last coded palette), a sharing flag is used to indicate this scenario. In this case, no new palette entry signaling or palette predictor update is necessary.

Note that the maximum sizes for the palette and palette predictor are signaled at the SPS (sequence parameter set) level. In the reference software, they are set to be 31 (plus one for escape index) and 64, separately. The update of palette predictor will stop if the maximum predictor size is reached. The last coded palette and palette predictor is set to zero size as an initialization at the beginning of each slice or, they are treated in a similar way as the CABAC status synchronization at the beginning of each WPP thread [Reference Li and Xu15,Reference Misra, Kim and Segall19]. That is, at the beginning of each CTU row, the last coded palette and palette predictor information for the first palette-coded CU will be set to the last used palette and palette predictor from the top-right CTU relative to the current CTU.

2) Coding of palette indices

The palette indices are coded sequentially using either horizontal or vertical traverse scan as shown in Fig. 5. The scan order is explicitly signaled. As the vertical scan can be regarded as horizontal scan over a transposed CU, it is assumed that the horizontal scan is used in this paper for simplicity.

Fig. 5. Horizontal and vertical traverse scans.

The palette indices are coded using two prediction modes: “COPY_INDEX” and “COPY_ABOVE”. The ESCAPE symbol is treated as the largest index. In the “COPY_INDEX” mode, the palette index is explicitly signaled. In the “COPY_ABOVE” mode, the palette index of the sample in the row above is copied. For both “COPY_INDEX” and “COPY_ABOVE” modes, a run value is signaled which specifies the number of subsequent samples that are also coded using the same mode. For an ESCAPE symbol, its pixel value (all components) needs to be signaled. The coding of palette indices is illustrated in Fig. 6.

Fig. 6. Example of coding of palette indices. (a) Palette indices and their modes. Dotted circle: signaled indices in “COPY_INDEX” mode; horizontal solid arrow: “COPY_INDEX” mode; vertical solid arrow: “COPY_ABOVE” mode. (b): Decoded string of syntax elements for the indices in (a).

C) Adaptive color transform (ACT)

An ACT operates at the CU level, where a 1-bit flag is used to signal whether a color space transform is used for each of the prediction residue pixels in this CU. The motivation of this tool is that for a given color space (e.g. RGB space), there are certain correlation among different components of the same pixel. Even after the prediction from a spatial or temporal neighboring pixel, the correlation among the residue pixel components still exists. A transform of color space may be helpful to concentrate the energy and therefore improve the coding performance. Note that in palette coding, ACT does not apply as there is no residue coding. A brief overview of ACT for lossy [Reference Zhang20] and lossless [Reference Henrique Malvar, Sullivan and Srinivasan21] coding will be given in this subsection.

1) ACT in lossy coding

For the prediction residue of each pixel, a color space transform is performed, prior to the compensation from intra or inter prediction. A decoder flow of ACT is shown in Fig. 7. In lossy coding, ACT uses the followin color space conversion equations:

(4)$$\eqalign{& {\rm Forward: }\left[ {\matrix{ {C_0^{\prime} } \cr {C_1^{\prime} } \cr {C_2^{\prime} } \cr } } \right] = \left[ {\matrix{ 1 & 2 & 1 \cr 2 & 0 & { - 2} \cr { - 1} & 2 & { - 1} \cr } } \right]\left[ {\matrix{ {{C_0}} \cr {{C_1}} \cr {{C_2}} \cr } } \right]\bigg/4 \cr & {\rm Inverse:}\,\left[ {\matrix{ {{C_0}} \cr {{C_1}} \cr {{C_2}} \cr } } \right] = \left[ {\matrix{ 1 & 1 & { - 1} \cr 1 & 0 & 1 \cr 1 & { - 1} & { - 1} \cr } } \right]\left[ {\matrix{ {C_0^{\prime} } \cr {C_1^{\prime} } \cr {C_2^{\prime} } \cr } } \right],} $$

where (C ₀, C ₁, C ₂) and (C′₀, C′₁, C′₂) are the three color components before and after color space conversion, respectively. The forward color transform from (C ₀, C ₁, C ₂) to (C′₀, C′₁, C′₂) is not normalized. In order to compensate for the non-normalized nature of the forward transform, for a given normal QP value for the CU, if ACT is turned on, the quantization parameter is set equal to (QP – 5, QP – 3, QP – 5) for (C′₀, C′₁, C′₂), respectively. The adjusted quantization parameter only affects the quantization and inverse quantization of the residuals in the CU. In the deblocking process, the normal QP value is still used.

Fig. 7. High Efficiency Video Coding extensions on screen content coding decoder flow with adaptive color transform.

2) ACT in lossless coding

For lossless coding (where no transform or quantization applies), color space conversion based on the YCoCg-R space (the reversible version of YCoCg) as depicted in Fig. 8 is used in ACT. The reversible color space transform increases the intermediate bit depth by 1-bit after forward transform. Note that the QP adjustment discussed in lossy coding does not apply.

Fig. 8. Flow chart of lifting operations of forward and inverse adaptive color transform in lossless coding.

D) Adaptive motion vector resolution

Screen content by definition is not camera-captured. The motion from one screen content image to another should have an integer displacement. Therefore, the aliasing effect due to the camera sampling of a temporal motion may not be valid. If fractional-pel motion compensation is not used at all, then the bits used to present fractional-pel motion vectors can be saved. However, this approach (use integer motion vector resolution always) is apparently not suitable for natural content. In addition, simulation results showed that significant losses will occur even for some screen content sequences, if integer motion resolution is enforced at all times. Therefore, it is proposed in [Reference Li, Xu, Sullivan, Zhou and Lin22] that a slice-level flag is used, to signal that whether the motion vectors in this slice is at integer-pel resolution or quarter-pel resolution. At the decoder side, the decoded motion vector needs to be left shifted by two if “integer motion vector” is used in the slice. As for the encoder design, different approaches were proposed to determine whether integer motion is adequate for the current slice. In the two-pass approach, the current slice is encoded twice using both integer motion and quarter-pel motion; the solution with better RD result is then selected. As a result, the encoder time is doubled. In an alternative approach, a pre-analysis of the slice content is performed using the original pixels. The percentage of 8 × 8 homogeneous blocks is calculated. The definition of homogeneous block is that it can find the perfect match in the first reference picture of list 0 for the slice, or it has no textual pattern (the whole block is single valued). If the estimated percentage of 8 × 8 homogeneous blocks is over a pre-defined threshold, the slice-level flag will be set on to use integer motion. By doing this, the two-pass coding process is avoid, while the majority of coding performance gain in two-pass approach can be preserved (3~ 4% BD rate saving) for text and graphics with motion 1080p class test sequences. Note that for camera-captured and animation video, no significant coding benefit can be observed by using this tool.

A) Simulation setup

In this section, the coding performance of current HEVC SCC is evaluated using its reference software SCM-3.0 [23], and compared with the coding performance of H.264/MPEG-4 AVC standard High 4:4:4 Predictive profile (reference software JM-18.6 [24]) and HEVC RExt Main 4:4:4 profile (reference software RExt-8.1) [25]. The simulations are conducted under HEVC SCC common test conditions listed in [Reference Yu, Cohen, Rapaka and Xu26], in which a set of non-camera-captured as well as camera-captured video sequences are tested in both RGB 4:4:4 and YUV 4:4:4 color formats. In the text and graphics with motion (TGM) class, seven sequences are selected to represent the most common screen content videos. In the mixed content (MC) class, three sequences are selected containing a mixture of both natural video and text/graphics. One animation (ANI) sequence and two camera-captured (CC) video sequences are also tested. Some screen shots of selected screen content video sequences are shown in Fig. 9. A summary of the test sequences is provided in Table 1. More details about these sequences can be found in [Reference Yu, Cohen, Rapaka and Xu26]. All the sequences are tested under testing configurations of all intra (AI), random access (RA), and low-delay B (LB). The results are measured in B-D rates [Reference Bjontegaard27] based on the calculation from four QP points, i.e. QP equals to 22, 27, 32, and 37. Negative values in the tables mean BD rate reductions or coding efficiency improvement. In addition, lossless coding results are also provided. Note that when RGB sequences are coded, the actual color component order is GBR based on the assumption that the human visual system is more sensitive to green color. Also, only the B-D rates of Y/G component of YUV/RGB format is presented in the following tables for illustration.

Fig. 9. Typical types of screen content video. (a) Flying Graphics (text and graphics with motion (TGM)). (b) Slide Show (TGM). (c) Mission Control Clip 3 (mixed content (MC)). (d) Robot (animation (ANI)).

Table 1. 4:4:4 test sequences.

B) Overall performance evaluation of HEVC SCC

The coding performance of HEVC SCC in comparison with the prior art standards, H.264/MPEG-4 AVC High 4:4:4 Predictive profile [Reference Li and Xu28] and HEVC RExt Main 4:4:4 profile are reported in this section. The relative B-D rate savings of HEVC SCC on top of H.264/MPEG-4 AVC High 4:4:4 Predictive profile and HEVC RExt Main 4:4:4 profile are demonstrated in Tables 2 and 3, respectively.

Table 2. SCM-3.0 performance comparison, lossy.

Table 3. SCM-3.0 performance comparison, lossless.

C) Individual tools evaluation

In addition to the overall performance evaluation, the individual tool evaluation is also performed [Reference Li and Xu28,Reference Lai, Liu and Lei29]. Among the four individual tools, the adaptive motion vector resolution is relatively simple and basically there is no technical change since its adoption. Its performance has been reported during the discussion hence no repeated test is performed in this section. For all other coding tools discussed earlier in this paper, including intraBC, palette coding, and ACT are turned off from SCM-3.0 one by one, to show the impacts on the performance. In Tables 4 and 5, the relative B-D rate increments (positive numbers) for turning off each tool are shown for lossy and lossless coding conditions, respectively. Here the anchor is SCM-3.0 with all tools turned on.

Table 4. SCM-3.0 with single tool off versus SCM-3.0, lossy.

Table 5. SCM-3.0 with single tool off versus SCM-3.0, lossless.

A) IntraBC

In motion compensation, the reference picture data are typically stored off-chip and put on-chip from time to time when necessary. In the hardware design, the off-chip memory is usually organized in group, e.g. in blocks of 8 × 2, etc. When the motion vector points to a particular position, all the memory blocks that contain at least one pixel of the reference block need to be accessed and loaded on chip. When interpolation is taken into consideration, an extra few lines surrounding the reference block also need to be accessed. Therefore, when a compensation block size is small, the overhead of reading unused pixels from memory blocks in the worst case is high. The memory bandwidth consumption is measured in (5) [Reference François, Tabatabai and Alshina30]:

(5)$$ P = {\displaystyle{{\lceil{(m - 1 + M + L - 1)/m} \rceil} \cdot {\lceil{(n - 1 + N + L - 1)/n} \rceil} \cdot m \cdot n} \over {M \cdot N}}, $$

where M and N respectively represent the width and height of the smallest unit for intra copying, m and n denote respectively the width and height of the memory access pattern (e.g. 4 × 2, 8 × 4, etc.), and L is related to the tap length of interpolation filter (e.g. L = 8 for an 8-tap filter and L = 1 for no interpolation). Compared with the existing HEVC version 1, there is no interpolation needed for an intraBC coded PU. In light of this fact, even though the 4 × 4 intraBC block is smaller than the smallest partitions in HEVC ($8 \times 4/4 \times 8$ partitions), the worst case in memory bandwidth consumption for the whole codec design is still maintained.

Another aspect of the memory bandwidth issue for intraBC is that there is some additional memory bandwidth requirement for extra reading/writing operation for unfiltered reference picture. Because the full-frame search is enabled in intraBC, previously reconstructed, but unfiltered samples need to be written off-chip for storage and read on-chip for compensation. Compared with a regular reference picture for inter motion compensation, this part is the extra effort.

Other aspects of the intraBC mode complexity are similar to the inter mode in HEVC.

B) Palette coding

As mentioned in the introduction of palette coding, no residue coding (transform or quantization) is performed for palette coding. In this aspect, the complexity of palette decoding is reduced, as compared with intra mode or inter mode in HEVC. However, from the parallel processing point of view, the operation of palette coding is per pixel basis, that is, in the worst case each pixel in a CU needs to be processed separately (assigning index, parsing run length, and converting index to pixel value). This is different from the line-based processing in intra mode and block-based processing in inter mode.

C) ACT

The extra operation this tool introduces is a one- dimensional transform per pixel (among three components). As for the encoder design, it adds one more dimension of mode decision loop to decide whether this tool should be used or not. In a brute-force decision approach, the computations will be doubled in mode decision. According to the results shown in Section IV and some previous study [Reference Lai, Liu and Lei31], it would be a practical solution to link this CU-level decision to the color format of the video. In particular, one can always enable this tool for all CUs in RGB format video and always disable it for YUV format.

D) Adaptive motion vector resolution

The extra complexity of this tool over the existing HEVC inter mode is negligible. For the encoder design, in order to avoid 2-pass coding, the pre-analysis of the slice is necessary, which requires some data access of the whole slice before the block-by-block processing of it.

A) Unification of intraBC mode and inter mode

It was proposed to consider the (partially) reconstructed current picture as an additional reference picture as opposed to conventional temporal reference pictures, such that the intraBC mode signaling can be unified with inter mode [Reference Xu, Liu and Lei32–Reference Li, Xu, Xu, Liu and Lei34]. Consequently, the indication of the use of intraBC mode is via a reference index in the reference picture list instead of an intraBC mode flag in each CU. Then when a PU's reference picture index points to this particular reference picture, this PU is intraBC coded. This approach has been studied for a few meeting cycles and was included in the draft standard in February 2015 [Reference Pang35]. With this adoption, the current picture is regarded as a reference picture (for intraBC mode) and is put in the last position of reference list 0 by default. In addition, merge, skip, and AMVP modes may be enabled when the reference picture is current picture. When unified with inter mode, no very significant coding performance change is observed compared with intraBC as a CU mode [Reference Pang35]. One main benefit identified for unifying intraBC with inter mode is that a large part of existing HEVC version 1 inter mode design can be shared and used for intraBC. In some implementations intraBC can be enabled with a few high-level changes and bitstream conformance constraints (e.g. (1) – (3) in Section II.A.3).

B) Extended to non-4:4:4 formats

HEVC SCC was initially designed for 4:4:4 color format only. It is under development to further extend the scope to cover potential non-4:4:4 applications (including monochrome format) as well. For this purpose, the palette coding has provided necessary supports for coding of non-4:4:4 video formats [Reference Ye, Liu and Lei36,Reference Joshi, Pu, Seregin, Karczewicz and Zou37] [Reference Xiu, Ye and He38], while the intraBC mode has already covered non-4:4:4 formats in its current design. The ACT will not be used in non-4:4:4 coding scenario.

C) Other tools experimented

One extension of intraBC is intra line copy (ILC) [Reference Chang, Chen, Liao, Kuo and Peng39], in which an intraBC PU can be further split into multiple lines and the prediction is performed on each line. The length of each line may be either 2N or N, depending on the direction of the line and the size of the PU. For example, a $2\hbox{N} \times \hbox{N}$ PU may be split into either N $\lpar 2\hbox{N} \times 1\rpar $ horizontal lines or 2N $\lpar \hbox{N} \times 1\rpar $ vertical lines. Each line has a line vector, pointing to the location of its predictor line within the current picture.

Similar to the run-length coding in palette coding, intra string copy (ISC) [Reference Zhao, Chen and Lin40] is proposed to HEVC SCC, in which a variable length of connected pixels (string) in scan order is predicted from reconstructed pixels with same pattern. The offset from the first pixel of this string to the first pixel of the predictor string is signaled, followed by the length of the string.

On top of SCM-3.0, additional coding performance gains are observed from using ILC or ISC. One issue currently under investigation is the memory bandwidth of these methods in the worst-case scenario. The worst-case memory bandwidth consumptions of ILC and ISC are presented in Table 6, compared with the worst-case memory bandwidth consumptions of HEVC version 1 and intraBC [Reference Chen, Chen, Xu, Lin and Wang41] calculated by (5). It is desired to reduce the worst-case memory bandwidth consumption of these methods such that they can be part of a practical design. As the standardization approaches its finalization stage, these experimented new tools are less likely to be included in the standard.

Table 6. Memory bandwidth assessment for different methods.