A) Image encryption for EtC systems

Block scrambling-based image encryption schemes [Reference Watanabe, Uchida, Fukuhara and Kiya15–Reference Sirichotedumrong, Chuman, Imaizumi and Kiya20] were proposed for EtC systems, in which a user wants to securely transmit an image I to an audience or a client via, SNS or CPSS providers, as shown in Fig. 1. The privacy of the image to be shared can be controlled by the user unless the user does not give the secret key K to the providers, although the image is generally recompressed by the providers. In contrast, in CtE systems, the disclosure of non-encrypted images is required before the recompression.

Fig. 1. EtC system.

B) Color-based image encryption

In the previous works [Reference Watanabe, Uchida, Fukuhara and Kiya15–Reference Kurihara, Imaizumi, Shiota and Kiya19], a full-color image (I _RGB) with X × Y pixels is divided into non-overlapping blocks each with B _x × B _y; then four block scrambling-based encryption steps are applied to the divided blocks as follows (see Fig. 2).

1. 1) Randomly permute the divided blocks by using a random integer generated by a secret key K ₁.

2. 2) Rotate and invert each block randomly (see Fig. 3) by using a random integer generated by a key K ₂.

3. 3) Apply negative–positive transformation to each block by using a random binary integer generated by a key K ₃, where K ₃ is commonly used for all color components. In this step, a transformed pixel value in the i-th block B _i, p′, is calculated using

   (1)$$p'= \left\{\matrix{p \hfill & (r(i)=0) \hfill \cr p \oplus (2^L-1) \hfill & (r(i)=1) \hfill}, \right.$$
   where r(i) is a random binary integer generated by K ₃, p ∈ B _i is the pixel value of the original image with L bit per pixel, and ⊕ is the bitwise exclusive-or operation. The value of occurence probability P(r(i)) = 0.5 is used to invert bits randomly.

4. 4) Shuffle three color components in each block by using an integer randomly selected from six integers generated by a key K ₄ as shown in Table 1.

Fig. 2. Color-based block scrambling image encryption.

Fig. 3. Block rotation and inversion. (a) Block rotation, (b) block inversion

Table 1. Permutation of color components for a random integer. For example, if the random integer is equal to 2, the red component is replaced by the green one, and the green component is replaced by the red one while the blue component is not replaced.

An example of an encrypted image with B _x = B _y = 16 is shown in Fig. 4(b) where Fig. 4(a) is the original one.

Fig. 4. Examples of images encrypted by using the color-based, conventional grayscale-based, and proposed schemes. n is the number of divided blocks. (a) Original image (X × Y = 512 × 512), (b) color-based scheme (B _x = B _y = 16, n = 1024), (c) conventional grayscale-based scheme (B _x = B _y = 8, n = 12288), (d) proposed scheme with 4:2:0 sub-sampling (B _x = B _y = 8, n = 6144), (e) proposed scheme without sub-sampling (B _x = B _y = 8, n = 12288).

Although images encrypted by using the color-based image encryption scheme are compatible with the JPEG standard and have almost the same compression performance as non-encrypted ones when using B _x = B _y = 16, there is a limitation that the scheme cannot achieve. The possible smallest block size of the scheme is 16 × 16 to avoid the effect of color sub-sampling. As shown in Fig. 5, if 4:2:0 color sub-sampling is applied to an encrypted image, each 8 × 8-block in the sub-sampled chroma components consists of four 4 × 4-blocks from different 8 × 8-block, which have low correlation among the blocks. As a result, block distortion is generated due to the interpolation of the sub-sampled chroma components with discontinuous pixel values. An example of the decrypted images including block distortion, where the JPEG quality factor ($Q_{f_u}$) of uploaded images was equal to 100, is shown in Fig. 6. If the block size is smaller than 16 × 16 pixels, such as 8 × 8 pixels, the compression performance decreases and some block distortion is generated. This is because when color sub-sampling is applied to the chroma components (C _b and C _r) of a color image in a JPEG encoder, the interpolation is carried out to the sub-sampled chroma components (C′_b and C′_r) to reconstruct the spatial resolution as that of the original image in a decoder.

Fig. 5. Sub-sampling in JPEG encoder and interpolation of chroma components in JPEG decoder.

Fig. 6. Example of the decrypted images with the color-based scheme including block distortion (B _x = B _y = 8, $Q_{f_u}=100$, and 4:2:0 sub-sampling). (a) Downloaded from Twitter (PSNR=29.44dB), (b) downloaded from Facebook (PSNR = 26.93 dB).

C) Grayscale-based image encryption

A conventional grayscale-based image encryption has been proposed to avoid the effect of color sub-sampling by encrypting I _RGB into the encrypted image (I _e) which consists of only one color channel [Reference Sirichotedumrong, Chuman, Imaizumi and Kiya20]. In order to generate I _e, four processing steps are carried out as follows (see Fig. 7).

1. 1) Split I _RGB into three RGB channels, i _R, i _G, and i _B and concatenate all channels to generate a conventional grayscale-based image ($I_g^{RGB}$) with 3(X × Y) pixels as shown in Fig. 8.

2. 2) Divide $I_g^{RGB}$ into blocks each with B _x × B _y and then randomly permute the divided blocks by using a random integer generated by a secret key K ₁.

3. 3) Rotate and invert each block randomly by using a random integer generated by a key K ₂.

4. 4) Apply negative–positive transformation to each block by using a random binary integer generated by a key K ₃. This step is carried out by using the same process as Step 3 of the color-based image encryption in Section II.B.

Fig. 7. Conventional grayscale-based encryption.

Fig. 8. Conventional grayscale-based image generation.

Since images encrypted by using the grayscale-based image encryption contain only one color channel, the encrypted images can avoid the effect of color sub-sampling. Therefore, the scheme allows us to use 8 × 8 as the block size, which is smaller than that with the color-based one, as shown in Fig. 4(c). Moreover, the number of blocks is larger because the block size is smaller, and the number of pixels of encrypted images is larger. As a result, the scheme enhances the security against various attacks [Reference Sirichotedumrong, Chuman, Imaizumi and Kiya20] and also provides the robustness against color sub-sampling due to the use of grayscale-based images. However, the compression performance of the conventional grayscale-based scheme decreases, because the correlation between RGB color channels is not used.

D) Security against ciphertext-only attacks

Security mostly refers to protection from adversarial forces. The proposed scheme aims to protect the visual information which allow us to identify an individual, a time, and the location of the taken photograph. Untrusted providers and unauthorized users are assumed as the adversary. This paper considers both brute-force and jigsaw puzzle solver attacks as ciphertext-only attacks.

1) Brute-force attack

In the conventional color-based encryption scheme, if an image with X × Y pixels is divided into blocks with B _x × B _y pixels, the number of blocks n is given by

(2)$$n = \left\lfloor \displaystyle{X \over B_x} \right\rfloor \times \left\lfloor \displaystyle{Y \over B_y} \right\rfloor,$$

where $\lfloor \cdot \rfloor $ is a function that rounds down to the nearest integer. The four block scrambling-based processing steps are then applied to the divided image.

The key space of the block scrambling (Step 1) N _scr(n), which is the number of permutation of n blocks, is given by

(3)$$N_{scr}(n) = { }_{n} P { }_{n} = n!.$$

Similarly, the key spaces of other encryption steps are given as

(4)$$N_{rot}(n) = 4^{n},\;N_{inv}(n) = 4^{n},\;N_{rot\&inv}(n) = 8^{n}\;$$

(5)$$N_{np}(n) = 2^{n}, \; N_{col}(n) = \bigl( { }_3 P { }_3 \bigr)^{n} = 6^{n}$$

where N _rot(n) and N _inv(n) are the key spaces of the block rotation and block inversion, and $N_{rot\&inv}(n)$ is the key space of the encryption combining them (Step 2). Note that $N_{rot\&inv}$ is the key space considering the collision between block rotation and inversion. Namely, rotating pieces 180 degrees is the same operation as inverting them horizontally and vertically. N _np(n) and N _col(n) are the key spaces of the negative–positive transformation (Step3) and color component shuffling (Step 4), respectively. Consequently, the key space of images encrypted by using all the proposed encryption steps, N _A(n), is represented by

(6)$$\eqalign{N_A(n) & = N_{scr}(n) \cdot N_{rot\&inv}(n)\cdot N_{np}(n) \cdot N_{col}(n)\cr &= {n}! \cdot 8^{n} \cdot 2^{n} \cdot 6^{n}.}$$

In comparison, since an image encrypted by using the conventional grayscale-based image encryption scheme is generated from $I_g^{RGB}$ with 3(X × Y) pixels, the number of blocks of the grayscale-based scheme (n _g) is three times larger than that of the color-based scheme. n _g is given by

(7)$$n_g = 3n.$$

Unlike the color-based scheme, color shuffling is not carried out by the conventional grayscale-based encryption scheme. Thus, the key space of the grayscale-based image encryption is calculated by

(8)$$\eqalign{N_B(n) &= N_{scr}(3n) \cdot N_{rot\&inv} (3n) \cdot N_{np}(3n) \cr &= {3n}! \cdot 8^{3n} \cdot 2^{3n} \gg N_{A}(n),}$$

where n is the number of blocks calculated from I with X × Y pixels in accordance withequation (2). Although the color shuffling is not applied to the scheme, the number of blocks is larger, as shown in equation (8). Therefore, the grayscale-based image encryption enhances the robustness against brute-force attacks.

A) Image encryption procedure

To generate an encrypted image (I _e), the following steps are carried out (see Fig. 9).

1. 1) Transform an RGB color image (I _RGB) with X × Y pixels into an image in YCbCr color space (I _YCbCr) as in [31]. Note that I _YCbCr consists of three individual channels which can be represented by i _Y, i _Cb, and i _Cr.

2. 2) Generate a proposed grayscale-based image (I _g) from I _YCbCr. This paper considers two types as I _YCbCr:

   1. (a) I _g without color sub-sampling: As shown in Fig. 10(a), i _Y, i _Cb, and i _Cr are combined into I _YCbCr with 3(X × Y) pixels.

   2. (b) I _g with 4:2:0 color sub-sampling: As shown in Fig. 10(b), i _Cb and i _Cr are sub-sampled to produce the sub-sampled chrominance (i′_Cb and i′_Cr) using the same method as in [26]. Then, i _Y, i′_Cb, and i′_Cr are combined into I _g with 3/2(X × Y) pixels.

3. 3) Divide I _g into blocks each with B _x × B _y and then randomly permute the divided blocks by using a random integer generated by a secret key K ₁.

4. 4) Rotate and invert each block randomly by using a random integer generated by a key K ₂.

5. 5) Apply negative–positive transformation to each block by using a random binary integer generated by a key K ₃.

Fig. 9. Proposed grayscale-based image encryption.

Fig. 10. Proposed grayscale-based image generation. (a) I _g without sub-sampling (4:4:4), (b) I _g with 4:2:0 sub-sampling.

An example of images encrypted by using the proposed scheme is shown in Figs 4(d) and 4(e).

B) Image decryption procedure

As shown in Fig. 1, to reproduce a decrypted image ($\hat {I}$) from an encrypted JPEG image ($\hat {I}_{ec}$), JPEG decompression is performed to $\hat {I}_{ec}$. As a result, the decompressed image ($\hat {I_e}$) is generated. Then, the following steps are carried out to decrypt $\hat {I_e}$ using the corresponding secret key K (see Fig. 11).

1. 1) Divide $\hat {I_e}$ into blocks, each with B _x × B _y.

2. 2) Apply inverse negative–positive transformation to each block with key K ₃.

3. 3) Inversely rotate and invert each block with key K ₂

4. 4) Assemble blocks based on key K ₁ to produce the grayscale-based image ($\hat {I}_g$)

5. 5) Reconstruct the color image in YCbCr color space ($\hat {I}_{YCbCr}$) from $\hat {I}_g$

6. 6) Transform $\hat {I}_{YCbCr}$ to RGB color space

7. 7) Integrate RGB components to generate $\hat {I}$

Fig. 11. Proposed grayscale-based image decryption procedure.

There are two methods used for reconstructing $\hat {I}_{YCbCr}$ from $\hat {I}_g$ in Step (5):

• • I _g without sub-sampling: When I _g was generated according to Fig. 10(a), $\hat {I}_g$ is separated into three color channels. Then, the three channels are integrated into $\hat {I}_{YCbCr}$, as shown in Fig. 12(a).

• • I _g with 4:2:0 sub-sampling: When I _g was generated according to Fig. 10(b), $\hat {I}_g$ is separated into one luminance ($\hat {i}_Y$) and two sub-sampled chrominance components ($\hat {i}'_{Cb}$ and $\hat {i}'_{Cr}$), as shown in Fig. 12(b). Then, $\hat {i}'_{Cb}$ and $\hat {i}'_{Cr}$ are interpolated to increase the spatial resolution as in [32]. Eventually, the luminance and interpolated chrominance components are integrated into $\hat {I}_{YCbCr}$.

Fig. 12. Reconstruction of decrypted images. (a) I _g without sub-sampling (4:4:4), (b) I _g with 4:2:0 sub-sampling.

C) Compression of grayscale-based image

This paper focuses on JPEG lossy compression, although the JPEG standard supports both lossy and lossless compressions, and the block scrambling-based image encryption schemes are applicable to lossless compression as discussed in [Reference Kurihara, Imaizumi, Shiota and Kiya19]. This is because most JPEG compression applications, especially SNS and CPSS providers, utilize lossy compression.

JPEG softwares, such as Independent JPEG Group (IJG) software [32], generally utilize two default quantization tables to quantize i _Y, i _Cb, and i _Cr of I _YCbCr called the luminance quantization table (Y-table), and the chrominance quantization table (CbCr-table), although image-dependent quantization tables can be designed by users [Reference Yang and Wang33]. However, grayscale-based images do not correspond to luminance or chrominance. Therefore, we propose a new quantization table called G-table to improve the compression performance of I _g.

In JPEG compression, all pixel values in each block of I _g are mapped from [0, 255] to [ − 127, 128] by subtracting 128, then each block is transformed using Discrete Cosine Transform (DCT) to obtain DCT coefficients.

The DCT coefficients are employed to generate G-table. Let DCT _m(i, j) be the DCT coefficient of the m ^th block at the position (i, j) where 1 ≤ i ≤ 8 and 1 ≤ j ≤ 8. Considering every block of I _g, the absolute value of DCT _m(i, j) is calculated, and the arithmetic mean of |DCT _m(i, j)| is expressed by

(9)$$c(i,j)=\displaystyle{1 \over n_g}\sum_{m=1}^{n_g}\vert DCT_m(i,j) \vert$$

where I _g consists of n _g blocks.

As a set of grayscale-based images which consists of N _I images is utilized to determine G-table, we define c _k(i, j) as c(i, j) of the k ^th image and calculate the average of every c(i, j) from N _I grayscale-based images. The average $\bar{c}(i,j)$ is calculated as follow.

(10)$$\bar{c}(i,j)=\displaystyle{1 \over N_I}\sum_{k=1}^{N_I}c_k(i,j)$$

To obtain G-table, q(i, j) represents the quantization step size at (i, j) and is derived from the ratio between $\bar{c}(1,1)$ and $\bar{c}(i,j)$. The step size can be calculated by

(11)$$q(i,j) = \left\lceil \displaystyle{\bar{c}(1,1) \over \bar{c}(i,j)} \right\rceil + \epsilon $$

where ε is set to 16 for adjusting the Y-table step size at (1, 1) as for IJG software [32].

To design G-table for grayscale-based images, we employed 1338 images with 512 × 384 pixels from Uncompressed Color Image Database (UCID) [Reference Schaefer and Stich34]. Grayscale-based images were generated from all images in the dataset. Then, the grayscale-based images were compressed by using IJG software [32] to obtain the DCT coefficients. To design G-table for I _g without sub-sampling, the DCT coefficients were calculated whereas n _g = 9216, N _I = 1338, and ε = 16. On the other hand, G-table for I _g with 4:2:0 sub-sampling was designed by using n _g = 4608. As a result, two types of G-table designed for I _g without sub-sampling and with 4:2:0 sub-sampling were designed as shown in Fig. 13.

Fig. 13. G-table. (a) I _g without sub-sampling, (b) I _g with 4:2:0 sub-sampling.

D) Security against ciphertext-only attacks

The proposed scheme is grayscale-based image encryption, so it provides the higher security level than the color-based image encryption scheme in both brute-force and jigsaw puzzle solver attacks as well as the conventional grayscale-based encryption.

In I _g without color sub-sampling, the number of pixels of I _g is the same as the conventional one. This means that the number of blocks is also identical. As a result, the key space of this type is equal to that with the conventional one inequation (8).

In contrast, I _g with 4:2:0 sub-sampling has 3/2(X × Y) pixels, so n _g = (3/2) n. As a result, the key space of this type ($N_{P_{sub}}(n)$) is given by

(12)$$N_{P_{sub}}(n)={\displaystyle{3 \over 2}n}!\cdot 8^{({3}/{2})n} \cdot 2^{({3}/{2})n}\gg N_{A}(n).$$

As described in Section IV, images encrypted by using the proposed scheme can provide almost the same robustness against jigsaw puzzle solver attacks as the conventional grayscale-based encryption.

E) Robustness against image recompression

In Table 2, we summarize the relationship between uploaded and downloaded JPEG images of typical SNS and CPSS providers in terms of sub-sampling ratios, the quality factor (Q _f), and the maximum resolutions [Reference Chuman, Iida and Kiya35]. $Q_{f_u}$ and $Q_{f_d}$ denote the uploaded and downloaded quality factors, respectively.

Table 2. Relationship between uploaded JPEG files and downloaded ones in terms of sub-sampling ratios and the maximum resolutions. Providers do not resize uploaded images when their resolutions are less than or equal to the maximum resolutions, e.g. the maximum resolutions of Twitter and Tumblr are 4096 × 4096 and 1280 × 1280, respectively [29].

Providers do not resize uploaded images when the resolution of uploaded images is less than or equal to the maximum resolution that each provider decided. For example, if the resolution of an image uploaded to Twitter is not larger than 4096 × 4096 pixels, the uploaded image is not resized. In this paper, we assume that the image resolution is less than or equal to the maximum resolution of each provider as well as in [Reference Kurihara, Kikuchi, Imaizumi, Shiota and Kiya17,Reference Kurihara, Imaizumi, Shiota and Kiya19,Reference Chuman, Iida and Kiya35].

The quality of images downloaded from SNS and CPSS providers is generally degraded due to recompression forced by the providers. As shown in Table 2, a color JPEG image uploaded to Facebook is always recompressed into the new JPEG image with 4:2:0 color sub-sampling. Therefore, images encrypted by the color-based scheme are affected by the recompression, even when they were compressed with 4:4:4 sub-sampling. In comparison, when grayscale-based images are uploaded to Facebook, the color sub-sampling is not carried out, although they are recompressed with different quality factors from those of uploaded images. Images encrypted by using the proposed scheme are not affected by the effect of color sub-sampling due to the use of grayscale-based images as described in [Reference Sirichotedumrong, Chuman, Imaizumi and Kiya20].

A) Compression performance

To evaluate the compression performance of the proposed scheme, we utilized 30 images from CSIQ dataset (512 × 512) [Reference Larson and Chandler36]. All images in the dataset were encrypted by using the proposed scheme with B _x = B _y = 8. Then, all encrypted images were compressed with specific quality factors, Q _f ∈ [70, 100], using the JPEG standard from IJG software [32]. Rate-distortion (RD) curves, which are the average peak signal-to-noise ratio (PSNR) values of all images per bits per pixel (bpp), were used for evaluating the compression performance. The parameters used in the experiment are shown in Table 3. Two types of G-table in Fig. 13 were used to quantize the DCT coefficients of grayscale-based images, while the IJG standard tables were employed for images encrypted by the color-based encryption scheme and non-encrypted ones.

Table 3. Parameters used for uploaded JPEG images

Figure 14 shows RD curves of JPEG compressed images without any encryption and with encryption. As shown in Fig. 14(a), when images were compressed without sub-sampling, the images encrypted by using the proposed method had higher PSNR values than those with the conventional grayscale-based encryption scheme, and moreover, had almost the same RD curves as the non-encrypted ones and the color-based scheme.

Fig. 14. R-D curves of uploaded JPEG images. (a) Without sub-sampling (4:4:4), (b) with 4:2:0 sub-sampling. Note that the conventional grayscale-based scheme cannot consider 4:2:0 color sub-sampling. (c) Comparison between the proposed scheme with 4:2:0 sub-sampling and the conventional one.

Regarding 4:2:0 color sub-sampling, the conventional grayscale-based scheme cannot consider 4:2:0 color sub-sampling because the conventional grayscale-based images consist of RGB color components. As shown in Fig. 14(b), when 4:2:0 sub-sampling was carried out, the proposed scheme provided slightly higher PSNR values, compared with non-encrypted images and the color-based scheme.

In order to clearly compare the difference between the conventional scheme and the proposed one, RD curves in a low bpp range were plotted in Fig. 14(c). The result showed that the proposed scheme can provide higher PSNR values than the conventional one. In addition, the proposed scheme with 4:2:0 sub-sampling is useful for the low bpp range. Therefore, the proposed scheme enables us to maintain almost the same image quality as non-encrypted images.

Moreover, other conventional encryption methods are considered in terms of compression performance. There are various encryption methods which can maintain an image format after encrypting as well as the proposed scheme. However, they are not suitable to EtC systems with JPEG compression, because they do not consider using JPEG compression. We numerically compared the encryption methods[Reference Gaata and Hantoosh25,Reference Wu, Noonan, Yang and Jin26] with the proposed scheme. Figures 15(d) and 15(e) indicate decrypted images after compressing and decompressing encrypted ones, where Fig. 15(a) is the original one. The image quality of decrypted images heavily decreased due to JPEG compression as shown in Figs 15(d) and 15(e) , because they do not consider using JPEG compression as well as most other conventional encryption methods. The proposed scheme can maintain the high quality of images as shown in Figs 15(b) and 15(c). Note that Fig. 15(e) is a grayscale image [Reference Wu, Noonan, Yang and Jin26], so the PSNR value is not listed.

Fig. 15. Decrypted images after compressing and decompressing encrypted ones (Q _f = 90). (a) Original image (X × Y = 512 × 512), (b) proposed scheme without color sub-sampling (PSNR = 35.72 dB), (c) proposed scheme with 4:2:0 color sub-sampling (PSNR = 33.93 dB), (d) encryption scheme in [Reference Gaata and Hantoosh25] (PSNR = 10.32 dB, sub-sampling ratio = 4:2:0), (e) encryption scheme in [Reference Wu, Noonan, Yang and Jin26].

B) Robustness against image recompression

We uploaded images encrypted by using the proposed scheme to Twitter and Facebook, and then downloaded them, as well as images encrypted using color-based scheme [Reference Watanabe, Uchida, Fukuhara and Kiya15–Reference Kurihara, Imaizumi, Shiota and Kiya19] with B _x = B _y = 16, and non-encrypted images to confirm the effectiveness of the proposed scheme.

2) Experimental results

Figure 16 shows the quality of images downloaded from Twitter, where the values in horizontal axis are bpp values of uploaded images. Although the color-based scheme could maintain almost the same PSNR values as non-encrypted images, the proposed scheme offered slightly higher PSNR values in both sub-sampling ratios, as shown in Figs 16(a) and 16(b). This is because color JPEG images are affected by 4:2:0 color sub-sampling carried out by Twitter while grayscale-based images can avoid the effect of color sub-sampling.

Fig. 16. R-D curves of downloaded JPEG images from Twitter. (a) Uploaded without sub-sampling (4:4:4), (b) uploaded with 4:2:0 sub-sampling.

Figure 17 shows the quality of images downloaded from Facebook, where the horizontal axis refers to bpp values of uploaded images. As shown in Fig. 17(a), when images were compressed with 4:4:4 sub-sampling, the quality of images encrypted by the color-based scheme were heavily degraded compared with the non-encrypted ones. In comparison, the images encrypted by the proposed scheme provided higher PSNR values than those encrypted by the color-based scheme and non-encrypted images. This is because the proposed scheme is not affected by 4:2:0 sub-sampling carried out by Facebook, although color JPEG images are affected by 4:2:0 sub-sampling. When images were compressed with 4:2:0 sub-sampling, the proposed scheme provided higher image quality than the color-based image encryption, as shown in Fig. 17(b). However, when bpp>1, PSNR values of images encrypted by the proposed scheme are slightly lower than non-encrypted ones, but the decrypted images did not include any block artifacts. This is because every grayscale JPEG image uploaded to Facebook is recompressed to the new grayscale JPEG image with $Q_{f_d}=71$ while Facebook recompresses JPEG color JPEG images with $Q_{f_d}\in [71,85]$, as shown in Table 2. Note that the PSNR values of images encrypted by the color-based scheme were much lower than those of the proposed scheme and non-encrypted ones. The reason is that the images encrypted by the color-based scheme included block artifacts due to the influence of color sub-sampling.

Fig. 17. R-D curves of downloaded JPEG images from Facebook. (a) Uploaded without sub-sampling (4:4:4), (b) uploaded with 4:2:0 sub-sampling.

Consequently, the proposed scheme is shown to have robustness against image recompression and color sub-sampling caused by providers as the conventional grayscale-based scheme [Reference Sirichotedumrong, Chuman, Imaizumi and Kiya20,Reference Chuman, Sirichotedumrong and Kiya21].

C) Robustness against jigsaw puzzle solver attacks

We evaluated the robustness against jigsaw puzzle solver attacks in terms of the difficulty of assembling encrypted images.

1) Experimental conditions

The following three measures [Reference Gallagher37,Reference Cho, Avidan and Freeman38] were used to evaluate the results.

Direct comparison (D _c): represents the ratio of the number of pieces which are in the correct position. D _c for image I _d, namely, D _c(I _d) is calculated as

(13)$$D_{c}(I_d) =\displaystyle{1 \over n} \displaystyle\sum_{i=1}^{n}d_{c}(i),$$

$$d_{c}(i) = \left\{\matrix{1, \hfill & {\rm if \;} I_{d}(i) {\rm \; is \; in \; the \; correct \; position} \hfill \cr 0, \hfill& {\rm otherwise} \hfill} \right.$$

where I _d(i) represents the position of a piece i in image I _d.

Neighbor comparison (N _c): is the ratio of the number of correctly joined blocks. N _c for image I _d, namely, N _c(I _d) is calculated as

(14)$$N_c(I_d) =\displaystyle{1 \over N_{bou}}\sum_{k=1}^{N_{bou}}n_c(k),$$

$$n_{c}(k) = \left\{ \matrix{1, \hfill & {\rm if \;} b_{k}{\rm \; is \; joined \; correctly} \hfill \cr 0, \hfill & {\rm \; otherwise} \hfill} \right.$$

where N _bou is the number of boundaries among pieces in I _d, and b _k is the kth boundary in I _d. For an image with u × v blocks, there are N _bou = 2uv − u − v boundaries in the image.

Largest component (L _c): is the ratio of the number of the largest joined blocks that have correct adjacencies to the number of blocks in an image. L _c for image I _d, namely, L _c(I _d) is calculated as

(15)$$L_{c}(I_d) = \displaystyle{1 \over n} \max_{j} \{ l_c(I_{d},j)\}, j=1,2,\ldots,m$$

where l _c(I _d, j) is the number of blocks in the jth partial correctly assembled area, and m is the number of partial correctly assembled areas.

In the measures, D _c, N _c, L _c ∈ [0, 1], a larger value means a higher compatibility.

We utilized 20 images from resized Ultra-Eye dataset (256 × 144) [Reference Nemoto, Hanhart, Korshunov and Ebrahimi39]. Thirty different encrypted images were produced from each image by using different 30 keys. The encrypted images were assembled by using the jigsaw puzzle solver, and the image that had the highest sum of D _c, N _c, and L _c was chosen. This procedure was carried out independently, and the average D _c, N _c, and L _c for the 20 images were calculated.

2) Experimental results

Table 4 shows the robustness against the extended jigsaw puzzle solver attack [Reference Chuman, Kurihara and Kiya27–Reference Chuman, Kurihara and Kiya29]. The scores for assembling both types of the proposed scheme were much lower than those of the color-based scheme and equal to those with conventional one. This is because images encrypted with the proposed scheme have a large number of encrypted blocks and no color information in the blocks.

Table 4. Security evaluation of the color-based, conventional, and proposed scheme against the extended jigsaw puzzle solver [Reference Chuman, Kurihara and Kiya27–Reference Chuman, Kurihara and Kiya29].

Figures 18(g) and 18(k) are images assembled from Figs 18(b) and 18(f), respectively, while Fig. 18(a) is the original one. Comparing Figs 18(g) and 18(h), it obviously shows that the difficulty of assembling encrypted images depends on the block size. Since most conventional jigsaw puzzle solvers employ color information for assembling puzzles, reducing color channels of each pixel makes assembling puzzles harder. As shown in Figs 18(i) and 18(j), it is more difficult to assemble the image encrypted by the proposed grayscale-based image encryption because the encrypted image has only one color channel.

Fig. 18. Assembled images by using the extended jigsaw puzzle solver [Reference Chuman, Kurihara and Kiya27–Reference Chuman, Kurihara and Kiya29]. a) Original image (X × Y = 256 × 144), (b) encrypted image (color-based scheme, B _x = B _y = 16, n = 144), (c) encrypted image (color-based scheme, B _x = B _y = 8, n = 576), (d) encrypted image (proposed scheme without sub-sampling, B _x = B _y = 8, n = 1728), (e) encrypted image (proposed scheme with 4:2:0 sub-sampling, B _x = B _y = 8, n = 864), (f) encrypted image (conventional scheme, B _x = B _y = 8, n = 1728), (g) assembled image (color-based scheme, B _x = B _y = 16, L _c = 0.208), (h) assembled image (color-based scheme, B _x = B _y = 8, L _c = 0.012), (i) assembled image (proposed scheme without sub-sampling, B _x = B _y = 8, L _c = 0.002), (j) assembled image (proposed scheme with 4:2:0 sub-sampling, B _x = B _y = 8, L _c = 0.002), (k) assembled image (conventional scheme, B _x = B _y = 8, L _c = 0.002).

Figure 19 shows the running time to assemble encrypted images by using the jigsaw puzzle solver [Reference Chuman, Kurihara and Kiya27–Reference Chuman, Kurihara and Kiya29], where the average time of 20 images from resized Ultra-eye dataset [Reference Nemoto, Hanhart, Korshunov and Ebrahimi39] were plotted. We compared the running time to assemble images encrypted with the color-based scheme (B _x = B _y = 16), the conventional scheme (B _x = B _y = 8), and the proposed one (B _x = B _y = 8). The jigsaw puzzle solver was implemented in MATLAB2017a on a PC with a 3.6 GHz processor and a main memory 16Gbytes (Processor:Intel Core i7-7700 3.6 GHz, OS:Ubuntu 16.04 LTS).

Fig. 19. Running time of assembling encrypted images from resized ultra-eye dataset (256 × 144).

As shown in Fig. 19, although the images encrypted by using the proposed method (B _x = B _y = 8) with 4:2:0 sub-sampling were solved in 10.33 min, the scores of assembled images were very low as L _c = 0.002 (see Table 4). It obviously takes more time to assemble encrypted images than that of images encrypted by the color-based scheme. The reason is that the proposed scheme can offer a smaller block size, the larger number of blocks and less color information. Moreover, the images encrypted by the proposed method without any sub-sampling were solved in 45.84 min which is almost the same as that of conventional one. As a result, the proposed scheme can enhance security against ciphertext-only attacks in terms of both computational complexity and the accuracy of assembled results.

D) Discussions on key space

Moreover, we evaluate robustness against brute-force attack in terms of the key space of encrypted images [Reference Nemoto, Hanhart, Korshunov and Ebrahimi39] where the number of pixels of the original images (N _p) is 256 × 144. The key space of the color-based scheme and the conventional scheme were calculated usingequations (6) and (8), respectively. On the other hand, the key space of the proposed scheme without sub-sampling and with sub-sampling were calculated by usingequations (8) and (12), respectively. As shown in Table 5, the key space of the proposed scheme without color sub-sampling is equal to that of the conventional one and greatly higher than the color-based scheme. Even if the key space of the proposed scheme with color sub-sampling is less than the conventional one due to the lower number of blocks, it is greatly higher than that of the color-based scheme.

Table 5. Key space of the color-based, conventional, and proposed scheme (N _p = 256 × 144)