Introduction

In this chapter we introduce low-density parity-check (LDPC) codes, a class of error correction codes proposed by Gallager in his 1962 PhD thesis 12 years after error correction codes were first introduced by Hamming (published in 1950). Both Hamming codes and LDPC codes are block codes: the messages are broken up into blocks to be encoded at the transmitter and similarly decoded as separate blocks at the receiver. While Hamming codes are short and very structured with a known, fixed, error correction ability, LDPC codes are the opposite, usually long and often constructed pseudo-randomly with only a probabilistic notion of their expected error correction performance.

The chapter begins by presenting parity bits as a means to detect and, when more than one is employed, to correct errors in digital data. Block error correction codes are described as a linear combination of parity-check equations and thus defined by their parity-check matrix representation. The graphical representation of codes by Tanner graphs is presented and the necessary graph theoretic concepts introduced.

In Section 2.4 iterative decoding algorithms are introduced using a hard decision algorithm (bit flipping), so that the topic is developed first without reference to probability theory. Subsequently the sum–product decoding algorithm is presented.

This chapter serves as a self-contained introduction to LDPC codes and their decoding. It is intended that the material presented here will enable the reader to implement LDPC encoders and iterative decoders.

Introduction

In this chapter we introduce turbo codes, the ground-breaking codes introduced by Berrou, Glavieux and Thitimajshima in 1993 [82], which sparked a complete rethink of how we do error correction. Turbo codes are the parallel concatenation of two convolutional codes which, at the receiver, share information between their respective decoders. Thus most of what we need for a turbo code has already been presented in the previous chapter. The turbo encoder uses two convolutional encoders (see Section 4.2) while the turbo decoder uses two copies of the log BCJR decoder (see Section 4.3).

The exceptional performance of turbo codes is due to the long pseudo-random interleaver, introduced below, which produces codes reminiscent of Shannon's noisy channel coding theorem, and to the low-complexity iterative algorithm that makes their implementation feasible.

In the first part of this chapter we discuss how the component convolutional codes and interleaver combine to form a turbo code. We then consider the properties of a turbo code that affect its iterative decoding performance and describe turbo code design strategies. Our aim here is to convey basic information about the encoding, decoding and design of turbo codes; a deeper understanding of these codes and of the decoding process is left to later chapters.

Turbo encoders

At the encoder, turbo codes use a parallel concatenation of two convolutional component encoders, as shown in Figure 5.1.

The field of error correction coding was launched with Shannon's revolutionary 1948 work showing – quite counter to intuition – that it is possible to transmit digital data with arbitrarily high reliability over noise-corrupted channels, provided that the rate of transmission is below the capacity of the channel. The mechanism for achieving this reliable communication is to encode a digital message with an error correction code prior to transmission and apply a decoding algorithm at the receiver.

Classical block and convolutional error correction codes were described soon afterwards and the first iterative codes were published by Gallager in his 1962 thesis; however they received little attention until the late 1990s. In the meantime, the highly structured algebraic codes introduced by Hamming, Elias, Reed, Muller, Solomon and Golay among others dominated the field. Despite the enormous practical success of these classical codes, their performance fell well short of the theoretically achievable performances set down by Shannon in his seminal 1948 paper. By the late 1980s, despite decades of attempts, researchers were largely resigned to this seemingly insurmountable theory–practice gap.

The relative quiescence of the coding field was utterly transformed by the introduction of “turbo codes”, proposed by Berrou, Glavieux and Thitimajshima in 1993, wherein all the key ingredients of successful error correction codes were replaced: turbo codes involve very little algebra, employ iterative, distributed, algorithms and focus on average (rather than worst-case) performance. This was a ground-shifting paper, forcing coding theorists to revise strongly held beliefs.

In the prisoners' problem, Alice and Bob are allowed to communicate but all messages they exchange are closely monitored by warden Eve looking for traces of secret data that may be hidden in the objects that Alice and Bob exchange. Eve's activity is called steganalysis and it is a complementary task to steganography. In theory, the steganalyst is successful in attacking the steganographic channel (i.e., the steganography has been broken) if she can distinguish between cover and stego objects with probability better than random guessing. Note that, in contrast to cryptanalysis, it is not necessary to be able to read the secret message to break a steganographic system. The important task of extracting the secret message from an image once it is known to contain secretly embedded data belongs to forensic steganalysis.

In Section 10.1, we take a look at various aspects of Eve's job depending on her knowledge about the steganographic channel. Then, in Section 10.2 we formulate steganalysis as a problem in statistical signal detection. If Eve knows the steganographic algorithm, she can accordingly target her activity to the specific stegosystem, in which case we speak of targeted steganalysis (Section 10.3). On the other hand, if Eve has no knowledge about the stegosystem the prisoners may be using, she is facing the significantly more difficult problem of blind steganalysis detailed in Section 10.4 and Chapter 12. She now has to be ready to discover traces of an arbitrary stegosystem. Both targeted and blind steganalysis work with one or more numerical features extracted from images and then classify them into two categories – cover and stego.

The goal of steganalysis is to detect the presence of secretly embedded messages. Depending on how much information the warden has about the steganographic channel she is trying to attack, the detection problem can accept many different forms. In the previous chapter, we dealt with the situation when the warden knows the steganographic method that Alice and Bob might be using. With this knowledge, Eve can tailor her steganalysis to the particular steganographic channel using several strategies outlined in Section 10.3. If Eve has no information about the steganographic method, she needs blind steganalysis capable of detecting as wide a spectrum of steganographic methods as possible. Design and implementation of practical blind steganalysis detectors is the subject of this chapter.

The first and most fundamental step for Eve is to accept a model of cover images and represent each image using a vector of features. In contrast to targeted steganalysis, where a single feature (e.g., an estimate of message length) was often enough to construct an accurate detector, blind steganalysis by definition requires many features. This is because the role of features in blind steganalysis is significantly more fundamental – in theory they need to capture all possible patterns natural images follow so that every embedding method the prisoners can devise disturbs at least some of the features. In Section 10.4, we loosely formulated this requirement as completeness of the feature space and outlined possible strategies for constructing good features.

The definition of steganographic security given in the previous chapter should be a guiding design principle for constructing steganographic schemes. The goal is clear – to preserve the statistical distribution of cover images. Unfortunately, digital images are quite complicated objects that do not allow accurate description using simple statistical models. The biggest problem is their non-stationarity and heterogeneity. While it is possible to obtain simple models of individual small flat segments in the image, more complicated textures often present an insurmountable challenge for modeling because of a lack of data to fit an accurate local model. Moreover, and most importantly, as already hinted in Chapter 3, digital images acquired using sensors exhibit many complicated local dependences that the embedding changes may disturb and leave statistically detectable artifacts. Consequently, the lack of good image models gives space to heuristic methods.

In this chapter, we discuss four major guidelines for construction of practical steganographic schemes:

1. • Preserve a model of the cover source (Section 7.1);

2. • Make the embedding resemble some natural process (Section 7.2);

3. • Design the steganography to resist known steganalysis attacks (Section 7.3);

4. • Minimize the impact of embedding (Section 7.4).

Steganographic schemes from the first class are based on a simplified model of the cover source. The schemes are designed to preserve the model and are thus undetectable within this model.

Steganalysis is the activity directed towards detecting the presence of secret messages. Due to their complexity and dimensionality, digital images are typically analyzed in a low-dimensional feature space. If the features are selected wisely, cover images and stego images will form clusters in the feature space with minimal overlap. If the warden knows the details of the embedding mechanism, she can use this side-information and design the features accordingly. This strategy is recognized as targeted steganalysis. The histogram attack and the attack on Jsteg from Chapter 5 are two examples of targeted attacks.

Three general strategies for constructing features for targeted steganalysis were described in the previous chapter. This chapter presents specific examples of four targeted attacks on steganography in images stored in raster, palette, and JPEG formats. The first attack, called Sample Pairs Analysis, detects LSB embedding in the spatial domain by considering pairs of neighboring pixels. It is one of the most accurate methods for steganalysis of LSB embedding known today. Section 11.1 contains a detailed derivation of this attack as well as several of its variants formulated within the framework of structural steganalysis. The Pairs Analysis attack is the subject of Section 11.2. It was designed to detect steganographic schemes that embed messages in LSBs of color indices to a preordered palette. The EzStego algorithm from Chapter 5 is an example of this embedding method. Pairs Analysis is based on an entirely different principle than Sample Pairs Analysis because it uses information from pixels that can be very distant.

In the previous chapter, we learned that one of the general guiding principles for design of steganographic schemes is the principle of minimizing the embedding impact. The plausible assumption here is that it should be more difficult for Eve to detect Alice and Bob's clandestine activity if they leave behind smaller embedding distortion or “impact.” This chapter introduces a very general methodology called matrix embedding using which the prisoners can minimize the total number of changes they need to carry out to embed their message and thus increase the embedding efficiency. Even though special cases of matrix embedding can be explained in an elementary fashion on an intuitive level, it is extremely empowering to formulate it within the framework of coding theory. This will require the reader to become familiar with some basic elements of the theory of linear codes. The effort is worth the results because the reader will be able to design more secure stegosystems, acquire a deeper understanding of the subject, and realize connections to an already well-developed research field. Moreover, according to the studies that appeared in [143, 95], matrix embedding is one of the most important design elements of practical stegosystems.

As discussed in Chapter 5, in LSB embedding or ±1 embedding one pixel communicates exactly one message bit. This was the case of OutGuess as well as Jsteg.

In Chapter 6, we learned that steganographic security can be measured with the Kullback–Leibler divergence between the distributions of cover and stego images. Four heuristic principles for minimizing the divergence were discussed in Chapter 7. One of them was the principle of minimal embedding impact, which starts with the assumption that each cover element, i, can be assigned a numerical value, ρ[i], that expresses the contribution to the overall statistical detectability if that cover element was to be changed during embedding. If the values ρ[i] are approximately the same across all cover elements, minimizing the embedding impact is equivalent to minimizing the number of embedding changes. The matrix embedding methods introduced in the previous chapter can be used to achieve this goal.

If ρ[i] is highly non-uniform, Alice may attempt to restrict the embedding changes to a selection channel formed by those cover elements with small ρ[i]. Constraining the embedding process in this manner, however, brings a fundamental problem. Often, the values ρ[i] are computed from the cover image or some side-information that is not available to Bob. Thus, Bob is generally unable to determine the same selection channel from the stego image and thus read the message. Channels that are not shared between the sender and the recipient are called non-shared selection channels. The main focus of this chapter is construction of methods that enable communication with non-shared selection channels.

Steganography is another term for covert communication. It works by hiding messages in inconspicuous objects that are then sent to the intended recipient. The most important requirement of any steganographic system is that it should be impossible for an eavesdropper to distinguish between ordinary objects and objects that contain secret data.

Steganography in its modern form is relatively young. Until the early 1990s, this unusual mode of secret communication was used only by spies. At that time, it was hardly a research discipline because the methods were a mere collection of clever tricks with little or no theoretical basis that would allow steganography to evolve in the manner we see today. With the subsequent spontaneous transition of communication from analog to digital, this ancient field experienced an explosive rejuvenation. Hiding messages in electronic documents for the purpose of covert communication seemed easy enough to those with some background in computer programming. Soon, steganographic applications appeared on the Internet, giving the masses the ability to hide files in digital images, audio, or text. At the same time, steganography caught the attention of researchers and quickly developed into a rigorous discipline. With it, steganography came to the forefront of discussions at professional meetings, such as the Electronic Imaging meetings annually organized by the SPIE in San Jose, the IEEE International Conference on Image Processing (ICIP), and the ACM Multimedia and Security Workshop. In 1996, the first Information Hiding Workshop took place in Cambridge and this series of workshops has since become the premium annual meeting place to present the latest advancements in theory and applications of data hiding.

The first steganographic techniques for digital media were constructed in the mid 1990s using intuition and heuristics rather than from specific fundamental principles. The designers focused on making the embedding imperceptible rather than undetectable. This objective was undoubtedly caused by the lack of steganalytic methods that used statistical properties of images. Consequently, virtually all early naive data-hiding schemes were successfully attacked later. With the advancement of steganalytic techniques, steganographic methods became more sophisticated, which in turn initiated another wave of research in steganalysis, etc. This characteristic spiral development can be expressed through the following quotation:

Steganography is advanced through analysis.

In this chapter, we describe some very simple data-hiding methods to illustrate the concepts and definitions introduced in Chapter 4 and especially Section 4.3. At the same time, we point out problems with these simple schemes to emphasize the need for a more exact fundamental approach to steganography and steganalysis.

In Section 5.1, we start with the simplest and most common steganographic algorithm – Least-Significant-Bit (LSB) embedding. The fact that LSB embedding is not a very secure method is demonstrated in Section 5.1.1, where we present the histogram attack. Section 5.1.2 describes a different attack on LSB embedding in JPEG images that can not only detect the presence of a secret message but also estimate its size.

Some of the first steganographic methods were designed for palette images, which is the topic of Section 5.2. We discuss six different ideas for hiding information in palette images and point out their weaknesses as well as other problematic issues pertaining to their design.

This book focuses on steganographic methods that embed messages in digital images by slightly modifying them. In this chapter, we explain the process by which digital images are created. This knowledge will help us design more secure steganography methods as well as build more sensitive detection schemes (steganalysis).

Fundamentally, there exist two mechanisms through which digital images can be created. They can be synthesized on a computer or acquired through a sensor. Computer-generated images, such as charts, line drawings, diagrams, and other simple graphics generated using drawing tools, could, in principle, be made to hold a small amount of secret data by the selection of colors, object types (line type, fonts), their positions or dimensions, etc. Realistic-looking computer graphics generated from three-dimensional models (or measurements) using specialized methods, such as ray-tracing or radiosity, are typically not very friendly for steganography as they are generated by deterministic algorithms using well-defined rules. In this book, we will mostly deal with images acquired with cameras or scanners because they are far more ubiquitous than computer-generated images and provide a friendlier environment for steganography. As with any categorization, the boundary between the two image types (real versus computer-generated) is blurry. For example, it is not immediately clear how one should classify a digital-camera image processed in Photoshop to make it look like Claude Monet's style of painting or a collage of computer-generated and real images.

In the previous chapter, we saw a few examples of simple steganographic schemes and successful attacks on them. We learned that the steganographic scheme called LSB embedding leaves a characteristic imprint on the image histogram that does not occur in natural images. This observation lead to an algorithm (a detector) that could decide whether or not an image contains a secret message. The existence of such a detector means that LSB embedding is not secure. We expect that for a truly secure steganography it should be impossible to construct a detector that could distinguish between cover and stego images. Even though this statement appears reasonable at first sight, it is vague and allows subjective interpretations. For example, it is not clear what is meant by “could distinguish between cover and stego images.” We cannot construct a detector that will always be 100% correct because it is hardly possible to detect the effects of flipping one LSB, at least not reliably in every cover. Just how reliable must a detector be to pronounce a steganographic method insecure?

Even though there are no simple practical solutions to the questions raised in the previous paragraph, they can in principle be studied within the framework of information theory. Imagine that Alice and Bob are engaging in a legitimate communication and do not use steganography. Let us suppose that they exchange grayscale 512 × 512 images in raster format that were never compressed.