Relaying is a well known technique to transmit signals from a source to a destination through one or several intermediate nodes (i.e., relay nodes (RNs)) without using increased power at the source [1–4]. In the past decade, many research efforts on relay technologies have been made to improve the cell coverage, enhance the transmission reliability, and increase the system throughput. More recently, RNs have become an important component in a heterogeneous cellular network (HCN) to provide service improvement and coverage extension at hotspots and cell edges. Layer three (L3) RN, which works as an independent base station (BS) except for the use of the wireless backhaul link, is specified in 3rd Generation Partnership Project (3GPP) Release 10 to realize flexible network deployment and increase network throughput without any additional infrastructure.

In terms of data forwarding, four types of relay have been widely studied, which are amplify-and-forward (AF), demodulate-and-forward (DMF), decode-and-forward (DCF), and estimate-and-forward (EF). Because there is no baseband signal processing function, an AF-relay-based wireless network is cost efficient. The main disadvantage of the AF relay is that the received noise and interference would also be forwarded to the destination. To mitigate the received noise and interference at an RN, the DMF relay and EF relay have been proposed to perform some simple signal processing according to the constellation used. With the decoding operation performed at the RN, a DCF relay can regenerate the source-transmitted signal perfectly if the received signal is decoded correctly.