In prior chapters we discussed how Dennard’s scaling combined with Moore’s law has resulted in continuous increase in single-threaded performance, through innovations to exploit instruction-level parallelism (ILP). Designs such as out-of-order (OoO) execution and speculation have been used to exploit the scaling properties of transistors. Recently, Dennard’s voltage scaling has hit its limits, with the supply voltage reduction coming to a near halt. Thus, power density grows as more transistors are integrated into a unit area. In fact, Moore’s law scaling seem to keep its momentum, leading to billions of transistors being integrated into chips. Overall, it is fair to say that the density of transistors has been scaling faster than power density. Recognizing this concern, the chip industry has shifted (at least partially) emphasis toward multi- and even many-core chip multiprocessors (CMPs). While scaling frequency has a cubic relationship to power consumption, scaling the cores has a linear relationship to the power. Graphics processing units (GPUs) have emerged as a promising many-core architectures for power-efficient throughput computing. With thousands of simple in-order cores that can run thousands of threads in parallel, GPUs derive several tera-flops of peak performance, primarily through thread-level parallelism (TLP).