Since the groundbreaking performance improvement by AlexNet at the ImageNet challenge, deep learning has provided significant gains over classical approaches in various fields of data science including imaging reconstruction. The availability of large-scale training datasets and advances in neural network research have resulted in the unprecedented success of deep learning in various applications. Nonetheless, the success of deep learning appears very mysterious. The basic building blocks of deep neural networks are convolution, pooling, and nonlinearity, which are primitive tools of mathematics. Interestingly, the cascaded connection of these primitive tools results in superior performance over traditional approaches. To understand this mystery, one can go back to the basic ideas of the classical approaches to understand the similarities and differences from modern deep-neural-network methods. In this chapter, we explain the limitations of the classical machine learning approaches, and provide a review of mathematical foundations to understand why deep neural networks have successfully overcome their limitations.

Inspired by the success of deep learning in computer vision tasks, deep learning approaches for various MRI problems have been extensively studied in recent years. Early deep learning studies for MRI reconstruction and enhancement were mostly based on image-domain learning. However, because the MR signal is acquired in the k-space domain, researchers have demonstrated that deep neural networks can be directly designed in k-space to utilize the physics of MR acquisition. In this chapter, the recent trend of k-space deep learning for MRI reconstruction and artifact removal are reviewed. First, scan-specific k-space learning, which is inspired by parallel MRI, is covered. Then we provide an overview of data-driven k-space learning. Subsequently, unsupervised learning for MRI reconstruction and motion artifact removal are discussed.

Ultrasound imaging (US) is susceptible to several types of artifacts. Most artifacts appear because of transducer limitations and simplified assumptions on the wave propagation. The artifacts are sometimes used as a component that contains tissue information; however, they often lead to a misinterpretation in the clinical diagnosis. Therefore, to improve the clinical utility of ultrasound in difficult-to-image patients and settings, a number of artifact removal methods have been proposed that aim at boosting image quality. Classical optimization-based methods have severe limitations due to their limited performance and high computation requirements. Furthermore, it is difficult to obtain parameters for producing high-quality output. A quick remedy for the aforementioned issues is the deep learning approach, which offers high performance compared with the traditional methods despite the significantly reduced runtime complexity. Another big advantage is that the same parameters as those learned during the training phase can be used to process different input images. This has motivated the scientific community to design deep-neural-network-based approaches for US artifact removal tasks.

From unconditional synthesis to conditional synthesis, GANs have shown that they are a perfect fit for such problems thanks to their ability to learn probability distributions. For unconditional synthesis, the objective is to stochastically generate MR images of target contrast. Conditional synthesis refers to the case where the model learns nonlinear mapping to the different MR tissue contrasts without altering the physiological information. Furthermore, by merging collaborative information of multiple contrast images, missing data imputation among many different domains is also effectively solved with GANs. Although promising results are seen, development in the area is still at its early stage. Interesting research directions are proposed from prior work, including the application of more advanced methods and rigorous validation in clinical settings. In effect, MRI image synthesis techniques should be able to reduce the burden of costly MR scans, benefiting both patients and hospitals.

With the present demand for high-quality image reconstruction and signal extraction from less (e.g., unfocused or parallel) transmissions that facilitate fast imaging, and the push towards compact probes, modern ultrasound imaging leans heavily on innovations in powerful digital receive channel processing. Beamforming, the process of mapping received ultrasound echoes to the spatial image domain, naturally lies at the heart of the ultrasound image formation chain. In this chapter, we discuss why and when deep learning methods can play a compelling role in the digital beamforming pipeline, and then show how these data-driven systems can be leveraged for improved ultrasound image reconstruction.

In this chapter, we show that image-domain deep-learning-only reconstruction methods have intrinsic limitations in reconstruction accuracy and generalizability to individual patients owing to the regressive nature of the method. The combination of deep learning methods with analytic reconstruction methods or statistical IR methods offers a promising opportunity to achieve personalized reconstruction with improved reconstruction accuracy and enhanced generalizability.