A) Feature extraction and fusion by 2D geometry feature learning module

Previous methods failed to predict an accurate disparity in ill-posed regions because the network did not understand the context well. On the other hand, semantic segmentation methods have a fantastic performance in understanding the context. Thus, we are inspired by many semantic segmentation methods and get a robust descriptor that determines the context relationship from the pixel (particularly for ill-posed regions) via the SPP struct. We design the 2D geometry feature learning (GFL) module, as shown in Fig. 2. For description convenience, we set the input resolution of the stereo pairs to be $H \times W$, and the basic number F of the convolution filter to be $32$.

Fig. 2.

The 2D geometry feature learning module (GFL). $x \times x,s,f$ denote the size of the convolution kernel, stride, and the number of convolution filters respectively. ${\times} n$ denotes the block repeats n times.

In this module, we apply a series of 2D convolutional operations to extract the semantic information, and each convolutional operation is followed by a batch normalization (BN) layer and a rectified linear unit (ReLU) layer. The GFL module consists of the unary feature extraction part and the multi-scale feature fusion part.

B) Cost volume construction by variance-based cost metric

In previous works [Reference Kendall, Martirosyan, Dasgupta and Henry13, Reference Zagoruyko and Komodakis19–Reference Zhong, Dai and Li21], the cost volume is the critical step which links 2D and 3D convolution. To achieve better performance, we aggregate the semantics feature of left and right image ${V_l},{V_r}$ to one cost volume C via variance-based cost metric (VBCM) ${\mathcal M}$. Let $W,H,D,F$ be the input image height and width, the disparity sample number, and the feature number. Thus, the size of the semantics feature is ${V_l} = {V_r} = (H/4) \times (W/4) \times F$, and the size of cost volume $C = (D/4) \times (H/4) \times (W/4) \times F$. We define the cost metric as the mapping ${{\mathcal M}}:\underbrace{{\{ {V_\textrm{l}},{V_{r,1}}\}, \ldots, \{ {V_\textrm{l}},{V_{r,(D/4)}}\} }}_{{D/4}} \to C$ that:

(1)\begin{equation}\begin{aligned} C& = {\mathcal M}\left( {\{{{V_l},{V_{r,1}}} \}, \cdots,\Big\{ {{V_l},{V_{r,\frac{D}{4}}}} \Big\}} \right)\\ & = stack\left( {\dfrac{{{{({{V_l} - \overline {{V_i}} } )}^2} + {{({{V_{r,i}} - \overline {{V_i}} } )}^2}}}{2}} \right), \end{aligned}\tag{1}\end{equation}

where ${V_{r,i}}$ means traversed right semantics feature with a preset disparity range i, $\overline {{V_i}}$ means the average of ${V_l}$ and ${V_{r,i}}$, and all operations above are element-wise.

Most traditional stereo matching methods aggregate the cost volume between the left and right images in a heuristic way. However, recent works apply the concatenation operation instead of the mean or subtraction operation [Reference Kendall, Martirosyan, Dasgupta and Henry13, Reference Zhang, Fang, Min, Sun, Yan and Tian36]. This is the way that depends on network learning entirely. Here we choose the variance-based operation instead of the concatenation operation, due to which provides no direction about what the networks should do in the feature matching module. In contrast, our variance-based operation explicitly measures the left–right feature difference, which reflects the similarities between them and saves about half of memory. The true matched pair should have the lowest cost value, whereas it should have a higher cost. The output size of variance-based cost volume is $D/4 \times H/4 \times W/4 \times F$.

C) Feature matching by non-local attention matching module

To regularize the matching cost volume along the disparity dimension as well as spatial dimensions, we propose a 3D CNN architecture for learning the matching feature: the NLAM module. In [Reference Wang, Girshick, Gupta and He25], the non-local block was designed to compute the response at a position as a weighted sum of the features at all positions, and it showed a significant improvement for video classification and poses estimation. However, the cost volume is too big for the non-local block, leading it cannot be directly applied to the matching task. From another point, the essence of the non-local block is the attention mechanism. Therefore, we could combine the non-local block and the hierarchical 3D convolution for setting up the NLAM module, as shown in Fig. 3.

Fig. 3.

The non-local attention matching module (NLAM). The NLAM module consists of feature matching part and scale recovery part. Note that the feature maps are shown as feature dimensions, e.g. $D \times H \times W \times F$means a feature map with disparity number$D$, height$H$, width W, and feature number$F$. Here, $L\ast$denotes different scale levels of the feature maps.

In this module, we apply a series of 3D convolutional operations to obtain the matching volume, and each convolutional operation is followed by a BN layer and a ReLU layer. The NLAM module consists of feature matching part and scale recovery part.

(1) The feature matching part

The feature matching part contains $26$ convolutions with stride one or two for regularizing the variance-based cost volume. This part has four levels, and we pass the feature maps between the same level to form the residual architecture, avoiding losing the critical information. Each level consists of an up-sampling or a sub-sampling convolution, and a residual block. After the $L4$ level, we adopt a non-local block as an attention block for further improving global matching learning.

To further understand the non-local attention mechanism, our feature matching part could be viewed as the group of the hierarchical 3D convolution block and the non-local block. The hierarchical 3D convolution block is an encoder–decoder architecture; it encodes the feature map by sub-sampling and decodes the encoded feature by up-sampling, as shown in Fig. 4.

Fig. 4.

The encoder–decoder architecture. The pink block means the encoding process. The green block means the decoding process.

As shown in Fig. 4, we define the encoder and decoder process, respectively as:

(2)\begin{equation}\begin{aligned} {E_{i + 1}} & ={{\mathcal F}_{{E_{i + 1}}}}({{E_i}} )\\ {D_i} & ={{\mathcal F}_{{D_i}}}({{D_{i - 1}}} )+ {E_{n - i}}, \end{aligned}\tag{2}\end{equation}

where n denotes the number of the encoder and decoder, i denotes the level of the encoder or decoder, and ${{\mathcal F}_E}$ or ${{\mathcal F}_D}$ denotes the process of each encoder or decoder.

In the encoder–decoder architecture, the worst drawback is that the convolution is a local operation. It causes the network to not further improve the receptive fields to evaluate the impact of global information on the current pixel. Thus, we use the non-local block as attention mode for promoting the understanding ability of global context, as shown in Fig. 3.

The non-local operation ${{\mathcal N}}({\cdot} )$ could be represented as:

(3)\begin{equation}{y_i} = \frac{1}{{{{\mathcal C}}(x )}}\sum\limits_{\forall j} f ({{x_i},{x_j}} )g({{x_j}} ),\tag{3}\end{equation}

where $x,y$ denotes the input and output, respectively, i denotes the index of an output position, j denotes the index of all possible positions, $f({\cdot} )$ denotes the response function of global influence on current position ${x_i}$, $g({\cdot} )$ denotes a representation of the input signal at the position ${x_j}$, and ${{\mathcal C}}({\cdot} )$ denotes the total influence for normalizing the response.

In our non-local block, we apply the Gaussian function to compute similarity in an embedding space. We set the response function $f({\cdot} )$ as:

(4)\begin{equation}f({{x_i},{x_j}} )= {e^{\theta {{({{x_i}} )}^T}\phi ({{x_j}} )}},\tag{4}\end{equation}

where $\theta ({{x_i}} )= {W_\theta } \cdot {x_i}$, $\phi ({{x_j}} )= {W_\phi } \cdot {x_j}$. Similarly, $g({\cdot} )$ could be set as $g({{x_j}} )= {W_g} \cdot {x_j}$. Thus, the ${{\mathcal C}}({\cdot} )$ could be set as ${{\mathcal C}}(x )= {\sum _{\forall j}}f({{x_i},{x_j}} )$. In this response function, the process of $(1/{{\mathcal C}}(x)){\sum _{\forall j}}f({{x_i},{x_j}} )$ could be viewed as a softmax operation. In addition, we add y and x for a residual learning. In our architecture, the non-local block could be defined as:

(5)\begin{equation}{D_0} = {{\mathcal N}}({{E_4}} )+ {E_4}.\tag{5}\end{equation}

After the non-local attention step, we feed the fused global feature into the decoding process as presented in equation (2) or Fig. 4. The non-local block could improve the performance of matching effectively. After this part, we could obtain the matching volume but in a low resolution $1/4H \times 1/4W \times 1/4D$. Thus, we should recover the scale to get the final matching volume.

D) Disparity map regression by soft argmin

In this step, we will estimate the initial disparity map from the matching volume. Thus, we naturally embed our matching volume into a 3D to 2D process. The simplest way to recover the initial disparity map $\hat{d}$ from the matching volume M is the pixel-wise winner-take-all such as an argmax operation. However, this way is unable to predict sub-pixel estimation and less robust [Reference Yao, Luo, Li, Fang and Quan5, Reference Kendall, Martirosyan, Dasgupta and Henry13]. Thus, we predict the disparity map by passing an argmin operation. First, we convert the matching volume M to the probability volume ${{\mathcal P}}$ via the softmax operation $\sigma ({\cdot} )$. Then, we take the sum of each disparity d weighted with its probability. The soft argmin process is defined as:

(6)\begin{equation}\hat{d} = \mathop \sum \limits_{d = 0}^{{D_{\max }}} d \times {{\mathcal P}}(d )= \mathop \sum \limits_{d = 0}^{{D_{\max }}} d \times \sigma ({ - {M_d}} ),\tag{6}\end{equation}

where ${{\mathcal P}}(d )$ denotes the probability estimation for all pixels of the image at disparity d. ${M_d}$ denotes all value of the $d$-th layer in the matching volume M.

The above method could accurately approximate the disparity d in the range from $0$ to ${D_{\max }}$. The output initial disparity map is the same size as the input image.

E) Disparity map refinement by geometry refinement module

The initial disparity map from the probability volume is a qualified output, but the boundaries may suffer from over-smoothing in the recovery size part of the NLAM module, or the completeness of the object suffers from the missing piece in the occlusion area. Notice that the input image contains complete boundary information, and the output of 2D GFL module contains the semantics feature, we thus use the input image and the semantics feature as guidance to refine the initial disparity map. Inspired by the recent multi-view stereo algorithm [Reference Yao, Luo, Li, Fang and Quan5], we redesign a geometry refinement (GR) module at the end of NLCA-Net, as shown in Fig. 5.

Fig. 5.

Geometry refinement module (GR). The initial disparity map, the left image, and the semantics feature are fed to the GR module. After this module, we get refined disparity map. Here, blue block means the 32 convolutions with the size$3 \times 3$, and green block means the $1$ convolution with the size$3 \times 3$.

In the GR module, the initial disparity map, the left image, and the semantics feature are concatenated as the input. Notice that the size of the semantics feature is only quarter in width and height dimension compared to input images. Therefore, we first upsample the semantics feature, then use the $1 \times 1$ convolutional filter to adjust it. Next, we concatenate them as a $38$-feature input and send them to eight-layer residual network, which consists of the $32$ convolutional filters with the size $3 \times 3$. After that, we apply one convolutional filter to obtain the difference, and the initial disparity map is added back to generate the refined disparity map. The last layer does not contain the BN layer and the ReLU layer. After the GR module, we could get the refined disparity map.

F) Loss function

The loss functions consider both the initial disparity map and the refined disparity map. We use the ${L_1}$ loss to evaluate the difference between the ground truth and the predicted disparity map. Due to the ground truth sparse disparity map, we only consider those pixels which are valid. The $Los{s_{{L_1}}}$ is defined as:

(7)\begin{align} Los{s_{{L_1}}} =\dfrac{1}{{{N_1}}}\sum\limits_{p \in {P_{valid}}} {||d(p )- {{\hat{d}}_i}(p )|{|_1}} + ||d(p )- {{\hat{d}}_r}(p )|{|_1},\tag{7}\end{align}

where ${P_{valid}}$ denotes the set of valid ground truth pixels, ${N_1}$ denotes the total number of valid ground truth pixels, $d(p )$ denotes the ground truth of pixel p, ${\hat{d}_i}(p )$ denotes the initial disparity map value of pixel p, and ${\hat{d}_r}(p )$ denotes the refined disparity map value of pixel p.

To further improve the robustness and performance of the non-occluded area, we apply the warping loss to our loss function. The warping function is widely used in the traditional methods, and the previous learning method utilizes the warping function to structure the warping loss which gives the network self-supervised ability [Reference Zhong, Dai and Li21]. To solve the different illumination between left–right images, we introduce a structural similarity (SSIM) term $S({\cdot} )$ [Reference Wang, Bovik, Sheikh and Simoncelli37] to improve the robustness. The $Los{s_w}$ is defined as:

(8)\begin{align} Los{s_w}({{I_L},I_L^\prime } ) =\dfrac{1}{{{N_2}}}\mathop \sum \limits_{p \in {\textbf{n}_{valid}}} {\lambda _1}\dfrac{{1 - S({{I_L},I_L^\prime } )}}{2} + {\lambda _2}|{{I_L} - I_L^\prime } |, \tag{8}\end{align}

where ${\textbf{n}_{valid}}$ denotes the set of pixels in the non-occluded area, ${N_2}$ denotes the total number of valid pixels in the non-occluded area, ${I_L}$ denotes the left image, ${I^{\prime}_L}$ denotes the warping image which is reconstructed from the right image and the refined disparity map, and ${\lambda _1},{\lambda _2}$ denote the balance between structural similarity and image appearance difference.

Our loss function for stereo matching is defined as:

(9)\begin{equation}Loss = \alpha Los{s_{{L_1}}} + \beta Los{s_w},\tag{9}\end{equation}

where $\alpha$ denotes the weight of $Los{s_{{L_1}}}$, and $\beta$ denotes the weight of $Los{s_w}$.

A) Implementation details

In this section, we implement NLCA-Net by Tensorflow with $5.46$ M trainable parameters, and the code will be released at the Github website.Footnote ¹ To obtain the final model, we should choose the hyper-parameters, train, and evaluate the model.

(3) Evaluating metric

For evaluating our model and comparing with the state-of-the-art methods published recently, we show our results' errors with the following metrics, which have been widely used in the website of KITTI dataset:

\begin{align*} MAE & = \frac{1}{{|N |}}\mathop \sum \limits_{i \in T} |{\hat{d} - d} |,\\ {D_{1 - bg}} & = \frac{{\mathop \sum \limits_{i \in bg} [{|{\hat{d} - d} |\gt m \wedge i = vaild} ]}}{{\mathop \sum \limits_{i \in bg} [{i = vaild} ]}},\\ {D_{1 - fg}} & = \frac{{\mathop \sum \limits_{i \in fg} [{|{\hat{d} - d} |\gt m \wedge i = vaild} ]}}{{\mathop \sum \limits_{i \in fg} [{d = vaild} ]}},\\ {D_{1 - all}} & = \frac{{\mathop \sum \limits_{i \in T} [{|{\hat{d} - d} |\gt m \wedge i = vaild} ]}}{{\mathop \sum \limits_{i \in T} [{d = vaild} ]}},\end{align*}

where d denotes the ground truth of disparity, $\hat{d}$ denotes the estimated depth, m denotes the threshold of error pixel, $[{\cdot} ]$ denotes the Iverson bracket, $bg$ represents the set of all points in background regions of the images, $fg$ represents the set of all points in foreground regions of the images, and T represents the set of all points in the images.

B) Ablation study for NLCA-Net

To verify our design's effectiveness, we conduct experiments with different settings to evaluate NLCA-Net on the SceneFlow dataset. We train different model variants like the pre-training process, as presented in Section A. We first compare the performance of different settings, including the 2D GFL module, VBCM, the NLAM module, and the GR module, as shown in Table 1. Then, we present the representative results of our model and ablation study of loss weight, as shown in Fig. 6 and Table 2. Moreover, we test the impact of the different numbers of the non-local on the model, as shown in Table 3.

Table 1.

Evaluation of NLCA-Net with different settings.

Computed the percentage of three-pixel-error on the SceneFlow and KITTI 2015 test set.

Fig. 6.

SceneFlow test data qualitative results. From left: left stereo input image, ground-truth, disparity prediction.

Table 2.

Influence of weight values for ${\lambda _1}$, ${\lambda _2}$, $\alpha$, and $\beta$ on three-pixel-error.

We empirically found that 0.85/0.15/0.8/0.2 yielded the best performance on the SceneFlow test set.

Table 3.

Influence of the different numbers of the non-local blocks on the model.

Here R denotes the number of the non-local blocks.

As shown in Fig. 6 and Table 1, it qualitatively demonstrates the benefits of using the modules which we proposed. First, the 2D GFL and NLAM modules show a significant performance improvement for the matching accuracy. For the GFL module, it enhances the scene understanding ability of the model effectively; for the NLAM module, it has a strong regularization ability to learning the matching rules and can facilitate the learning process. Second, the GR module and $Los{s_w}$ function improve the matching accuracy a little. On a good baseline, the GR module could further improve the performance and $Los{s_w}$ function could achieve $1\%$ improvement on whole pixels and $4\%$ on non-occluded pixels. Finally, the VBCM provides better testing accuracy compared with traditional concatenation way, and it saves half-memory of the cost volume, which reduces about 300 M running memory of the whole framework.

As shown in Table 2, we conduct experiments with various combinations of loss weights which have the relationship about ${\lambda _1} + {\lambda _2} = 1$ and $\alpha + \beta = 1$. For the baseline, we only use the ${L_1}$ loss function and then continuously adjust the loss weights to find the weights which yield the best performance. The result shows that the weight settings of $0.85$ for ${\lambda _1}$, $0.15$ for ${\lambda _2}$, 0.8 for $\alpha$, and 0.2 for $\beta$ obtain the best performance, which is a $2.87\%$ three-pixel-error rate on the SceneFlow test set and a $1.96\%$ three-pixel-error rate on the KITTI test set.

Finally, we also test our model with different non-local block (NL-block) numbers. In this part, we set a model without NL-block as the baseline. The impact of NL-block is tested by increasing the NL-block number in the model. As shown in Table 3, we list the results with different NL-blocks on the SceneFlow and KITTI 2015, indicating that performance can be improved by introducing NL-block into the model. It is also noticed that the improvement decays gradually as the number of NLB increases, showing the boundary effect.

C) KITTI 2012 and 2015 benchmark results

To evaluate the performance of our model, we compare the performance of NLCA-Net with other state-of-the-art methods on the KITTI dataset. We use the multi-step training method to train the model, as shown in Section A. We present the representative images of our model and other state-of-the-art methods, as shown in Figs 7 and 8. Then, we evaluate the performance of our model on the KITTI website,Footnote ² as shown in Tables 4 and 5. Besides, we compare our model with other competing algorithms in the reflective regions, as presented in Table 6.

Fig. 7.

KITTI 2012 test data qualitative results. We compare our approach with state-of-the-art methods (HD³-S and GwcNet), and we highlight our advantage in the error maps. Note that, in the error maps, the deeper red pixels mean higher error rate in the occluded regions and white pixels denote ≥5 pixels error in the non-occluded regions.

Fig. 8.

KITTI 2015 test data qualitative results. From left: left stereo input image, disparity prediction, error map. Note that, in the error maps, depicting correct estimates (<3 px or <5% error) in blue and wrong estimates in red color tones.

Table 4.

Results on KITTI 2012 stereo benchmark.

Average end-point-error (EPE) and the percentage of different pixel-error are used for evaluations on the KITTI 2012 test set. Compared with other algorithms, our approach achieves the best performance in most cases.

* The number of the non-local blocks is 3.

Table 5.

Results on KITTI 2015 stereo benchmark.

Our approach achieves comparable performance to state-of-the-art methods.

* The number of the non-local blocks is 3.

Table 6.

Comparisons of different state-of-the-art methods in the reflective regions.

Average end-point-error (EPE) and the percentage of different pixel-error are used for evaluations on the KITTI 2012 test set. Compared with other algorithms, our approach achieves the best performance in most cases.

* The number of the non-local blocks is 3.

As shown in Figs 7 and 8, the proposed method could produce dense and clear disparity maps. For the non-occluded region, the proposed method shows the powerful performance; the disparity maps predicted by NLCA-Net are sharper and more complete than other learning methods. Even if in the occluded region, the proposed method still provides high-level performance and significantly outperforms other state-of-the-art methods, as shown in Fig. 9. The results show that our method can notably reduce the error rate on the occluded area and handle well with the large textureless regions. It also means the semantic information plays an important role, which offers the boundary information to perfect the edge pixels of objects. Overall, the proposed method shows the incredible expressiveness in the matching task.

Fig. 9.

Part zoom-up of the error maps on the occluded region. From left: original image, HD³-Stereo, GwcNet, and ours. The result shows that our method can notably reduce the error rate on the occluded area and handle well with the large textureless regions.

As shown in Tables 4 and 5, it qualitatively demonstrates the performance of the proposed method. Compared with other methods, the proposed method yields more precise and robust disparity maps, particularly in the non-occluded regions. For the KITTI 2012 dataset, our approach is very close to HD³-Stereo [Reference Yin, Darrell and Yu22] on non-occluded (Out-Noc) in the error rate of 2 pixels; but for other quality indexes, our method all achieves the best performance as shown in the table. For the KITTI 2015 dataset, our approach shows comparable performance and markedly outperforms other competing algorithms, including the previous best result (HD³-Stereo [Reference Yin, Darrell and Yu22] and GwcNet [Reference Guo, Yang, Yang, Wang and Li18]). In short, the proposed approach is superior to state-of-the-art methods in most cases.

As shown in Table 6, it qualitatively shows the advantage of our model in the reflective regions. In our designs, we use attention mechanisms and semantic information to enhance the ability of scene understanding. Thus, in the reflective region, the proposed method achieves the best performance of all quality indexes and significantly outperforms other state-of-the-art methods. It indicates the effectiveness of our matching network based on the contextual attention mechanism, which exhibits robustness to the reflective regions.