2.1. Datasets and annotation criteria

Datasets have moved from simple classification tasks of the MNIST dataset, where each sample contain one class, to current big curated datasets, which include frames with multiple objects and different annotations and classes (Deng, Reference Deng2012; Xiao et al., Reference Xiao, Rasul and Vollgraf2017). Existing works on dataset annotation cover different issues that are strongly related to the specific task of the neural network. Some datasets used for object detection have been manually annotated using proprietary software or annotation tools, e.g. WoodScape (Uřičář et al., Reference Uřičář, Křížek, Sistu and Yogamani2019), RADIATE (Sheeny et al., Reference Sheeny, De Pellegrin, Mukherjee, Ahrabian, Wang and Wallace2021), or nuScenes (Caesar et al., Reference Caesar, Bankiti, Lang, Vora, Liong, Xu, Krishnan, Pan, Baldan and Beijbom2020). nuScenes has also published instructions for their labeling; however, these labels are open to interpretation and produce errors similar to the ones identified in other datasets, e.g. KITTI MoSeg dataset, as shown in Figure 1. Other datasets have been developed and annotated using deep learning methods such as active learning and neural networks (Angus et al., Reference Angus, ElBalkini, Khan, Harakeh, Andrienko, Reading, Waslander and Czarnecki2018; Janosovits, Reference Janosovits2022; Meyer & Kuschk, Reference Meyer and Kuschk2019). These methods of annotation are otherwise known as semiautomatic. Further techniques include the development of new annotation tools to adapt current datasets to specific use cases in AAD (Arief et al., Reference Arief, Arief, Zhang, Liu, Bhat, Indahl, Tveite and Zhao2020; Wang et al., Reference Wang, Acuna, Ling, Kar and Fidler2019). However, in the above-mentioned datasets, the specific criteria used to define how annotations should be implemented are missing, not explicit or ambiguous.

Figure 1.

Examples of errors (highlighted by dotted rectangles) in the BBs (green rectangles) of KITTI MoSeg (left) and nuScenes (right): upper frames show missing BBs, and lower frames show BBs not belonging to any objects

Interestingly, the VOC dataset has published the 2011 annotation guidelines used for labeling (Everingham et al., Reference Everingham, Van Gool, Williams, Winn and Zisserman2009). The guidelines provide guidance on which images to label and how they should be categorized. Examples include categorizing an object as occluded if more than 5% of the object within a BB is occluded, and there are images considered too difficult to segment and left unlabeled, e.g. a nest of bicycles.

2.2. Quality of annotations

Due to the effort and complexity required to annotate datasets, there are often errors associated with some of the labels. Recent works are trying to understand how these errors affect the DNN performance and how to improve the quality of the annotations. Notably, Ma et al. have presented a work in which they reannotated the MS COCO and Google Open Images datasets by providing a description of the object as well as providing some common examples of positive or negative cases which are open to the interpretation of the annotators (Ma et al., Reference Ma, Ushiku and Sagara2022). They have trained five neural network models based on the original labels and the new labels. For the COCO dataset, only some of the considered performance metrics improved due to reannotation, whereas the results on Google Open Image are improved after training and/or testing with the new annotations. Northcutt et al. provided a study into commonly used datasets for ML to understand and categorize the errors in labeling (Northcutt et al., Reference Northcutt, Athalye and Mueller2021). In addition, they provided results on benchmarking datasets, comparing the original labels with corrected labels. Their results showed that lower capacity/simpler networks are more robust against the effects of erroneous labels. Tsipras et al. have focused on ImageNet dataset (Tsipras et al., Reference Tsipras, Santurkar, Engstrom, Ilyas and Madry2020). The original ImageNet dataset provided only one class per image; however, each image may contain several secondary objects or clutter. Tsipras relabeled a subset of the ImageNet database (10 images per class, 10,000 images in total) and provided multiclass annotations where appropriate in their subset. The neural network performed worse on images that had multiple object classes in the scene with respect to images with only one object. Additionally, Tsipras et al. investigated the performance of the trained DNN when evaluating real-world data by comparing the neural network predictions with the classes identified by human annotators. More accurate models are in better agreement with the human annotators, and even when some network predictions are technically wrong, there is often an agreement with the classification by the human annotators (that is identification of dog breeds by nonspecialists).

2.3. DNN-based object detection

As previously mentioned, DNNs are expected to play a key role in AAD functions. One of the basic and most important tasks is the detection of the road stakeholders, and particularly vehicles, as they cause most of the accidents (Feng et al., Reference Feng, Harakeh, Waslander and Dietmayer2021; Rashed et al., Reference Rashed, Mohamed, Sistu, Kumar, Eising, El-Sallab and Yogamani2021). Object detectors can be broadly divided into two classes. Moreover, recently, there have been several implementations of object detectors using transformers (He et al., Reference He, Zhou, Li, Niu, Cheng, Li, Liu, Tong, Ma and Zhang2021; Carion et al., Reference Carion, Massa, Synnaeve, Usunier, Kirillov and Zagoruyko2020); a comprehensive review of object detectors is given in (Zaidi et al., Reference Zaidi, Ansari, Aslam, Kanwal, Asghar and Lee2022). Vision transformers are becoming very popular due to their promising performance; however, they usually require more epochs to converge and typically do not perform well on small objects (Yao et al., Reference Yao, Ai, Li and Zhang2021). In terms of the two “traditional” categories of object detectors, the one-stage and two-stage detectors, several architectures have been proposed through the years. Usually, one-stage detectors are faster but have lower accuracy than two-stage detectors. However, the two-stage Faster R-CNN have a good balance in terms of speed and accuracy; hence, they are frequently used for AAD functions.

As a part of the hereby proposed work, the authors have selected one implementation for each type of object detector (one-stage, two-stage, and transformer) to understand if the observed trends in the results, when reannotating the datasets, are common across different architectures for the same perception task, that is object detection.

3.1. KITTI MoSeg

The KITTI MoSeg dataset is a subset of the highly cited KITTI Benchmarking Vision Suite dataset (Siam et al., Reference Siam, Mahgoub, Zahran, Yogamani, Jagersand and El-Sallab2017; Geiger et al., Reference Geiger, Lenz and Urtasun2012). Compared to the KITTI dataset, KITTI MoSeg provides chronological sequences of frames, totaling 1449 frames (Siam et al., Reference Siam, Mahgoub, Zahran, Yogamani, Jagersand and El-Sallab2017). The KITTI MoSeg dataset provides 2D BB labels for car and van; the proposed work merged them into a single vehicle class. This choice was made due to the poor ratio between the two classes in the dataset.

In the KITTI MoSeg dataset, the original annotations were expanded by using the information available in the original dataset (3D BB, odometry information), where 3D ground truth BBs were converted into 2D BBs, associating BBs across chronological frames to obtain estimated velocity per BB. Finally, a filtering process was applied to keep objects consistently identified across frames (Siam et al., Reference Siam, Mahgoub, Zahran, Yogamani, Jagersand and El-Sallab2017). As the experiments proposed in this paper entail manually reannotating the same dataset three times, a dataset with moderate size was the best choice. However, the BB errors tackled in this paper are common across different automotive datasets.

3.2. Re-annotation criteria

From preliminary findings, the original labels in some automotive datasets were visually inspected to identify potential BB problems which can affect DNN evaluation metrics. The majority of the identified errors can be categorized using the definitions below:

• • Missing: There are clear and obvious vehicles in the frames which are not labeled.

• • Incorrect: There is no object in the labeled class in the BB.

• • Bad Fit: The BB size and/or position are not appropriate for the identified object.

• • Occlusion: The BB predicts the full size of the object, not only what is visible.

Based on these identified problems, three sets of BB criteria were defined to ensure that the annotation process is more consistent, see Table 1. These sets of criteria were implemented one at a time to completely reannotate the KITTI MoSeg manually, using an ad hoc Matlab app developed by the authors. Although human labeling error can still exist, some of the criteria can be coded to ensure the drawn BB meets the criteria. The defined sets of criteria are not specific to this dataset and can be applied to any other datasets.

Table 1.

The three sets of criteria for the reannotation of the KITTI MoSeg dataset for object detection

* Incorrect stands for BBs that clearly do not belong to any target objects

For Criteria 1 (C1), the guidelines are defined as much as possible in an objective way and based on what features are expected to be learnt by the neural network (that is wheels, shapes, etc.). C1.1 specifies the minimum dimension of a BB based on an analysis of box sizes in the original KITTI MoSeg labels. In the specific automotive use case, vehicles in the far distance cover less pixels, and they are not of immediate safety concern. However, the minimum BB dimension can be tailored based on the detection requirements. For C1.2, a 3-pixel error was defined to consider annotation inaccuracies to identify the exact edge of an object and issues due to the lack of contrast at the edges, but avoiding significant errors at the smallest size based on C1.1. C1.3 ignores occluded regions from the BB, avoiding incorrect features to be learnt by the network. A minimum threshold of 20% is applied in C1.4 to determine the minimum visibility of the object. The 20% visibility is inline with the 20% visibility by nuScenes nuImages Annotator Instructions (Caesar et al., Reference Caesar, Bankiti, Lang, Vora, Liong, Xu, Krishnan, Pan, Baldan and Beijbom2020). However, the determination of the 20% is subjective to the annotator or even to an “automated” implementation as it is based on estimation.

In Criteria 2 (C2), the labels were created to the best of the annotator ability to identify vehicles in each frame but still keeping a small set of guidelines. The annotator was one of the authors. Object identified according to C2 are expected to be closer to real world, where all the vehicles should be identified by the DNN.

Finally, in Criteria 3 (C3), the annotator did not resize nor add any BBs, but only removed the BBs not belonging to real objects; in the case of multiple BBs encompassing the same vehicle, the annotator kept only the BB deemed as the best fit for that vehicle.

3.3. Neural networks

The four annotated dataset variations were used to train three different network architectures: Faster R-CNN (Ren et al., Reference Ren, He, Girshick and Sun2015), YOLOv5 (Jocher et al., Reference Jocher, Chaurasia, Stoken, Borovec, Kwon, TaoXie, Michael, Fang, imyhxy, Wong, Yifu, Montes, Wang, Fati, Nadar, Skalski, Hogan, Strobel, Jain, Mammana and xylieong2022), and DETR (Carion et al., Reference Carion, Massa, Synnaeve, Usunier, Kirillov and Zagoruyko2020). Faster R-CNN is an example of two-stage detector, which predicts BBs from region proposals. On the other hand, YOLO is a single-stage detector that does BB regression from anchors. Finally, DETR directly performs BB prediction with respect to the input image. For each network model described in the list below, the training process was performed four times from the base model (one for each dataset variation based on the annotation criteria, that is Original, C1, C2, and C3) on the whole original or relabeled training image set, leading to four different trained DNNs per architecture.

• • Faster R-CNN. The network of choice consists of a ResNet-50 backbone with FPN (Feature Pyramid Network) feature extractor from the torchvision library (He et al., Reference He, Zhang, Ren and Sun2016; Lin et al., Reference Lin, Dollár, Girshick, He, Hariharan and Belongie2017). It was originally pretrained on COCO and then fine-tuned over the dataset of interest. The training was performed using the AdamW optimizer with a learning rate set to $ {10}^{-3} $ and weight decay set to 0.2. In addition, learning rate scheduler with gamma 0.9 and step size 25 was used.

• • YOLOv5. The training was performed starting from the pre-trained yolov5m with most parameters left to their default value. The backbone was frozen, image size set to 640 px, and optimiser set to AdamW.

• • DETR. The network of choice consists of a ResNet-50 convolutional feature extractor from the hugging face library (He et al., Reference He, Zhang, Ren and Sun2016). It was originally pretrained on COCO and then fine-tuned over the dataset of interest. The training was performed using the AdamW optimiser with learning rate set to $ {10}^{-5} $ for the backbone and $ {10}^{-4} $ for the other layers and weight decay set to $ {10}^{-4} $ .

3.4. Evaluation metrics

The trained networks were all evaluated and compared using $ {mAP}_{50} $ , due to the presence of small size objects in the dataset. In fact, minor location offsets and size errors for predicted boxes can result in a much lower Intersection over Union (IoU) for small objects. For safety critical functions in automated vehicle, it is still important to detect the object, even if the location/size may be slightly off. The YOLOv5 repository provides this evaluation metric, and the torchmetrics library was used to compute $ {mAP}_{50} $ for Faster R-CNN and DETR implementations. Thus, the three reannotation criteria are compared using the three main types of neural network-based object detectors.