2.1. ALPR systems

ALPR systems consist of two primary components: LPD and LPR. In the literature, various studies have proposed ALPR systems that address these two sub-tasks either through a single end-to-end trainable network or a two-stage system. The former typically leverages a multi-task learning approach, using shared features between the LPD and LPR tasks. For instance, Qin et al. [Reference Qin and Liu6] used feature pyramid networks (FPNs) [Reference Lin, Dollár, Girshick, He, Hariharan and Belongie17] to improve the feature extraction process of convolutional neural networks (CNNs). Li and al. [Reference Li, Wang and Shen18] used a region proposal network [Reference Ren, He, Girshick, Sun, Cortes, Lawrence, Lee, Sugiyama and Garnett19] on top of CNN extracted features to generate region proposals. These two methods are based on region of interest pooling to extract a fixed-size feature map, which is then processed through two branches: fully connected layers for bounding box regression, and recurrent neural networks (RNNs) with connectionist temporal classification (CTC) loss for LPR. For step-wise or two-module ALPR systems, for example, Kessentini et al. [Reference Kessentini, Besbes, Ammar and Chabbouh20] combined YOLOv2 and bidirectional RNNs. Laroca et al. [Reference Laroca, Severo, Zanlorensi, Oliveira, Gonçalves, Schwartz and Menotti12] also used YOLOv2 to detect LPs and CR-NET [Reference Montazzolli and Jung21] to segment and recognize LP characters. Wang et al. [Reference Wang, Bian, Zhou and Chau22] proposed two cascaded CNNs: VertexNet for LPD and SCR-Net for LPR, with the main focus of achieving high inference speed.

2.2. LPR systems

As our main contribution to this work is a novel LPR engine, recent approaches related to this sub-task are presented in subsections 2.3 and 2.4.

Over the last decade, most of the LPR systems have relied essentially on deep learning (DL). Numerous DL-based solutions continually push the boundaries and achieve new state-of-the-art results across many standards [Reference Shashirangana, Padmasiri, Meedeniya and Perera23]. Considering the perspective from which LP characters can be seen – namely as objects or texts – we classify recent solutions into two categories: object recognition and sequence labeling.

2.3. Object recognition

Recent works have approached LPR by framing it as an object recognition problem, with each character being processed as a separate object. For instance, Rizvi et al. [Reference Rizvi, Patti, Björklund, Cabodi and Francini24] presented a 12-layer fully CNN with two branches (detector and localizer). Using shared connections of convolutional layers, the detector predicts character class, while the localizer outputs bounding box coordinates. Montazzolli et al. [Reference Montazzolli and Jung21] presented LPS/CR-NET, which is a FAST-YOLO based detector. First, minor changes were introduced to the original architecture to make it suitable for Brazilian LP layouts. They then followed their module predictions using two heuristic rules to eliminate confusion between letters and digits. Their architecture was later used by Silva and al. [Reference Silva, Jung, Ferrari, Hebert, Sminchisescu and Weiss7] and Laroca et al. [Reference Laroca, Zanlorensi, Gonçalves, Todt, Schwartz and Menotti25] to recognize distorted LP in unconstrained scenarios. Relying on YOLOv2, Kessentini et al. [Reference Kessentini, Besbes, Ammar and Chabbouh20] adjusted the original architecture to fit the Tunisian LP specifications. Similarly, Henry et al. [Reference Henry, Ahn and Lee26] used an improved version of the YOLOv3 detector that integrates a spatial pyramid pooling (SPP) block [Reference He, Zhang, Ren and Sun27], called YOLOv3-SPP. Their contribution perfectly fits the double-line plates scenario across many countries. Selmi et al. [Reference Selmi, Halima, Pal and Alimi28] adopted Mask-RCNN to segment and extract character candidates. To improve detection, they applied a set of rules to remove non-character regions.

The key benefit of these solutions is their ability to provide a rapid response time while maintaining a satisfactory rate of prediction. However, to train these models, an enormous amount of training data with costly character-level annotation is required. In addition, these solutions have been proven to be ineffective for distorted and tilted LPs.

2.4. Sequence labeling

Researchers have also addressed LPR as a standard character sequence labeling task. In earlier research works [Reference Kessentini, Besbes, Ammar and Chabbouh20,Reference Cao, Fu and Ma29,Reference Wang, Huang, Qian, Cao and Dai30], a scheme of convolution layers combined with recurrent ones, followed by CTC layer [Reference Graves, Liwicki, Fernández, Bertolami, Bunke and Schmidhuber31], was adopted. Specifically, LP sequence features are encoded through CNNs and RNNs, and decoded by CTC. Bidirectional long short-term memory (Bi-LSTM) networks [Reference Hochreiter and Schmidhuber32] were commonly used. For instance, Li et al. [Reference Li, Wang and Shen18] proposed a solution that can handle oblique and normal LPs. However, integrating the CTC layer in their model for sequence decoding yielded a weak performance. Wang et al. [Reference Wang, Huang, Qian, Cao and Dai30] proposed a DL architecture similar to the regular CNN-RNN-CTC stack, but before the CNN, they fed the detected LPs to spatial transformer networks [Reference Jaderberg, Simonyan, Zisserman, Kavukcuoglu, Cortes, Lawrence, Lee, Sugiyama and Garnett33] to adjust it (i.e. to obtain aligned characters of uniform heights and widths). Lately, the CTC layer has been replaced by an attention mechanism. For instance, Xu et al. [Reference Xu, Yang, Meng, Lu, Huang, Ying, Huang, Ferrari, Hebert, Sminchisescu and Weiss34] proposed a model, called RPnet, that uses the attention mechanism in the detection module to direct the recognition module to a specific area from which it should gather features that are useful for predicting the LP number. Most recently, a scheme of encoder-decoder was highly adopted. Precisely, the encoder is a CNN structure, while the decoder is an attention-based RNN. For example, He et al. [Reference He and Hao35] and Zou et al. [63] used a 1D-attention decoder to only focus on useful character features. Whereas these methods perform well in single-line LPs, they can not deal with double-line LPs. To cope with these limitations, 2D attention was applied instead. Both Zhang et al. [Reference Zhang, Wang, Li, Li, Shen and Zhang10] and Xu et al. [Reference Xu, Zhou, Li, Liu, Li and Shi36] integrated a 2D-attention mechanism to perform decoding and retrieve characters on the 2D feature maps. This technique yielded promising results on multi-line LPs, and unaligned LPs as well. Kumar et al. [Reference Kumar, Shivakumara, Chowdhury, Pal and Liu37] replaced the 2D attention with a transformer-based decoder to generate feature maps using an eight-head attention mechanism. They showed that their module had faster training and inference compared to RNN, but it relied on CNN for feature extraction.

3.1. YOLOv7x

YOLOv7, one of the official releases in the YOLO series of object detection models [Reference Wang, Bochkovskiy and Liao38], is known for its efficiency-speed trade-off, making it popular for applications where real-time detection is crucial, such as in video surveillance, particularly LPD. At the time of its publication, YOLOv7 surpassed all known object detectors in speed and accuracy and was able to reach up to 160 FPS (GPU V100). The major upgrades in YOLOv7 are mainly:

• • Extended efficient layer aggregation network (E-ELAN) [Reference Wang, Liao and Yeh39]: E-ELAN combines the features of different groups by shuffling and merging cardinality to enhance the network’s learning. It allows for feature extraction, allowing for better gradient flow, and feature reuse, improving performance without increasing complexity.

• • Model Scaling: YOLOv7 applies compound model scaling techniques, balancing depth, width, and resolution to improve performance across various computing resources and image sizes.

• • Bag-of-freebies: YOLOv7 uses RepConv [Reference Ding, Zhang, Ma, Han, Ding and Sun40], which allows it to have more efficient inference by converting multi-branch convolutions into a single branch during inference, boosting speed without losing in terms of accuracy.

In our proposed ALPR system, we use the “extend” version YOLOv7x. This variant uses the compound scaling method to perform scaling-up of the depth of computational block by 1.5 times and width of transition block by 1.25 times [Reference Wang, Bochkovskiy and Liao38].

3.2. ViTLPR

ViTLPR is a recurrence- and convolution-free model based on the self-attention mechanism. It leverages advancements in using attention mechanisms for various vision tasks. It is an extension of vision transformer that was originally introduced for image classification [Reference Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit and Houlsby15]. It approaches the LPR task as a sequence labeling problem for multiple reasons: internal variability of LPs (e.g. font style, orientation, shape, size, color, texture, illumination) and external influences (e.g. camera sensor orientation, location, and imperfections resulting in blur, noise, or distortions). The ViTLPR architecture is depicted in Fig. 3.

Figure 3.

Vision transformer-based LP recognizer architecture. Raw license plate images are partitioned into square patches and transformed into a sequence of vectors. After adding positional information, vectors are passed through a stacking of $L$ Vanilla transformer encoders. Finally, feature sequence is fed to the prediction head for character recognition.

The ViTLPR workflow is detailed in what follows. The LP image $x\in \mathbb{R}^{H \times W \times 3}$ ( $H$ and $W$ denote the image height and width, respectively) is partitioned into patches $x_{p}\in \mathbb{R}^{N\times (P \times P\times 3)}$ , where $N=\frac{H \times W}{P \times P}$ denotes the effective input sequence length for the encoder block. $P \times P$ denotes the resolution of each patch. Each patch is unrolled into a vector, then linearly projected to $d_{model}$ dimension. The projection is performed using a matrix $E$ . Positional embeddings are added to patch embeddings. $L$ ViT encoder blocks process the patch embeddings and produce, using a prediction head, the LP transcription. In what follows, we describe the three building blocks of ViTLPR.

1. 1. Linear embedding: First, LP image is divided into $P \times P \times 3$ patches. Each patch is reshaped to $(P \times P \times 3)\times 1$ vector. A dense layer is applied to each vector $z_{i}=Ex_{i}+b$ , where $E$ and $b$ are parameters to be learned during the training phase. Vectors $z_{i\in [1 \dots N]}$ are patch embeddings linearly projected into a new space with a dimension $d_{model}$ . To add positional information, learnable 1D position embeddings are accordingly associated with patch embeddings.

2. 2. Encoder: Each layer of the encoder block uses the multi-head self-attention (MHSA) and 2-layer perceptron (MLP) techniques. The skip connections are used in between to maintain lower-level features. In this work, the embedding sequence $Z \in \mathbb{R}^{(N+1) \times d_{model}}$ is linearly mapped into a new representation subspace using three trainable parameter matrices: $W^{Q}\in \mathbb{R}^{d_{model} \times d_{q}}$ , $W^{K}\in \mathbb{R}^{d_{model} \times d_{k}}$ , and $W^{V}\in \mathbb{R}^{d_{model} \times d_{v}}$ . This projection produces a triplet: query ( $Q$ ), key ( $K$ ), and value ( $V$ ), where $Q=Z \times W^{Q}$ , $K=Z \times W^{K}$ , and $V=Z \times W^{V}$ . The attention operation is defined by Equation 1.

   (1) \begin{equation} \text{Z}=\text{Attention(Q, K, V)}=\text{Softmax}\!\left(\frac{Q\cdot K^{T}}{\sqrt{d_{model}}}\right)\cdot \text{V} \end{equation}

   The output of the Softmax function is called an attention filter. To improve the performance of the Vanilla self-attention layer, the idea is to perform in parallel $h$ self-attention operations independently [Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin, Guyon, von Luxburg, Bengio, Wallach, Fergus, Vishwanathan and Garnett41]. The single-head self-attentions have the same inputs. However, they do not share parameter matrices. For each head, different $Q$ , $K$ , and $V$ matrices are learned. The MHSA operation is defined by Equation 2.

   (2) \begin{equation} \text{MultiHead}(Q', K', V') = \text{Concat}(\text{head}_{1}, \ldots , \text{head}_{h}) \cdot W^{o} \end{equation}
   where:
   (3) \begin{align} \text{head}_{i} = \text{Attention}(Q_{i}, K_{i}, V_{i}). \end{align}

   The MHSA outputs are fed into a 2-layer perceptron. It has the form GELU $((W_{1}Z + b_{1})W_{2}+ b_{2})$ , where $W_{1}\in \mathbb{R}^{d_{model} \times d_{latent}}$ and $W_{2}\in \mathbb{R}^{d_{latent} \times d_{model}}$ are two parameter matrices shared across the sequence $z_{i\in [0..N]}$ . The encoder block is repeated $L$ times to output “feature sequence” of shape ( $N \times d_{model}$ ). Each position ( $1, \ldots , N$ ) is a $d_{model}$ -dimensional vector that carries rich, context-aware representations of its corresponding LP patch.

3. 3. Prediction head: The “feature sequence” is transformed into class scores in order to produce a sequence of characters. First, we slice the ( $N \times d_{model}$ ) matrix to obtain ( $M \times d_{model}$ ) matrix, where $M$ is the maximum number of characters in the LP (variable). We add two additional rows (embeddings) for the start ( $start$ ) and end ( $e$ ) tokens and learn them during training. Then, we apply a linear layer to transform it to ( $M+2$ , $C=36$ alphanumeric characters) sequence. This layer will map each of the $d_{model}$ -dimensional embeddings to a $C$ -dimensional vector and predict the character sequence.

4.1. Datasets

We validate the robustness of the proposed system by showing experimental results on the PGTLP-v2 dataset and five benchmark ALPR datasets: PGTLP-v2, LSV-LP [Reference Wang, Lu, Zhang, Yuan and Li11], RodoSol-ALPR [Reference Laroca, Cardoso, Lucio, Estevam, Menotti, Farinella, Radeva and Bouatouch42], UFPR-ALPR [Reference Laroca, Severo, Zanlorensi, Oliveira, Gonçalves, Schwartz and Menotti12], CCPD [Reference Xu, Yang, Meng, Lu, Huang, Ying, Huang, Ferrari, Hebert, Sminchisescu and Weiss34], and AOLP [Reference Hsu, Chen and Chung43]. Image samples from the six benchmark datasets and their respective plates extracted based on the provided ground truth annotations are shown in Fig. 5. The specifications of the six datasets used in our experiments are listed in Table I.

Table I.

Specifications of the six benchmarks used in our experiments to evaluate the performance of the proposed automatic license plate recognition (ALPR) system.

$^\ast$ 170,999 images do not contain LPs. $^{\ast \ast }$ 47,055 LPs without transcription.

PGTLP-v2 is a Tunisian image dataset that counts $5$ k high-resolution images captured by cameras mounted on PGuard. Images were extracted from videos after patrolling parking, entrance gates, roundabouts, and driveways. This dataset provides large and diverse set of variations in templates, resolutions, and weather conditions. One particular feature of this dataset compared to existing datasets is that it contains several LPs per image. This dataset is an extension to previous version of PGTLP dataset that was initially introduced in [Reference Ismail, Mehri, Sahbani and Amara16] for LPD [Reference Ismail, Mehri, Sahbani and Amara44]. PGTLP-v2 includes three annotation levels: rectangular boxes for the LP, four vertices of the LP region, and labels for the full LP sequence (i.e. transcription). Figure 4 depicts few samples in the PGTLP-v2 dataset.

LSV-LP is a Chinese large-scale video-based dataset recorded using driving recorders, street camera shooting, and mobile phone shooting. The shooting locations include highways, streets, parking lots, and other scenes, covering $27$ provinces of China mainland. The LSV-LP dataset contains $1,402$ video clips of $300$ frames resulting in more than $400$ k frames and $364$ k LPs. The annotations include the bounding box around the vehicles, four vertices around the LP, and LP transcription [Reference Wang, Lu, Zhang, Yuan and Li11].

RodoSol-ALPR is a Brazilian image dataset collected by static cameras placed at pay tolls where the distance from the vehicle to the camera varies slightly. It contains $20$ k images that have a resolution of $1,280 \times 720$ pixels and are divided as follows: $5$ k images of cars with Brazilian LPs, $5$ k images of motorcycles with Brazilian LPs, $5$ k images of cars with Mercosur LPs, and $5$ k images of motorcycles with Mercosur LPs. The annotations cover the LP location and its related transcription [Reference Laroca, Cardoso, Lucio, Estevam, Menotti, Farinella, Radeva and Bouatouch42].

UFPR-ALPR includes $4,500$ images of $150$ Brazilian vehicles. It is divided into three subsets: training, validation, and test, where each subset contains $60$ , $30$ , and $60$ tracks, respectively. Each track has $30$ frames capturing the same vehicle. The UFPR-ALPR images have the following annotations: LP positions, LP numbers, and positions of its characters [Reference Laroca, Severo, Zanlorensi, Oliveira, Gonçalves, Schwartz and Menotti12].

CCPD (v2019) is a large-scale Chinese LP dataset that was collected from roadside parking using handheld devices. Hence, it exhibits strong variations in vehicle distance, shooting angle, light condition, and image background. Images in CCPD have at most one annotated LP in the foreground. We used the latest available version released in 2019. The entire dataset counts $355,003$ images. Annotations in CCPD are mainly LP numbers, LP bounding boxes, four vertices locations, and tilt degrees [Reference Xu, Yang, Meng, Lu, Huang, Ying, Huang, Ferrari, Hebert, Sminchisescu and Weiss34].

AOLP is a widely used benchmark for evaluating LPR approaches. This dataset contains $2,049$ images of Taiwanese LPs. It is divided into three subsets: access control (AC), law enforcement (LE), and road patrol (RP). The subsets contains $681$ , $757$ , and $611$ images, respectively. The RP subset is considered the most challenging since it has many samples with oblique LPs. Ground-truth annotations contain coordinates of LP bounding boxes and their sequence labels [Reference Hsu, Chen and Chung43].

Figure 4.

Image samples in PGTLP-v2. From top to down, the first row presents images ( $1,920 \times 1,080$ ) at entrance checkpoint. The second row contains $180^{\circ }$ panoramic view images ( $2,560 \times 1,024$ ) at restricted access plant.

Figure 5.

Image samples from the six datasets used in our experiments and their respective license plates with respect to the ground-truth annotations.

4.2. Evaluation protocol

All the experiments were conducted on a desktop with Intel^® Xeon^® CPU E5-2660 v3@2.60 GHz (20cores), 32 GB DDR4 RAM, and one Nvidia^® TITAN^® RTX^TM GPU with 24 GB GDDR6 RAM. Table II presents the number of images and LPs used for training, testing, and validation in each dataset used in our experiments.

Table II.

An overview of the number of images used for training, testing, and validation in each dataset.

To evaluate the performance of the stage of detecting LPs, we computed precision and recall metrics. The detected LP is considered correct if the intersections over union (IoU) between the detection and ground truth is greater than $0.5$ ( $0.7$ for CCPD dataset [Reference Xu, Yang, Meng, Lu, Huang, Ying, Huang, Ferrari, Hebert, Sminchisescu and Weiss34]). To evaluate the performance of the stage of recognizing LPs, we calculate the LPR rate (LP-RR) [Reference Kessentini, Besbes, Ammar and Chabbouh20]. LP-RR represents the amount of correctly recognized LPs over all LPs in the dataset (see Equation 4). Prediction is considered correct if and only if all LP characters are correctly recognized.

(4) \begin{equation} \text{LP-RR} = \left (\frac{\text{# of correctly recognized LPs}}{\text{Total # of LPs}}\right ) \times 100 \end{equation}

To evaluate the performance of the stage of recognizing LPs on the LSV-LP dataset, we computed two metrics: Accuracy_6C and Accuracy_7C, as defined in the original protocol by Wang et al. [Reference Wang, Lu, Zhang, Yuan and Li11]. Accuracy_6C measures the capability of the model to correctly recognize the 6 last characters (i.e. we exclude the first character representing the Chinese code region). Accuracy_7C measures the model’s ability to correctly recognize all seven characters of an LP (i.e. we include the first character representing the Chinese code region).

In our ALPR system, we use ViTLPR, which has similar properties to the base version of the vision transformer (ViT-Base [Reference Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit and Houlsby15]). Table III lists the architectural parameters of ViTLPR. Parameter $M$ is selected based on the number of LP characters in each dataset.

Table III.

Architectural parameters of vision transformer-based LP recognizer.

The training setup of the two stages of the proposed ALPR system is presented in Table IV. To train the LPD model YOLOv7x, we used the following setup (epochs = 40, batch size = 16, optimizer = SGD, initial learning rate = 0.01). Since ViT-based models require large data volumes to perform well, we initialized the ViTLPR encoder with weights provided by Touvron et al. [Reference Touvron, Cord, Douze, Massa, Sablayrolles, Jegou, Meila and Zhang45] before being fine-tuned on downstream datasets. We applied a few data augmentation techniques [Reference Cubuk, Zoph, Shlens and Le46] to raw images. The selected augmentation functions are projective distortion, blurring, noise, and rotation. To train ViTLPR, we only used LP regions as input and their character sequences as labels.

Table IV.

Selected hyperparameters for training the two modules of the proposed automatic license plate recognition system (YOLOv7x and vision transformer-based LP recognizer (ViTLPR)).