2.2.1. Jetson nano main controller

As shown in Figure 3, the Jetson Nano serves as the primary processing unit and ROS controller, handling high-level processing and system communication. It interfaces with sensors, actuators, and peripherals, processing data for real-time decision-making. For instance, when the LiDAR detects an obstruction, the Jetson Nano processes the input and sends movement commands to the STM32F407 microcontroller for obstacle avoidance. With its GPU-accelerated computing capabilities and support for deep learning frameworks, the Jetson Nano enables real-time perception and control while operating within a low-power envelope, making it suitable for autonomous robotic applications in resource-constrained environments.

Figure 3. Hardware integration of the robot platform, detailing physical connections and data communication pathways between components such as sensors, actuators, controllers, and power systems to illustrate their collaborative functionality.

The Jetson Nano’s capacity to process multiple neural networks simultaneously makes it possible to implement sophisticated deep learning tasks such as scene comprehension, object recognition, and semantic segmentation, necessary for effective damage detection. Since these abilities are essential for autonomous navigation and task completion, the Jetson Nano plays a key role in the robot’s capacity to engage with and adjust to its complex surroundings, typical for civil infrastructure embedded within the existing natural environment. Its assistance with GPU acceleration also improves the performance of the AI model for swift decision-making and stable system operation. Its assistance with GPU acceleration allows for real-time inference of AI models, ensuring that the robot can make low-latency decisions when detecting obstacles, identifying damage, or adjusting its navigation path in challenging environments.

2.2.2. STM32 robot controller

The robot controller’s central component, the STM32F407VET6 microcontroller, runs at a clock speed of 168 MHz and includes onboard flash memory and SRAM for real-time data processing. It manages low-level motion control and sensor integration, ensuring precise movement and stable navigation. It operates multiple DC encoder motors for chassis movement and a servo motor that adjusts the camera’s position, enabling dynamic viewpoint adjustments. Additionally, it processes data from the integrated IMU, which includes an accelerometer, providing essential feedback for motion stabilization and localization. This integration allows the STM32 to execute movement commands received from the Jetson Nano while also processing sensor feedback, ensuring accurate control and real-time adjustments in challenging conditions.

2.2.3. Depth camera

The Astra Pro Plus depth camera on the robot captures both depth and RGB data using 3D structured light technology. It consists of four main components: a depth processing unit, an IR projector, an infrared (IR) camera, and an RGB module. The IR projector emits structured light patterns onto a target scene, which the IR camera captures as infrared images. The depth processing unit then determines depth values by analyzing these images against a pre-calibrated reference pattern using structured light triangulation. The RGB module captures color images that complement the depth data.

The combined depth and RGB information is crucial for crack detection and localization. When the robot identifies a crack, it measures the distance to the object and logs the crack’s location. The location data are especially helpful in preventing duplicate recordings of the same crack, ensuring efficient and precise documentation of structural damage.

2.2.4. Light detection and ranging (LiDAR)

LiDAR is a sensor technology that measures distances with extreme precision using laser pulses. The time it takes for a sequence of laser beams to return after hitting an item is determined by the sensor. LiDAR uses these time-of-flight data to generate a point cloud, which is a very accurate depiction of the environment that provides precise distance data.

Features such as SLAM mapping, obstacle avoidance, and autonomous navigation are made possible by LiDAR sensor. The robot’s strong spatial awareness, which is essential real-time navigation tasks, is provided by the LiDAR system, which continuously scans its surroundings and locates anything. By continuously scanning its surroundings and updating its position, the LiDAR system supports autonomous real-time path planning, precise localization, and adaptive obstacle avoidance (Gatesichapakorn et al., Reference Gatesichapakorn, Takamatsu and Ruchanurucks2019; Sun et al., Reference Sun, Zhao, Hu, Gao and Yu2023).

Before any movement, the robot first analyses the LiDAR data to assess its surroundings and make navigation decisions accordingly. Suppose that the LiDAR detects an obstruction, such as overgrown vegetation encroaching onto a pavement near a retaining wall or plant roots pushing through the tunnel floor. In that case, the robot can stop and adjust its movement to navigate around the blockage safely. LiDAR helps the robot distinguish between static and dynamic objects, ensuring safe movement around construction barriers or dense foliage (Bartlett et al., Reference Bartlett, Santos, Moreno, Dalai, Fahy, Dow, Gonzalez, Trslic, Riordan and Omerdic2024). These capabilities are crucial for infrastructure maintenance, where the robot must accurately perceive environmental changes and precisely localize its position while manoeuvring through confined spaces.

2.3.1. Main node

As shown in Figure 5, the main node acts as the robot’s core controller, directing and coordinating the actions of all other nodes to ensure accurate and effective inspection. It controls the robot’s movement, damage detection, and position adjustment in addition to initialising all subsystems.

The auxiliary nodes, including the camera, LiDAR, YOLO, and mapping nodes (see Figure 5), are provided by existing robot technology. These nodes handle specific tasks such as sensor data acquisition, object detection, and environment mapping. The main node starts by activating all the auxiliary nodes. Each subsystem is checked to ensure readiness before proceeding with the inspection workflow.

As can be seen in Figure 6, the initial scanning process involves the robot systematically evaluating potential damage by adjusting the camera position to each side with a 90° angle and analysing the results, as outlined in the following steps:

1. 1. To begin, the robot sets the camera position to the left and looks for damage in its range of vision.

2. 2. If no damage is detected, the camera turns to the right and scans again.

3. 3. If no cracks are found with a confidence score greater than the given threshold, the camera returns to the front position, and the robot starts navigating.

Figure 6. Detailed methodology of the main node in the autonomous inspection system, including object detection, position adjustment, damage recording, and navigation control.

As shown in Figure 6, during navigation, the robot continuously performs sweeping motions with the camera, analyzing its surroundings for damage. When damage is detected while navigating, the following sequence is activated:

1. 1. The robot calculates the angle of the crack’s center relative to its position.

2. 2. The camera turns to align with the center of the crack, adjusting its servo angle to point at the damage.

3. 3. The robot continues its forward motion, keeping the camera angle fixed on the crack.

These three steps repeat until the angle of camera exceeds 75° and the robot aligns with the crack. Then, the robot rotates the camera to 90° toward the crack and begins adjusting its position to capture detailed images (see Figure 7). Pose adjustment is a critical phase to ensure that the damage is properly framed for documentation. The robot follows these steps:

1. 1. The robot checks each edge of the bounding box of damage. If no edge is in the frame, take appropriate action:

   1. (a) Handling left or right edge out of frame: If the left or right edge of the bounding box is not visible, the robot adjusts its position by moving forward or backward. This is because the camera is oriented at an angle relative to the robot’s forward-facing direction, allowing lateral adjustments to bring the missing edge into the frame.

   2. (b) Handling both left and right edges out of frame: If both the left and right edges are out of frame, it indicates that the robot is too close to the wall. In this case, the robot moves away from the wall to ensure the entire bounding box is visible. However, if the robot cannot move further back and both edges remain out of frame, it retraces its path to locate the starting point of the crack. Once the starting point is identified, the robot begins recording and moves forward, capturing multiple images to document the entire extent of the damage.

   3. (c) Handling top or bottom edge out of frame: If the top or bottom edge of the bounding box is out of frame, the robot moves away from the wall to adjust its position and bring the missing edge into view. If further adjustment is not possible (e.g., due to physical constraints), the robot records the crack in its current state.

2. 2. If the robot is able to fit every edge in the frame, it generates a temporary object that records the damage’s image, bounding box, location, and area.

3. 3. While the robot is in motion, it maintains a buffer containing the three damage objects with the highest detection confidence. For every new frame of the same damage instance, the buffer entry is updated if the new observation has (i) a higher confidence score or (ii) a larger visible area. This allows the robot to refine each damage record before committing it to permanent memory. At the end of the process, the version with the highest confidence score is stored permanently, even if a previous frame offered a larger visible area.

4. 4. The robot uses a YOLO-based model to detect cracks, which assigns a confidence score to each detection. A minimum threshold of 0.5 is applied to filter out unreliable predictions. During operation, the robot continually evaluates successive frames of the same damaged object and retains the one with the highest confidence score. The process stops once no higher confidence score is detected after a given number of frames or distance traveled, indicating that the optimum observation has likely been reached.

Figure 7. Flowchart illustrating the adaptive pose adjustment process for crack detection. The robot iteratively refines its position based on bounding box proximity to frame edges and confidence scores, ensuring optimal crack visibility before capturing detailed images.

These procedures are repeated until the robot approaches the wall at a distance shorter than 15 cm, which is the minimum threshold required for the depth camera to accurately measure distance or until it is determined that the adjustments made do not improve the confidence rate. If any of these requirements are met, the system transfers the location, time, confidence score, bounding box, and damage snapshot from temporary to permanent memory. The robot then continues its navigation to identify and detect additional instances of damage.

As illustrated in the workflow in Figure 6 the inspection process terminates when the robot attains the predefined maximum inspection distance. Upon reaching this threshold, the system performs a final validation of temporary data registries to mitigate potential data loss caused by occlusions or dynamic prioritization of cracks during navigation. Cracks exceeding a confidence threshold, identified but not yet archived due to transient sensor limitations or algorithmic focus shifts, are systematically preserved to ensure comprehensive damage documentation. This protocol guarantees robust data integrity before concluding the inspection cycle.

2.3.2. Sensor nodes

The LiDAR node (Section 2.2.4) processes raw laser scan data and transforms it into a structured format that can be efficiently used by other nodes in the system. It filters, interprets, and organizes distance measurements to ensure the data is accurate and accessible for downstream tasks. As can be seen in Figure 1, the processed laser data are sent to the main node (Section 2.3.1) for obstacle avoidance. Meanwhile, the LiDAR node provides refined distance data to the mapping node (Figure 2.3.5), enabling Gmapping algorithm (Grisetti et al., Reference Grisetti, Stachniss and Burgard2007) to create detailed environmental maps.

The depth camera node (Section 2.2.3) is responsible for converting the sensors’ raw RGB and depth data into usable information before transmitting it to the main node (see 2.3.1). The RGB stream is directed to the YOLO node (Section 2.3.4), where it is used for damage detection (see Figure 8). The depth data are analyzed by the main node to provide accurate distance measurements to the detected damage.

Figure 8. Detailed architecture of the auxiliary nodes.

Other sensor nodes, such as the IMU node and odom node, work together to provide the robot with a comprehensive understanding of its position, orientation, and kinematic state. These nodes integrate data from inertial sensors, motor encoders, and localization systems to track the robot’s movement, publish its real-time state, and ensure accurate coordination of its components.

2.3.3. Position node

The position node uses odometry data to track the robot’s path and records its position and orientation. This information is published to the main node, enabling it to determine the spatial coordinates of detected cracks. By maintaining accurate positional data, the system ensures that cracks identified by the YOLO node are associated with their precise locations, facilitating efficient reporting and reconstruction of as-is digital models. Additionally, the system uses a temporary crack registry to track detected damage instances (see Section 2.4, preventing redundant inspections by recognizing previously recorded cracks.

2.3.4. Yolo node

The robot’s real-time object detection algorithm is based on the YOLOv5 model, which is used for rapid and accurate crack identification. This system operates with minimal latency, efficiently detecting potential cracks and generating bounding boxes around damage areas for further analysis (see Figure 9). To enhance the detection accuracy for different structural surfaces, the model was trained on two distinct datasets, each tailored for specific material types.

Figure 9. Overview of the YOLOv5-based crack detection pipeline with GPU-accelerated inference. The pipeline consists of three main stages: preprocessing, where images are resized, padded, and normalized; inference, performed on a GPU using a TRT-optimized YOLOv5 model; and post-processing, where the final detection results are extracted using non-maximum suppression (NMS).

The first dataset, Concrete Defects (Multiverse, 2023), consists of 1239 annotated images depicting various types of cracks on concrete surfaces. This dataset plays a crucial role in enabling the model to recognize crack patterns typically found in retaining walls and other concrete-based structures. The annotated images include defects such as surface fractures, deep fissures, and irregular cracking patterns, ensuring that the trained model can generalize well across different real-world conditions.

The second dataset, MCrack1300 (Ye et al., Reference Ye, Lovell, Faramarzi and Ninić2024), is specifically designed for masonry structures. It provides annotated images of cracks on brick and stone walls, addressing the unique texture and failure patterns present in these materials. Unlike concrete cracks, which often exhibit linear or web-like formations, masonry cracks can follow mortar joints, form stair-step patterns, or result from material separation. Incorporating this dataset into the training process significantly improves the model’s ability to distinguish actual structural damage from natural separations between bricks.

To enhance the model’s generalization across different structural materials, both datasets were merged and trained together as a single dataset. The MCrack1300 dataset contributed 1000 images for training, while the Concrete Defects dataset provided 1239 training images, resulting in a total of 2239 training images. The ratio of masonry to concrete images in the combined dataset is approximately 1:1.24, ensuring a balanced representation of both material types. Merging these datasets provides several advantages: (1) it allows the model to learn diverse crack formations, making it more robust when detecting cracks on different surfaces; (2) it improves the model’s adaptability to varying lighting conditions and textures; and (3) it reduces dataset bias by ensuring a more comprehensive range of crack appearances.

Finally, the model’s performance was evaluated on three separate datasets: Concrete Defects, MCrack1300, and the Merged Dataset, with results summarized in Table 1. The results indicate that the model achieved the highest performance when trained and tested on the Concrete Defects dataset, with a precision of 83.64% and a mean average precision at a 0.5 threshold (mAP_0.5) of 77.73%, suggesting that it effectively detects cracks in concrete surfaces. However, the performance on the MCrack1300 dataset was significantly lower, with a precision of 62.68% and an mAP_0.5 of 50.61%, highlighting the challenges of detecting cracks in masonry structures. This can be attributed to the model’s limited exposure to masonry-specific cracks during training and the inherent complexity of distinguishing cracks from mortar joints and natural separations in brick walls.

Table 1. Performance evaluation of the trained model on different datasets

Note: The table presents precision, recall, and mean average precision (mAP_0.5) for the model trained separately on the Concrete Defects dataset, the MCrack1300 dataset, and the merged dataset.

Training on the merged dataset provided a balanced improvement, with a precision of 73.59% and an mAP_0.5 of 67.08%, demonstrating enhanced generalization across both concrete and masonry surfaces. While the performance was slightly lower than the concrete-only model, it represents a more adaptable solution for real-world applications where both material types are present. These results emphasize the benefits of dataset diversification in improving crack detection across different structural conditions.

Upon activation of the main node, the YOLO node is initialized immediately and processes RGB images supplied by the camera node. To ensure efficient GPU utilization, these images are processed through an optimized workflow leveraging TRT (TensorRT) (NVIDIA Corporation, 2023) and pre-compiled engine files. TRT is a deep learning inference library developed by NVIDIA and optimized to run neural networks on GPUs. A serialized version of the TRT-generated optimized model is called an engine file. It contains every layer, weight, and configuration required to run the YOLO model effectively on the GPU. The engine file is customized to the particular hardware it is producing on in order to guarantee optimal performance during inference. This workflow enhances inference speed and accuracy, ensuring real-time performance suitable for demanding operational environments.

For robotic systems to identify damage in real-time, asynchronous processes are essential to the inference process. They make it possible for several tasks to be completed simultaneously without affecting the main thread, which is crucial for preserving system efficiency and responsiveness. In our case, these tasks include processing sensor input, running detection algorithms, and making decisions in real-time.

In YOLOv5, the confidence score for each detected object is computed automatically by the model as the product of two values: the objectness score and the class probability $ P\left(\mathrm{class}|\mathrm{object}\right) $ . The objectness score reflects the model’s confidence that a bounding box contains any object, while the class probability represents the likelihood that the object belongs to a specific class (e.g., a crack). During inference, YOLOv5 outputs this combined confidence score for each predicted bounding box. In the second inspection scenario, the increase in confidence scores suggests that pose adjustment led to better bounding box alignment and stronger classification certainty, resulting in more reliable crack detection.

The YOLO node follows a systematic workflow (see Figure 9) that includes pre-processing, inference, and post-processing to extract the optimal bounding boxes from the input RGB images:

1. 1. Preprocessing: In the preprocessing step for TRT, the raw image is prepared for loading onto the GPU to perform damage detection. As part of this process, the image is transformed into the NCHW format, which represents the following dimensions: batch size (N), number of channels (C), depth (D), height (H), and width (W). This format is widely used for storing multidimensional arrays, data frames, or matrices in memory, where the data can be efficiently represented as a one-dimensional array. The NCHW format is particularly advantageous for GPU-based computation because it aligns the data layout with the memory access patterns optimized for deep learning operations. The GPU can process the data more effectively by organizing it in this way, which guarantees quicker computation and lower latency during inference.

2. 2. Inference: As shown in Figure 8, the preprocessed image data is transferred from the host (CPU) to the GPU memory using an asynchronous operation that utilizes a stream for non-blocking data transfer. The used approach optimizes throughput by reducing idle time and enabling simultaneous operation of the CPU and GPU. The inference process is initiated by executing the TRT model on the GPU. Using the input bindings and stream, the GPU processes the data through the model’s computational layers to generate predictions.

3. 3. Post-processing: The raw predictions generated by the TRT model undergo a series of post-processing steps to refine the results and extract actionable data. The system first determines how many bounding boxes were found in the image overall. In order to make it easier to manipulate and analyze bounding box coordinates and confidence scores, the raw predictions are then transformed into a structured and easily interpretable format. Specifically, the TRT model outputs a tensor of shape $ \left(B,N,7\right) $ , where $ B $ represents the batch size, $ N $ is the number of predictions, and the seven values include the bounding box coordinates, confidence scores, and class probabilities. The confidence score, which indicates the likelihood that a bounding box contains a crack, is computed as the product of the objectivity score and the predicted class probability. The objectivity score is obtained by applying the sigmoid function to the raw objectivity outputs, ensuring the value falls between 0 and 1:

(2.1) $$ P\left(\mathrm{object}\right)=\sigma (s)=\frac{1}{1+{e}^{-s}} $$

Since the model is trained for a single-class detection task, the class probability is also derived using the sigmoid function:

(2.2) $$ p=\sigma (c)=\frac{1}{1+{e}^{-c}} $$

where $ c $ is the raw class score output by the model. The sigmoid function serves two critical roles: it normalizes unbounded raw scores into probabilistic values within [0,1], enabling meaningful confidence interpretation; and its non-linear compression amplifies score distinctions near decision boundaries, enhancing sensitivity to marginal detections. The final confidence score is then computed as:

(2.3) $$ C=P\left(\mathrm{object}\right)\times p $$

To facilitate further processing, the bounding box coordinates, originally in center-based format are converted to corner-based format, which is more commonly used for object detection tasks.

Then, only bounding boxes with the highest confidence ratings for each detected object are kept after redundant ones are removed using NMS. By successfully eliminating overlapping detections, this phase lowers the possibility of false positives and increases the detection process’s overall reliability. The improved bounding boxes are then extracted for additional usage, together with the class labels and confidence scores that go with them.

2.3.5. Mapping node

The Gmapping algorithm (Grisetti et al., Reference Grisetti, Stachniss and Burgard2007), based on the SLAM (Whyte, Reference Whyte2006) approach is used to create a map of the environment. Figure 10 illustrates the process flow of the algorithm, which uses LiDAR scans to gather data on the surroundings and integrates odometry data obtained from wheel encoders, inertial measurement units (IMUs), or other sensors, to enhance localization. These data are then processed to identify landmarks and features, which are then used to update both the map and the robot’s position. The algorithm typically includes two main processes: localization, which estimates the robot’s position and orientation, and mapping, which updates the map with new information.

Figure 10. Process flow of the Gmapping algorithm. The algorithm integrates LiDAR scans and odometry data to estimate the robot’s pose and update the map.

As shown in Figure 10, the localization loop in the Gmapping algorithm relies on a particle filter (Djuric et al., Reference Djuric, Kotecha, Zhang, Huang, Ghirmai, Bugallo and Miguez2003), which iteratively performs prediction, update, and resampling steps to estimate the robot’s pose. This process refines localization by integrating odometry and LiDAR data, ensuring that the most probable robot positions contribute to map updates, making Gmapping effective in environments with high uncertainty and noise, such as geotechnical sites surrounded by vegetation. This continuous process allows for real-time navigation and mapping.

The particle filter has the following steps:

1. 1. Initialization: A collection of particles, or samples, is created, each of which represents a possible system state, such as the orientation and location of the robot.

2. 2. Prediction: Each particle moves according to a motion model, which forecasts the next state based on the current state and control inputs, such as the movements of the robot.

3. 3. Update: The weights of the particles are updated using sensor measurements. Higher weights are given to particles that more closely match the sensor data.

4. 4. Resampling: Depending on their weights, particles undergo a resampling process. Higher-weighted particles have a greater chance of being chosen, but lower-weighted particles are likely to be eliminated. This stage guarantees that the most likely states receive the majority of the computing resources.

5. 5. Iteration: The process is iterated, with particles continuously predicted, updated, and resampled as new sensor data becomes available.

2.3.6. Rviz node

For tracking the robot’s activities, the RViz node offers a real-time visualization interface, as illustrated in Figure 11. It shows sensor data, including LiDAR scans, grid maps, and it visualizes the robot’s location and path through the surroundings. On the basis of real-time feedback, it also enables operators to step in or make modifications.

Figure 11. Visualization of the robotic inspection system in RViz, showcasing the robot model, sensor data, and navigation path.