Hostname: page-component-848d4c4894-cjp7w Total loading time: 0 Render date: 2024-06-17T00:11:35.787Z Has data issue: false hasContentIssue false
  • English
  • Français

SLAMB&MAI: a comprehensive methodology for SLAM benchmark and map accuracy improvement

Published online by Cambridge University Press:  30 January 2024

Shengshu Liu Open the ORCID record for Shengshu Liu [Opens in a new window] ,
Erhui Sun  and
Xin Dong
Shengshu Liu
Affiliation:
Department of Mechanical, Materials and Manufacturing Engineering, University of Nottingham, Nottingham NG7 2RD, UK
Erhui Sun
Affiliation:
Department of Mechanical, Materials and Manufacturing Engineering, University of Nottingham, Nottingham NG7 2RD, UK
Xin Dong*
Affiliation:
Department of Mechanical, Materials and Manufacturing Engineering, University of Nottingham, Nottingham NG7 2RD, UK
*
Corresponding author: Xin Dong; Email: xin.dong@nottingham.ac.uk
Get access Rights & Permissions [Opens in a new window]

Abstract

SLAM Benchmark plays a pivotal role in the field by providing a common ground for performance evaluation. In this paper, a novel methodology of simultaneous localization and mapping benchmark and map accuracy improvement (SLAMB&MAI) is introduced. It can objectively evaluate errors of localization and mapping, and further improve map accuracy by utilizing evaluation results as feedback. The proposed benchmark transforms all elements into a global frame and measures the errors between them. The comprehensiveness consists in the benchmark of both localization and mapping, and the objectivity consists in the consideration of the correlation between localization and mapping by the preservation of the original pose relations between all reference frames. The map accuracy improvement is realized by first obtaining the optimization that minimizes the errors between the estimated trajectory and ground truth trajectory and then applying it to the estimated map. The experimental results showed that the map accuracy can be improved by an average of 15%. The optimization that yields minimal localization errors is obtained by the proposed Centre Point Registration-Iterative Closest Point (CPR-ICP). This proposed Iterative Closest Point (ICP) variant pre-aligns two point clouds by their centroids and least square planes and then uses traditional ICP to minimize the error between them. The experimental results showed that CPR-ICP outperformed traditional ICP, especially in cases involving large-scale environments. To the extent of our knowledge, this is the first work that can not only objectively benchmark both localization and mapping but also revise the estimated map and increase its accuracy, which provides insights into the acquisition of ground truth map and robot navigation.

Keywords

comprehensive and objective SLAM benchmarkmap ground truthmap error correctionmap accuracy improvementground truth acquisition
Type
Research Article
Information
Robotica , Volume 42 , Issue 4 , April 2024 , pp. 1039 - 1054
Copyright
© The Author(s), 2024. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Sturm, J., Engelhard, N., Endres, F., Burgard, W. and Cremers, D., “A Benchmark for the Evaluation of RGB-D SLAM Systems,” In: 2012 EEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura-Algarve (IEEE, 2012) pp. 573–580.10.1109/IROS.2012.6385773CrossRefGoogle Scholar
Geiger, A., Lenz, P., Stiller, C. and Urtasun, R., “Vision meets robotics: The kitti dataset,” Int. J. Rob. Res. 32(11), 1231–1237 (2013).10.1177/0278364913491297CrossRefGoogle Scholar
Burri, M., Nikolic, J., Gohl, P., Schneider, T., Rehder, J., Omari, S., Achtelik, M. W. and Siegwart, R., “The EuRoC micro aerial vehicle datasets,” Int J Rob Res 35(10), 1157–1163 (2016).10.1177/0278364915620033CrossRefGoogle Scholar
Wulf, O., Nuchter, A., Hertzberg, J. and Wagner, B., “Ground Truth Evaluation of Large Urban 6D SLAM,” In: 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, San Diego, CA (IEEE, 2007) pp. 650–657.10.1109/IROS.2007.4399026CrossRefGoogle Scholar
Ceriani, S., Fontana, G., Giusti, A., Marzorati, D., Matteucci, M., Migliore, D., Rizzi, D., Sorrenti, D. G. and Taddei, P., “Rawseeds ground truth collection systems for indoor self-localization and mapping,” Auton Robot 27(4), 353–371 (2009).CrossRefGoogle Scholar
Havangi, R., “Robust square-root cubature fastSLAM with genetic operators,” Robotica 39(4), 665–685 (2021).10.1017/S026357472000065XCrossRefGoogle Scholar
Geiger, A., Lenz, P. and Urtasun, R., “Are We Ready for Autonomous Driving? The Kitti Vision Benchmark Suite,” In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI (IEEE, 2012) pp. 3354–3361.10.1109/CVPR.2012.6248074CrossRefGoogle Scholar
Wasenmüller, O., Meyer, M. and Stricker, D., “CoRBS: Comprehensive RGB-D Benchmark for SLAM Using Kinect v2,” In: 2016 EEE Winter Conference on Applications of Computer Vision (WACV), Lake Placid, NY (IEEE, 2016) pp. 1–7.CrossRefGoogle Scholar
Handa, A., Newcombe, R. A., Angeli, A. and Davison, A. J., “Real-Time Camera Tracking: When is High Frame-Rate Best?,” In: European Conference on Computer Vision, (Springer, Berlin, Heidelberg, 2012) pp. 225–235.Google Scholar
Funke, J. and Pietzsch, T., “A Framework For Evaluating Visual SLAM,” In: Proceedings of the British Machine Vision Conference (BMVC) (2009).10.5244/C.23.69CrossRefGoogle Scholar
Handa, A., Whelan, T., Donald, M. and Davison, J., “A Benchmark for RGB-D Visual Odometry, 3D Reconstruction and SLAM,” In: 2014 IEEE International Conference on Robotics and Automation (ICRA), Hong Kong, China (IEEE, 2014) pp. 1524–1531.10.1109/ICRA.2014.6907054CrossRefGoogle Scholar
Nardi, L., Bodin, B., Zeeshan Zia, M., Mawer, J., Nisbet, A., Kelly, P. H. J., Davison, A. J., Luján, M., O’Boyle, M. F. P., Riley, G., Topham, N. and Furber, S., “Introducing SLAMBench, a Performance and Accuracy Benchmarking Methodology for SLAM,” In: 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, WA (IEEE, 2015) pp. 5783–5790.10.1109/ICRA.2015.7140009CrossRefGoogle Scholar
Klette, R., Koscham, A., Schlüns, K.and Rodehorst, V., “Evaluation of Surface Reconstruction Methods,” In: New Zealand Image and Vision Computing Workshop, (Citeseer, New Jersey, 1995) pp. 3–12.Google Scholar
Eid, A. and Farag, A., “A Unified Framework for Performance Evaluation of 3-D Reconstruction Techniques.” In: 2004 Conference on Computer Vision and Pattern Recognition Workshop, Washington, DC (IEEE, 2004), p. 33.Google Scholar
Seitz, S. M., Curless, B., Diebel, J., Scharstein, D. and Szeliski, R., “A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms,” In: 2006 IEEE computer society conference on computer vision and pattern recognition (CVPR’06), New York, NY (IEEE, 2006) pp. 519–528.Google Scholar
Bellmann, A., Hellwich, O., Rodehorst, V. and Yilmaz, U., “A Benchmarking Dataset for Performance Evaluation of Automatic Surface Reconstruction Algorithms,” In: 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN (IEEE, 2007) pp. 1–8.CrossRefGoogle Scholar
Krolla, B. and Stricker, D., “Heterogeneous Dataset Acquisition for a Continuously Expandable Benchmark (CEB),” In: International Conference on Computer Graphics, Visualization and Computer Vision (WSCG), Pilzen, Czech Republic (2015).Google Scholar
Chen, Y. and Medioni, G., “Object Modeling by Registration of Multiple Range Images,” In: Proceedings. 1991 IEEE International Conference on Robotics and Automation, Sacramento, CA (1991) pp. 2724–2729.Google Scholar
Besl, P. J. and McKay, N. D., “Method for Registration of 3-D Shapes,” In: Proceedings SPIE 1611, Sensor Fusion IV: Control Paradigms and Data Structures (1992).Google Scholar
Li, X. and Guskov, I., “Multi-Scale Features for Approximate Alignment of Point-based Surfaces,” In: Proceedings of the Third Eurographics Symposium on Geometry Processing, Vienna, Austria (Citeseer, 2005) pp. 217–226.Google Scholar
Aiger, D., Mitra, N. J. and Cohen-Or, D., “4-Points Congruent Sets for Robust Pairwise Surface Registration,” In: ACM SIGGRAPH 2008 Papers, Los Angeles, California (2008) pp. 1–10.Google Scholar
Fischler, M. A. and Bolles, R. C., “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM 24(6), 381–395 (1981).CrossRefGoogle Scholar
Rusinkiewicz, S. and Levoy, M., “Efficient Variants of the ICP Algorithm,” In: Proceedings Third International Conference on 3-D Digital Imaging and Modeling, Quebec City, QC (IEEE, 2001) pp. 145–152.Google Scholar
Low, K.-L.. Linear Least-Squares Optimization for Point-to-Plane Icp Surface Registration, (University of North Carolina, Chapel Hill, 2004) pp. 1–3.Google Scholar
Censi, A., “An ICP Variant Using a Point-to-Line Metric,” In: 2008 IEEE International Conference on Robotics and Automation, Pasadena, CA (IEEE, 2008) pp. 19–25.10.1109/ROBOT.2008.4543181CrossRefGoogle Scholar
Cignoni, P., Rocchini, C. and Scopigno, R., “Metro: Measuring error on simplified surfaces,” Comp Graph For 17(2), 167–174 (1998).Google Scholar
Aspert, N., Santa-Cruz, D. and Ebrahimi, T., “Mesh: Measuring Errors Between Surfaces Using the Hausdorff Distance,” In: Proceedings. IEEE International Conference on Multimedia and Expo, Lausanne, Switzerland (IEEE, 2002) pp. 705–708.Google Scholar
Girardeau-Montaut, D., “Détection De Changement Sur Des Données géométriques Tridimensionnelles,” Ph.D. Dissertation (Télécom ParisTech, Paris, France, 2006).Google Scholar
Girardeau-Montaut, D.. CloudCompare (EDF R&D Telecom ParisTech, France, 2016) pp. 11.Google Scholar
Meister, S., Kohli, P., Izadi, S., Hämmerle, M., Rother, C. and Kondermann, D., “When Can We Use Kinectfusion for Ground Truth Acquisition,” In: Proceedings Workshop on Color-Depth Camera Fusion in Robotics, Vilamoura, Portugal (IEEE, 2012).Google Scholar
Arun, K. S., Huang, T. S. and Blostein, S. D., “Least-squares fitting of two 3-D point sets,” IEEE Trans. Pattern Anal 9(5), pp. 698–700 (1987).10.1109/TPAMI.1987.4767965CrossRefGoogle ScholarPubMed
Chen, Y. and Medioni, G., “Object modelling by registration of multiple range images,” Image Vision Comput 10(3), 145–155 (1992).CrossRefGoogle Scholar
Zhang, Z., “Iterative point matching for registration of free-form curves and surfaces,” Int J Comput Vision 13(2), 119–152 (1994).CrossRefGoogle Scholar
Stewart, G. W., “On the early history of the singular value decomposition,” SIAM Rev 35(4), 551–566 (1993).10.1137/1035134CrossRefGoogle Scholar
Curless, B. and Levoy, M., “A Volumetric Method for Building Complex Models from Range Images,” In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, New York, NY, (1996) pp. 303–312.Google Scholar
Krishnamurthy, V. and Levoy, M., “Fitting Smooth Surfaces to Dense Polygon Meshes,” In: Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, New York, NY (1996) pp. 313–324.Google Scholar
Turk, G. and Levoy, M., “Zippered Polygon Meshes from Range Images,” In: Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques, New Yoirk, NY (1994) pp. 311–318.Google Scholar
Mur-Artal, R. and Tardós, J. D., “ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras,” IEEE Trans. Robot. 33(5), 1255–1262 (2017).10.1109/TRO.2017.2705103CrossRefGoogle Scholar