Hostname: page-component-848d4c4894-hfldf Total loading time: 0 Render date: 2024-06-02T07:48:24.956Z Has data issue: false hasContentIssue false
  • English
  • Français

A flexible wearable e-skin sensing system for robotic teleoperation

Published online by Cambridge University Press:  16 September 2022

Chuanyu Zhong Open the ORCID record for Chuanyu Zhong [Opens in a new window] ,
Shumi Zhao ,
Yang Liu ,
Zhijun Li ,
Zhen Kan  and
Ying Feng
Chuanyu Zhong
Affiliation:
Department of Automation, University of Science and Technology of China, Hefei 230026, China
Shumi Zhao
Affiliation:
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China
Yang Liu
Affiliation:
Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China
Zhijun Li*
Affiliation:
Department of Automation, University of Science and Technology of China, Hefei 230026, China Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China
Zhen Kan
Affiliation:
Department of Automation, University of Science and Technology of China, Hefei 230026, China
Ying Feng
Affiliation:
College of Automation Science and Engineering, South China University of Technology, Guangzhou 510641, China
*
*Corresponding author. E-mail: zjli@ieee.org
Get access Rights & Permissions [Opens in a new window]

Abstract

Electronic skin (e-skin) is playing an increasingly important role in health detection, robotic teleoperation, and human-machine interaction, but most e-skins currently lack the integration of on-site signal acquisition and transmission modules. In this paper, we develop a novel flexible wearable e-skin sensing system with 11 sensing channels for robotic teleoperation. The designed sensing system is mainly composed of three components: e-skin sensor, customized flexible printed circuit (FPC), and human-machine interface. The e-skin sensor has 10 stretchable resistors distributed at the proximal and metacarpal joints of each finger respectively and 1 stretchable resistor distributed at the purlicue. The e-skin sensor can be attached to the opisthenar, and thanks to its stretchability, the sensor can detect the bent angle of the finger. The customized FPC, with WiFi module, wirelessly transmits the signal to the terminal device with human-machine interface, and we design a graphical user interface based on the Qt framework for real-time signal acquisition, storage, and display. Based on this developed e-skin system and self-developed robotic multi-fingered hand, we conduct gesture recognition and robotic multi-fingered teleoperation experiments using deep learning techniques and obtain a recognition accuracy of 91.22%. The results demonstrate that the developed e-skin sensing system has great potential in human-machine interaction.

Keywords

stretchable electronic skingesture recognitionhuman-machine interactionflexible printed circuitdeep learningrobotic hand teleoperation
Type
Research Article
Information
Robotica , Volume 41 , Issue 3 , March 2023 , pp. 1025 - 1038
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Tam, S., Boukadoum, M., Campeau-Lecours, A. and Gosselin, B., “A fully embedded adaptive real-time hand gesture classifier leveraging HD-sEMG and deep learning,” IEEE Trans. Biomed. Circuits Syst. 14(2), 232–243 (2020).CrossRefGoogle ScholarPubMed
Shaeffer, D. K., “MEMS inertial sensors: A tutorial overview,” IEEE Commun. Mag. 51(4), 100–109 (2013).CrossRefGoogle Scholar
Jain, R., Semwal, V. and Kaushik, P., “Stride segmentation of inertial sensor data using statistical methods for different walking activities,” Robotica 40(8), 2567–2580 (2021).CrossRefGoogle Scholar
Shao, Y., Hu, H. and Visell, Y., “A wearable tactile sensor array for large area remote vibration sensing in the hand,” IEEE Sens. J. 20(12), 6612–6623 (2020).CrossRefGoogle Scholar
Ohka, M., Takata, J., Kobayashi, H., Suzuki, H., Morisawa, N. and Yussof, H. B., “Object exploration and manipulation using a robotic finger equipped with an optical three-axis tactile sensor,” Robotica 27(5), 763–770 (2009).CrossRefGoogle Scholar
Hammock, M. L., Chortos, A., Tee, B. C.-K., Tok, J. B.-H. and Bao, Z., “25th anniversary article: The evolution of electronic skin (e-skin): A brief history, design considerations, and recent progress,” Adv. Mater. 25(42), 5997–6038 (2013).CrossRefGoogle ScholarPubMed
Yang, J. C., Mun, J., Kwon, S. Y., Park, S., Bao, Z. and Park, S., “Electronic skin: Recent progress and future prospects for skin-attachable devices for health monitoring, robotics, and prosthetics,” Adv. Mater. 31(48), 1–50 (2019).Google ScholarPubMed
Yu, Y., Nassar, J., Xu, C., Min, J., Yang, Y., Dai, A., Doshi, R., Huang, A., Song, Y., Gehlhar, R., Ames, A. and Gao, W., “Biofuel-powered soft electronic skin with multiplexed and wireless sensing for humanmachine interfaces,” Sci. Robot. 5(41), eaaz7946 (2020).CrossRefGoogle ScholarPubMed
Hua, Q., Sun, J., Liu, H., Bao, R., Yu, R., Zhai, J., Pan, C. and Wang, Z. L., “Skin-inspired highly stretchable and conformable matrix networks for multifunctional sensing,” Nat. Commun. 9(1), 1–11 (2018).CrossRefGoogle ScholarPubMed
Kim, K., Kim, B. and Lee, C. H., “Printing flexible and hybrid electronics for human skin and eye-interfaced health monitoring systems,” Adv. Mater. 32(15), 1–22 (2020).Google ScholarPubMed
Lee, J. H., Heo, J. S., Kim, Y.-J., Eom, J., Jung, H. J., Kim, J. W., Kim, I., Park, H.-Y., Mo, H. S., Kim, Y. H. and Park, S. K., “A behavior-learned cross-reactive sensor matrix for intelligent skin perception,” Adv. Mater. 32(22), 1–11 (2020).Google ScholarPubMed
Shih, B., Shah, D., Li, J., Thuruthel, T. G., Park, Y. L., Iida, F., Bao, Z., Kramer-Bottiglio, R. and Tolley, M. T., “Electronic skins and machine learning for intelligent soft robots,” Sci. Robot. 5(41), 18453663 (2020).CrossRefGoogle ScholarPubMed
Gu, G., Zhang, N., Xu, H., Lin, S., Yu, Y., Chai, G., Ge, L., Yang, H., Shao, Q., Sheng, X., Zhu, X. and Zhao, X., “A soft neuroprosthetic hand providing simultaneous myoelectric control and tactile feedback,” Nat. Biomed. Eng. (2021). doi: 10.1038/s41551-021-00767-0.CrossRefGoogle ScholarPubMed
Boutry, C. M., Negre, M., Jorda, M., Vardoulis, O., Chortos, A., Khatib, O. and Bao, Z., “A hierarchically patterned, bioinspired e-skin able to detect the direction of applied pressure for robotics,” Sci. Robot. 3(24), eaau6914 (2018).CrossRefGoogle ScholarPubMed
Rahiminejad, E., Parvizi-Fard, A., Iskarous, M. M., Thakor, N. V. and Amiri, M., “A biomimetic circuit for electronic skin with application in hand prosthesis,” IEEE Trans. Neural Syst. Rehabil. Eng. 29, 2333–2344 (2021). doi: 10.1109/TNSRE.2021.3120446.CrossRefGoogle ScholarPubMed
Liu, Y., Yiu, C., Song, Z., Huang, Y., Yao, K., Wong, T., Zhou, J., Zhao, L., Huang, X., Nejad, S., Wu, M., Li, D., He, J., Guo, X., u, J. Y., Feng, X., Xie, Z. and Yu, X., “Electronic skin as wireless humanmachine interfaces for robotic VR,” Sci. Adv. 8(2), eabl6700 (2022).CrossRefGoogle ScholarPubMed
Lee, G., Son, J., Lee, S., Kim, S., Kim, D., Nguyen, N., Lee, S. and Cho, K., “Fingerpad-inspired multimodal electronic skin for material discrimination and texture recognition,” Adv. Sci. 8(9), 2002606 (2021).CrossRefGoogle ScholarPubMed
Chen, W., Khamis, H., Birznieks, I., Lepora, N. F. and Redmond, S. J., “Tactile sensors for friction estimation and incipient slip detection - toward dexterous robotic manipulation: A review,” IEEE Sens. J. 18(22), 9049–9064 (2018).CrossRefGoogle Scholar
Jiang, S., Li, L., Xu, H., Xu, J., Gu, G. and Shull, P. B., “Stretchable e-Skin patch for gesture recognition on the back of the hand,” IEEE Trans. Ind. Electron. 67(1), 647–657 (2020).CrossRefGoogle Scholar
Tang, L., Shang, J. and Jiang, X., “Multilayered electronic transfer tattoo that can enable the crease amplification effect,” Sci. Adv. 7(3), eabe3778 (2021).CrossRefGoogle ScholarPubMed
Sundaram, S., Kellnhofer, P., Li, Y., Zhu, J.-Y., Torralba, A. and Matusik, W., “Learning the signatures of the human grasp using a scalable tactile glove,” Nature 569(7758), 698–702 (2019).CrossRefGoogle ScholarPubMed
Gao, W., Emaminejad, S., Nyein, H., Challa, S., Chen, K., Peck, A., Fahad, H., Ota, H., Shiraki, H., Kiriya, D., Lien, D.-H., Brooks, G., Davis, R. and Javey, A., “Fully integrated wearable sensor arrays for multiplexed in situ perspiration analysis,” Nature 529(7587), 509–514 (2016).CrossRefGoogle ScholarPubMed
Li, C., Liu, D., Xu, C., Wang, Z., Shu, S., Sun, Z., Tang, W. and Wang, Z. L., “Sensing of joint and spinal bending or stretching via a retractable and wearable badge reel,” Nat. Commun. 12(1), 2950 (2021).CrossRefGoogle ScholarPubMed
Liu, Y., Yiu, C., Song, Z., Huang, Y., Yao, K., Wong, T., Zhou, J., Zhao, L., Huang, X., Nejad, S., Wu, M., Li, D., He, J., Guo, X., Yu, J., Feng, X., Xie, Z. and X. Yu, “Electronic skin as wireless human machine interfaces for robotic VR,” Sci. Adv. 8(2), 509 (2022).Google ScholarPubMed
Miyata, N., Kouchi, M., Kurihara, T. and M. Mochimaru, “Modeling of Human Hand Link Structure from Optical Motion Capture Data,” In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), vol. 3 (2004) pp. 2129–2135.Google Scholar