Skip to main content
×
×
Home

Gesture-based system for next generation natural and intuitive interfaces

  • Jinmiao Huang (a1), Prakhar Jaiswal (a2) and Rahul Rai (a2)
Abstract

We present a novel and trainable gesture-based system for next-generation intelligent interfaces. The system requires a non-contact depth sensing device such as an RGB-D (color and depth) camera for user input. The camera records the user's static hand pose and palm center dynamic motion trajectory. Both static pose and dynamic trajectory are used independently to provide commands to the interface. The sketches/symbols formed by palm center trajectory is recognized by the Support Vector Machine classifier. Sketch/symbol recognition process is based on a set of geometrical and statistical features. Static hand pose recognizer is incorporated to expand the functionalities of our system. Static hand pose recognizer is used in conjunction with sketch classification algorithm to develop a robust and effective system for natural and intuitive interaction. To evaluate the performance of the system user studies were performed on multiple participants. The efficacy of the presented system is demonstrated using multiple interfaces developed for different tasks including computer-aided design modeling.

  • View HTML
    • Send article to Kindle

      To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

      Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

      Find out more about the Kindle Personal Document Service.

      Gesture-based system for next generation natural and intuitive interfaces
      Available formats
      ×
      Send article to Dropbox

      To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

      Gesture-based system for next generation natural and intuitive interfaces
      Available formats
      ×
      Send article to Google Drive

      To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

      Gesture-based system for next generation natural and intuitive interfaces
      Available formats
      ×
Copyright
Corresponding author
Author for correspondence: Rahul Rai, E-mail: rahulrai@buffalo.edu
References
Hide All
Babu, SSS, Jaiswal, P, Esfahani, ET and Rai, R (2014) Sketching in air: a single stroke classification framework. In ASME 2014 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers, pp. V02AT03A005V02AT03A005.
Bahlmann, C, Haasdonk, B and Burkhardt, H (2002) Online handwriting recognition with support vector machines-a kernel approach. In Proceedings of the Eighth International Workshop on Frontiers in Handwriting Recognition. IEEE, pp. 4954.
Baudel, T and Beaudouin-Lafon, M (1993) Charade: remote control of objects using free-hand gestures. Communications of the ACM 36(7), 2835.
Belaid, A and Haton, J-P (1984) A syntactic approach for handwritten mathematical formula recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 6(1), 105111.
Bhat, R, Deshpande, A, Rai, R and Esfahani, ET (2013) BCI-touch based system: a multimodal CAD interface for object manipulation. In ASME 2013 International Mechanical Engineering Congress and Exposition. American Society of Mechanical Engineers, pp. V012T13A015V012T13A015.
Bretzner, L, Laptev, I and Lindeberg, T (2002) Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering. In Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, pp. 423428.
Chang, C-C and Lin, C-J (2011) LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST) 2(3), 27.
Chang, KI, Bowyer, W and Flynn, PJ (2006) Multiple nose region matching for 3D face recognition under varying facial expression. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(10), 16951700.
Connell, SD and Jain, AK (2001) Template-based online character recognition. Pattern Recognition 34(1), 114.
Cortes, C and Vapnik, V (1995) Support-vector networks. Machine Learning 20(3), 273297.
Deepu, V, Madhvanath, S and Ramakrishnan, AG (2004) Principal component analysis for online handwritten character recognition. In Proceedings of the 17th International Conference on Pattern Recognition (ICPR). IEEE, vol. 2, pp. 327330.
Dimitriadis, YA and López Coronado, J (1995) Towards an ART based mathematical editor, that uses on-line handwritten symbol recognition. Pattern Recognition 28(6), 807822.
Doliotis, P, Athitsos, V, Kosmopoulos, D and Perantonis, S (2012) Hand shape and 3D pose estimation using depth data from a single cluttered frame. In Advances in Visual Computing, Part of the Lecture Notes in Computer Science book series (LNCS, volume 7431). Berlin, Heidelberg: Springer Berlin Heidelberg, pp. 148158.
Domingos, P (2012) A few useful things to know about machine learning. Communications of the ACM 55(10), 7887.
Durgun, FB and Özgüç, B (1990) Architectural sketch recognition. Architectural Science Review 33(1), 316.
Eggli, L, Brüderlin, BD and Elber, G (1995) Sketching as a solid modeling tool. In Proceedings of the Third ACM Symposium on Solid Modeling and Applications. ACM, pp. 313322.
Forman, G (2003) An extensive empirical study of feature selection metrics for text classification. Journal of Machine Learning Research 3, 12891305.
Freeman, WT and Roth, M (1995) Orientation histograms for hand gesture recognition. In International Workshop on Automatic Face and Gesture Recognition, vol. 12. Zurich: IEEE, pp. 296301.
Jaiswal, P, Bajad, AB, Nanjundaswamy, VG, Verma, A and Rai, R (2013) Creative exploration of scaled product family 3D models using gesture based conceptual computer aided design (C-CAD) tool. In ASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers, pp. V02AT02A038V02AT02A038.
Jenkins, DL and Martin, RR (1992) Applying constraints to enforce users’ intentions in free-hand 2-D sketches. Intelligent Systems Engineering 1(1), 3149.
Jojic, N, Brumitt, B, Meyers, B, Harris, S and Huang, T (2000) Detection and estimation of pointing gestures in dense disparity maps. In Proceedings. Fourth IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, pp. 468475.
Jolliffe, IT (1986) Principal component analysis and factor analysis. In Principal Component Analysis. New York, NY: Springer, pp. 115128.
Keskin, C, Kiraç, F, Kara, YE and Akarun, L (2011) Real time hand pose estimation using depth sensors. In IEEE International Conference on Computer Vision Workshops (ICCV Workshops). IEEE, pp. 12281234.
Koschinski, M, Winkler, H-J and Lang, M (1995) Segmentation and recognition of symbols within handwritten mathematical expressions. In International Conference on Acoustics, Speech, and Signal Processing, 1995 (ICASSP-95). IEEE, vol. 4, pp. 24392442.
Lacquaniti, F, Terzuolo, C and Viviani, P (1983) The law relating the kinematic and figural aspects of drawing movements. Acta Psychologica 54(1), 115130.
Lamb, D and Bandopadhay, A (1990) Interpreting a 3D object from a rough 2D line drawing. In Proceedings of the 1st Conference on Visualization'90. IEEE Computer Society Press, pp. 5966.
Lank, E, Thorley, JS and Chen, SJ-S (2000) An interactive system for recognizing hand drawn UML diagrams. In Proceedings of the 2000 Conference of the Centre for Advanced Studies on Collaborative Research. IBM Press, p. 7.
Li, Y (2012) Hand gesture recognition using Kinect. In IEEE 3rd International Conference on Software Engineering and Service Science (ICSESS). IEEE, pp. 196199.
Lien, C-C and Huang, C-L (1998) Model-based articulated hand motion tracking for gesture recognition. Image and Vision Computing 16(2), 121134.
Lipson, H and Shpitalni, M (1995) A new interface for conceptual design based on object reconstruction from a single freehand sketch. CIRP Annals-Manufacturing Technology 44(1), 133136.
Liu, J, Pan, Z and Xiangcheng, L (2010) An accelerometer-based gesture recognition algorithm and its application for 3D interaction. Computer Science and Information Systems 7(1), 177188.
Liu, X and Fujimura, K (2004) Hand gesture recognition using depth data. In Proceedings of the Sixth IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, pp. 529534.
Low, K-L (2004) Linear Least-Squares Optimization for Point-to-Plane ICP Surface Registration. Chapel Hill: University of North Carolina.
Lu, X, Jain, AK and Colbry, D (2006) Matching 2.5 D face scans to 3D models. IEEE Transactions on Pattern Analysis and Machine Intelligence 28(1), 3143.
Maji, S (2006) A Comparison of Feature Descriptors. Berkeley: University of California.
Malassiotis, S, Aifanti, N and Strintzis, MG (2002) A gesture recognition system using 3D data. In Proceedings of the First International Symposium on 3D Data Processing Visualization and Transmission. IEEE, pp. 190193.
Martí, E, Regincós, J, López-Krahe, J and Villanueva, JJ (1993) Hand line drawing interpretation as threedimensional objects. Signal Processing 32(1), 91110.
Matsakis, NE (1999) Recognition of handwritten mathematical expressions. PhD diss. Massachusetts Institute of Technology.
Muñoz-Salinas, R, Medina-Carnicer, R, Madrid-Cuevas, FJ and Carmona-Poyato, A (2008) Depth silhouettes for gesture recognition. Pattern Recognition Letters 29(3), 319329.
Nanjundaswamy, VG, Kulkarni, A, Chen, Z, Jaiswal, P, Shankar, SS, Verma, A and Rai, R (2013) Intuitive 3D computer-aided design (CAD) system with multimodal interfaces. In ASME 2013 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers, pp. V02AT02A037V02AT02A037.
Nishino, H, Utsumiya, K and Korida, K (1998) 3d object modeling using spatial and pictographic gestures. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology. ACM, pp. 5158.
Oikonomidis, I, Kyriazis, N and Argyros, AA (2011) Efficient model-based 3D tracking of hand articulations using Kinect. In BmVC. vol. 1, p. 3.
Osada, R, Funkhouser, T, Chazelle, B and Dobkin, D (2002) Shape distributions. ACM Transactions on Graphics (TOG) 21(4), 807832.
Pavlidis, T and Van Wyk, CJ (1985) An automatic beautifier for drawings and illustrations. SIGGRAPH Computer Graphics 19(3), 225234.
Pavlovic, VI, Sharma, R and Huang, TS (1997) Visual interpretation of hand gestures for human-computer interaction: a review. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7), 677695.
Plamondon, R and Srihari, SN (2000) Online and off-line handwriting recognition: a comprehensive survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(1), 6384.
Press, WH (2007) Numerical Recipes 3rd Edition: The Art of Scientific Computing. New York, NY: Cambridge University Press.
Ren, Z, Yuan, J, Meng, J and Zhang, Z (2013) Robust part-based hand gesture recognition using kinect sensor. IEEE Transactions on Multimedia 15(5), 11101120.
Ren, Z, Yuan, J and Zhang, Z (2011) Robust hand gesture recognition based on finger-earth mover's distance with a commodity depth camera. In Proceedings of the 19th ACM international conference on Multimedia. ACM, pp. 10931096.
Rokach, L (2010) Ensemble-based classifiers. Artificial Intelligence Review 33(1), 139.
Rubine, D (1991) Specifying gestures by example. SIGGRAPH Computer Graphics 25(4), 329337.
Shotton, J, Fitzgibbon, A, Cook, M, Sharp, T, Finocchio, M, Moore, R, Kipman, A and Blake, A (2011) Real-time human pose recognition in parts from single depth images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, pp. 12971304.
Shpitalni, M and Lipson, H (1995) Classification of sketch strokes and corner detection using conic sections and adaptive clustering. ASME Journal of Mechanical Design 119, 131135.
Stenger, B, Mendonça, PRS and Cipolla, R (2001) Model-based 3D tracking of an articulated hand. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, vol. 2, pp. II–310.
Sturman, DJ and Zeltzer, D (1994) A survey of glove-based input. Computer Graphics and Applications, IEEE 14(1), 3039.
Suryanarayan, P, Subramanian, A and Mandalapu, D (2010) Dynamic hand pose recognition using depth data. In 20th International Conference on Pattern Recognition (ICPR). IEEE, pp. 31053108.
Thakur, A and Rai, R (2015) User study of hand gestures for gesture based 3D CAD modeling. In ASME 2015 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers, pp. V01BT02A017V01BT02A017.
Tuytelaars, T and Mikolajczyk, K (2008) Local invariant feature detectors: a survey. Foundations and Trends® in Computer Graphics and Vision 3(3), 177280.
Vinayak, SM, Piya, C and Ramani, K (2012) Handy-Potter: rapid 3D shape exploration through natural hand motions. In ASME 2012 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. American Society of Mechanical Engineers, pp. 1928.
Wen, Y, Hu, C, Yu, G and Wang, C (2012) A robust method of detecting hand gestures using depth sensors. In IEEE International Workshop on Haptic Audio Visual Environments and Games (HAVE). IEEE, pp. 7277.
Wobbrock, JO, Wilson, AD and Li, Y (2007) Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology. ACM, pp. 159168.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

AI EDAM
  • ISSN: 0890-0604
  • EISSN: 1469-1760
  • URL: /core/journals/ai-edam
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Keywords

Metrics

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed