Please wait a minute...
Tsinghua Science and Technology  2019, Vol. 24 Issue (06): 654-662    doi: 10.26599/TST.2018.9010096
Skill Learning for Human-Robot Interaction Using Wearable Device
Bin Fang*, Xiang Wei, Fuchun Sun, Haiming Huang, Yuanlong Yu, Huaping Liu
∙ Bin Fang, Funchun Sun, and Huaping Liu are with the Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China. E-mail:;
∙ Haiming Huang is with the College of Information Engineering, Shenzhen University, Shenzhen 518060, China. E-mail:
∙ Xiang Wei and Yuanlong Yu are with Fuzhou University, Fuzhou 350108, China. E-mail:;
Download: PDF (9559 KB)      HTML
Export: BibTeX | EndNote (RIS)      


With the accelerated aging of the global population and escalating labor costs, more service robots are needed to help people perform complex tasks. As such, human-robot interaction is a particularly important research topic. To effectively transfer human behavior skills to a robot, in this study, we conveyed skill-learning functions via our proposed wearable device. The robotic teleoperation system utilizes interactive demonstration via the wearable device by directly controlling the speed of the motors. We present a rotation-invariant dynamical-movement-primitive method for learning interaction skills. We also conducted robotic teleoperation demonstrations and designed imitation learning experiments. The experimental human-robot interaction results confirm the effectiveness of the proposed method.

Key wordsskill learning      interaction      teleoperation      dynamical movement primitive     
Received: 09 February 2018      Published: 20 June 2019
Corresponding Authors: Bin Fang   
About author:

Haiming Huang received the PhD degree from Beihang University in 2016. He is currently a postdoctoral in the College of Information Engineering, Shenzhen University. His research interests include soft robotics, flexible sensor, embedded mechatronics control, and robotics.

Cite this article:

Bin Fang, Xiang Wei, Fuchun Sun, Haiming Huang, Yuanlong Yu, Huaping Liu. Skill Learning for Human-Robot Interaction Using Wearable Device. Tsinghua Science and Technology, 2019, 24(06): 654-662.

URL:     OR

Fig. 1 Proposed skill learning system for human-robot interaction.
Fig. 2 Wearable device.
Human armBaxter robotic arm
Robotic jointData glove
Fore-arm4Yaw of upper and fore arm relative angle
5Roll of fore and upper arm relative angle
Palm6Pitch of palm and fore arm relative angle
7Roll of palm and fore arm relative angle
Table 1 Mapping commands of 7-DOF robotic teleoperation system.
Fig. 3 Teleoperation scheme of 7-DOF robotic system.
Fig. 4 DMP-based skill learning and generation process
Fig. 5 Teleoperation system.
Fig. 6 Circular movement process based on imitation learning.
Fig. 7 Analysis of results in which the red curve indicates the original trajectory corresponding to Fig. 6a and the black curve indicates the reproduced trajectories corresponding to Fig. 6b.
Fig. 8 Imitation learning results for different original states.
Fig. 9 Motion trajectories of the three situations.
Fig. 10 Schematic of movement primitive library.
Movement primitiveDistance (cm)Similarity (%)
Write “a”2.267498.65
Write “8”2.240398.76
Draw a triangle1.488398.85
Draw a rectangle1.476598.90
Take something2.206798.72
Draw a circle2.671198.36
Table 2 Experimental analysis of movement primitive library.
Fig. 11 Human-robot interaction system (each line represents the “hello” action process when the robot detects the person’s position).
[1]   Peer A., Einenkel S., and Buss M., Multi-fingered telemanipulation-mapping of a human hand to a three- finger gripper, in Proc. 17th IEEE Int. Symp. on Robot and Human Interactive Communication, Munich, Germany, 2008, pp. 465-470.
[2]   Rosell J., Suárez R., Rosales C., and Pérez A., Autonomous motion planning of a hand-arm robotic system based on captured human-like hand postures, Autonom. Rob., vol. 31, no. 1, pp. 87-102, 2011.
[3]   Pao L. and Speeter T. H., Transformation of human hand positions for robotic hand control, in Proc. 1989 Int. Conf. on Robotics and Automation, Scottsdale, AZ, USA, 1989, pp. 1758-1763.
[4]   Lin Y. and Sun Y., Grasp mapping using locality preserving projections and kNN regression, in Proc. 2013 IEEE Int. Conf. on Robotics and Automation, Karlsruhe, Germany, 2013, pp. 1076-1081.
[5]   Bócsi B., Csató L., and Peters J., Alignment-based transfer learning for robot models, in Proc. 2013 Int. Joint Conf. on Neural Networks, Dallas, TX, USA, 2013, pp. 1-7.
[6]   Zhou J. T., Tsang I. W., Pan S. J., and Tan M. K., Heterogeneous domain adaptation for multiple classes, in Proc. 17th Int. Conf. on Artificial Intelligence and Statistics, Reykjavik, Iceland, 2014, pp. 1095-1103.
[7]   Schaal S., Dynamic movement primitives—A framework for motor control in humans and humanoid robotics, in Adaptive Motion of Animals and Machines, Kimura H., Tsuchiya K., Ishiguro A., and Witte H., eds. Springer, 2006, pp. 261–280.
[8]   Pastor P., Hoffmann H., Asfour T., and Schaal S., Learning and generalization of motor skills by learning from demonstration, in Proc. 2009 IEEE Int. Conf. on Robotics and Automation, Kobe, Japan, 2009, pp. 763-768.
[9]   Ijspeert A. J., Nakanishi J., Hoffmann H., Pastor P., and Schaal S., Dynamical movement primitives: Learning attractor models for motor behaviors, Neural Comput., vol. 25, no. 2, pp. 328-373, 2013.
[10]   Metzen J. H., Fabisch A., Senger L., de G. Fernández J., and Kirchner E. A., Towards learning of generic skills for robotic manipulation, Künstl. Intell., vol. 28, no. 1, pp. 15-20, 2014.
[11]   Yu T. H., Finn C., Xie A. N., Dasari S., Zhang T. H., Abbeel P., and Levine S., One-shot imitation from observing humans via domain-adaptive meta-learning, arXiv preprint arXiv: 1802.01557, 2018.
[12]   Hussein A., Gaber M. M., Elyan E., and Jayne C., Imitation learning: A survey of learning methods, ACM Comput. Surv., vol. 50, no. 2, p. 21, 2017.
[13]   Kober J., Bagnel J., and Peters J., Reinforcement learning in robotics: A survey, Int. J. Rob. Res., vol. 32, no. 11, pp. 1238-1274, 2013.
[14]   Ijspeert A. J., Nakanishi J., and Schaal S., Movement imitation with nonlinear dynamical systems in humanoid robots, in Proc. 2002 IEEE Int. Conf. on Robotics and Automation, Washington, DC, USA, 2002, pp. 1398-1403.
[15]   Herzog S., W?rg?tter F., and Kulvicius T., Optimal trajectory generation for generalization of discrete movements with boundary conditions, in Proc. 2016 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Daejeon, Korea, 2016, pp. 3143-3149.
[16]   Fang B., Sun F. C., Liu H. P., and Guo D., Development of a wearable device for motion capturing based on magnetic and inertial measurement units, Scientific Programming, vol. 2017, p. 7594763, 2017.
[17]   Fang B., Sun F. C., Liu H. P., Guo D., Chen W. D., and Yao G. D., Robotic teleoperation systems using a wearable multimodal fusion device, Int. J. Adv. Rob. Syst., vol. 14, no. 4, pp. 1-11, 2017.
[18]   Ang K. H., Chong G., and Li Y., PID control system analysis, design, and technology, IEEE Trans. Control Syst. Technol., vol. 13, no. 4, pp. 559-576, 2005.
[19]   Vakanski A., Mantegh I., Irish A., and Janabi-Sharifi F., Trajectory learning for robot programming by demonstration using hidden markov model and dynamic time warping, IEEE Trans. Syst., Man, Cybern. B: Cybern., vol. 42, no. 4, pp. 1039-1052, 2012.
[1] Yajun Liu, Junying Zhang, Aimin Li, Zhaowen Liu, Zhongzhen He, Xiguo Yuan, Shouheng Tuo. Prediction of Cancer-Associated piRNA–mRNA and piRNA–lncRNA Interactions by Integrated Analysis of Expression and Sequence Data[J]. Tsinghua Science and Technology, 2018, 23(2): 115-125.
[2] Lei Chen,Jing Zhang,Lijun Cai,Ziyun Deng. Fast Community Detection Based on Distance Dynamics[J]. Tsinghua Science and Technology, 2017, 22(6): 564-585.
[3] Xiaoqing Peng,Xiaodong Yan,Jianxin Wang. Framework to Identify Protein Complexes Based on Similarity Preclustering[J]. Tsinghua Science and Technology, 2017, 22(1): 42-51.
[4] Jiancheng Zhong,Jianxin Wang,Wei Peng,Zhen Zhang,Min Li. A Feature Selection Method for Prediction Essential Protein[J]. Tsinghua Science and Technology, 2015, 20(5): 491-499.
[5] Wei Lan,Jianxin Wang,Min Li,Wei Peng,Fangxiang Wu. Computational Approaches for Prioritizing Candidate Disease Genes Based on PPI Networks[J]. Tsinghua Science and Technology, 2015, 20(5): 500-512.
[6] Zhao Muwei,Zhong Wei,He Jieyue. PBNA: An Improved Probabilistic Biological Network Alignment Method[J]. Tsinghua Science and Technology, 2014, 19(6): 658-667.
[7] Guo Xuan,Yu Ning,Gu Feng,Ding Xiaojun,Wang Jianxin,Pan Yi. Genome-Wide Interaction-Based Association of Human Diseases — A Survey[J]. Tsinghua Science and Technology, 2014, 19(6): 596-616.
[8] . Simulations of Interaction Among GMRs in a Nano-Sized Biosensor Array[J]. Tsinghua Science and Technology, 2011, 16(2): 151-156.
[9] . A Computational Model of Concept Generalization in Cross-Modal Reference[J]. Tsinghua Science and Technology, 2011, 16(2): 113-120.
[10] . Causal Inference in Graph-Text Constellations: Designing Verbally Annotated Graphs*[J]. Tsinghua Science and Technology, 2011, 16(1): 7-12.
[11] . Load Distribution Assessment of Reinforced Concrete Buildings During Construction with Structural Characteristic Parameter Approach[J]. Tsinghua Science and Technology, 2009, 14(6): 746-755.
[12] . Influences of Shrinkage, Creep, and Temperature on the Load Distributions in Reinforced Concrete Buildings During Construction[J]. Tsinghua Science and Technology, 2009, 14(6): 756-764.
[13] . Dynamic Stresses in a Francis Turbine Runner Based on Fluid-Structure Interaction Analysis[J]. Tsinghua Science and Technology, 2008, 13(5): 587-592.
[14] . Thermal Stresses in a Cylinder Block Casting Due to Coupled Thermal and Mechanical Effects[J]. Tsinghua Science and Technology, 2008, 13(2): 132-136.
[15] Jingbo Liu, Yin Gu, Yan Wang, Bin Li. Efficient Procedure for Seismic Analysis of Soil-Structure Interaction System[J]. Tsinghua Science and Technology, 2006, 11(6): 625-631.