2.1. Manipulation of tools

Tools use has been a fundamental issue in cognitive science studies that seeks to comprehend the nature of intelligence [Reference Sanz, Call and Boesch15–Reference Van Lawick-Goodall17]. To enable robots to perform complex tasks that require the use of tools, many advanced studies have focused on recognizing affordance-specific features on tool objects [Reference Chen, Liang, Chen, Sun and Zhang18,Reference Xu, Chu, Tang, Liu and Vela19], which describes the potential physical interaction between object and manipulator and associates the regional part to planned sequential actions. These approaches have been successful in equipping robots with the capability to understand how objects may serve different purposes. In addition to recognizing the features of tool objects, learning, and planning are essential components in the manipulation of tools, as demonstrated in various studies [Reference Qin, Fang, Zhu, Fei-Fei and Savarese4,Reference Turpin, Wang, Tsogkas, Dickinson and Garg6,Reference Murali, Liu, Marino, Chernova and Gupta20]. Researchers have also explored methods of incorporating real-time feedback and environmental factors to improve the accuracy and precision of tool grasping [Reference Al-Shanoon and Lang21,Reference Ribeiro, de Queiroz Mendes and Grassi22]. Some studies aim to identify suitable tools for a certain task, they learn an embedding knowledge by DNN model between grasping tool, desired action, and target goal [Reference Saito, Ogata, Funabashi, Mori and Sugano23,Reference Sun and Gao24].

2.2. Learning from human videos

Plenty of recent research has explored utilizing human videos to improve the efficiency of RL in robots. Some works use data from the egocentric view to enable robots learning human skills [Reference Nair, Rajeswaran, Kumar, Finn and Gupta25,Reference Xiong, Fu, Zhang, Bao, Zhang, Huang, Xu, Garg and Lu26]. However because of the variety of data source and the slight difference of view, it becomes difficult to use the pre-trained representation in a specific manipulation task. In order to mend the domain gap, another type of work utilizes the in-domain human demonstration, where the sequence of human pose is recorded by motion capture [Reference Taheri, Ghorbani, Black and Tzionas27] or by a view from the third person [Reference Xiong, Li, Chen, Bharadhwaj, Sinha and Garg28]. Data of this kind have a narrower disparity between the human and robot domains, which makes it possible to construct efficient reward function for training imitation learning algorithms. Instead of extracting and re-targeting the whole human pose from the video, we focus on the motion of parts of important joints and the motion of the tool, which allows a flexible transfer from human morphology to robot morphology. By cooperating with the task-oriented approach with an AMP, we enable our system to learn the movement of robot arm using unstructured motion data.

2.3. Generative adversarial imitation learning

Generative adversarial imitation learning (GAIL) [Reference Ho and Ermon29] is inspired by the idea developed for generative adversarial networks (GAN) [Reference Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville and Bengio30]. It aims to train generators that learn policies matching the trajectory distribution of the dataset, meanwhile, the discriminators serve as the reward functions to judge whether the generated behaviors look like the demonstrations. By using a small number of demonstration data from experts, GAIL learns both the unknown environment’s policy and reward function. Although these methods have demonstrated success in low-dimensional domains [Reference Ho and Ermon29], their performance in high-dimensional tasks is not outstanding. Recently, Peng et al. [Reference Peng, Ma, Abbeel, Levine and Kanazawa14] have introduced AMP, which integrates task goals with generative adversarial imitation learning. This allows simulated agents to perform high-level tasks by learning to imitate behaviors from extensive motion dataset. Escontrela et al. [Reference Escontrela, Peng, Yu, Zhang, Iscen, Goldberg and Abbeel31] also apply this adversarial technique with a limited number of reference motion clips to learn locomotion skills for legged robots. In manipulation areas, we apply this technique to guide robots to interpret and perform behavior shown by demonstration and we show that the agents have more flexibility to perform more natural and feasible behaviors.

3.1. Tool keypoint definition

For each manipulation task with tool, we make the assumption that the tool objects interact with the environment containing one or more target objects. Inspired by ref. [Reference Qin, Fang, Zhu, Fei-Fei and Savarese4], in the framework of HMAMP, we define some keypoints in each task manually, which consists of a set of keypoints on the tool $\boldsymbol{K}_{\boldsymbol{o}}$ and a set of keypoints in the environment $\boldsymbol{K}_{\boldsymbol{e}}$ . Specifically, we consider $\boldsymbol{K}_{\boldsymbol{o}} =\left [ \boldsymbol{x_g}, \boldsymbol{x_f}, \boldsymbol{x_m} \right ]$ , where $\boldsymbol{x}_{\boldsymbol{g}}$ characterizes the grasping position on the tools and function point, $\boldsymbol{x}_{\boldsymbol{f}}$ characterizes the functional part that make contact with the target object and $\boldsymbol{x}_{\boldsymbol{m}}$ represents an auxiliary point that aides to determine the direction of tool. Environment keypoints are represented as $\boldsymbol{K}_{\boldsymbol{e}} =\left [ \boldsymbol{x}_{\boldsymbol{c}}\right ]$ , where $\boldsymbol{x}_{\boldsymbol{c}}$ denotes the position where the target interacts with the tool. The keypoints of tools and environment are mostly used to determine the goal reward which needs to be optimized in a reinforcement learning problem.

3.2. Rewards design

As mentioned in ref. [Reference Peng, Ma, Abbeel, Levine and Kanazawa14], the total reward $r_t$ is determined as a goal reward $r^g_t$ , which is specific to the task, helps to describe the completion of task and a style reward $r^s_t$ that evaluates whether the behaviors produced by the agent is similar to the behaviors produced via the distribution of reference motion set. The more similar the behaviors are, the higher value the style reward is. The proportion between style reward and goal reward is adjusted manually before training.

(1) \begin{equation} r_t = \alpha ^g r^g_t + \beta ^s r^s_t \end{equation}

3.2.1. Goal reward about task

The goal reward is related to the task and must be designed specifically. For the tool manipulation tasks where there is a contact with environment, we design the goal reward $r^g_t$ composed by two terms at one instance $t$ .

(2) \begin{equation} \begin{aligned} r^g_t &= \omega ^f r^f_t + \omega ^d r^d_t \\ r^f_t &= \left \{ \begin{array}{rcl} \lVert F^{s}_t(\boldsymbol{x}_f,\boldsymbol{x}_c)\rVert /F^d & & {F^{s}_t \leq F^d} \\ 1 \quad \quad \quad \quad & & {F^{s}_t \gt F^d} \end{array}\right . \\ r^d_t &= 1- \tanh {(\left \lVert \boldsymbol{x}_f - \boldsymbol{x}_c\rVert _t\right )} \end{aligned} \end{equation}

The first term $r_t^f$ stimulates task completion, which is defined with the contact force detected on the target object that is exerted by the tool, the force is captured by force sensor in the simulation. The detected force is encouraged to converge towards the force $F_d$ that we desire. This reward has values only when contact occurs and after the detection of contact force, the policy terminates. The second term $r_t^d$ is defined between tool function point $\boldsymbol{x}_f$ and environment target point $\boldsymbol{x}_c$ to guide to exploit a policy that minimizes the distance between tool and target. The utilization of $\tanh$ function aims to bind the reward to $[0, 1]$ . These terms are weighted with manually specified coefficients $\omega ^f$ and $\omega ^d$ , however, the magnitude of second term is much lower than the first.

3.2.2. Style reward with motion prior

As the idea mentioned in ref. [Reference Peng, Ma, Abbeel, Levine and Kanazawa14], we define a discriminator $D_{\phi }$ as a neural network with parameter $\phi$ , the discriminator is trained to distinguish whether a transition $(s, s')$ is a fake one produced by an agent or a true one sampled from a real motion distribution $d^{\mathcal{M}}$ .

We update the objective of discriminator as:

(3) \begin{equation} \begin{aligned} \underset {\phi }{\operatorname {argmin}}\quad &\mathbb{E}_{d^{\mathcal{M}}(s,s')}\left [\left (D_{\phi }(s,s')-1\right )^2\right ] \\+&\mathbb{E}_{d^\pi (s,s')}\left [\left (D_{\phi }\left (s,s'\right )+1\right )^2\right ] \\ +&\dfrac {w^{gp}}{2} \mathbb{E}_{d^{\mathcal{M}}(s,s')} \left [\lVert \nabla _{\phi }D_{\phi }\left (s,s'\right )\rVert ^2\right ] \end{aligned} \end{equation}

The former two terms serve to motivate the discriminator to differentiate between the input state derived from a policy and the input state derived from the reference motion data. They are proposed in LSGAN [Reference Mao, Li, Xie, Lau and Wang32] to solve the challenge of vanishing gradients caused by the standard GAN objective function where a sigmoid cross-entropy loss function is usually used. LSGAN prefer to optimize $\chi ^2$ divergence between the reference distribution and the policy distribution, which may alleviate the mode collapse problem and lead to stable performance during the training process [Reference Mao, Li, Xie, Lau, Zhen and Smolley33]. The last term in (3) is a gradient penalty term to penalize nonzero gradients on samples, which may avoid oscillations and improve training stability. The $w^{gp}$ in the formula is a coefficient adjusted manually.

The style reward is then defined by:

(4) \begin{equation} r^s_t = \max \left [0, 1-\gamma ^d (D_{\phi }(s,s')-1)^2 \right ] \end{equation}

with the additional offset and scale, the style reward is bounded between $[0, 1]$ .

Figure 2. Framework of HMAMP. With human manipulation video clips, we extract the keypoints of human arm and manipulation tools. Then we do keypoints alignment between robot arm in simulation and real-world human motion clips. The AMP Discriminator is to discriminate whether an action sequence is a real human expert motion or generated by the policy network. The AMP reward and task reward for manipulation task is added to be the total reward for RL training.

The training process of policy and discriminator is shown in Figure 2. The agent steps to interact with environment and produces a state transition $(s,s')$ , the observation in the environment is used to calculate the goal reward $r^g_t$ . The discriminator takes the state transition from a simulated environment and from a reference motion clips to calculate the style reward $r^s_t$ . In the end, the combined reward is used to optimize competitively the policy and the discriminator. The training details are demonstrated in Algorithm 1.

Algorithm 1. HMAMP: Learning Human-like Manipulation Skills by Adversarial Motion Prior

4.1. Data preprocess

The raw data used for HMAMP consist of video clips capturing manipulation skills from a third-person perspective. This type of data offers several advantages: it minimizes the domain gap between simulation and reality, and it is relatively easy to acquire, making it a practical choice for training purposes. In this work, we create the dataset which records human skill: hammering. The duration of the human motion in each video clips is less than one second, and we collect five pieces of video clips that perform the hammering movement by two persons. Then, we use the most popular keypoint detection algorithm BlazePose [Reference Bazarevsky, Grishchenko, Raveendran, Zhu, Zhang and Grundmann34] to detect human joint including Hip, Elbow, Wrist, Hand, and we use a CV algorithm to extract time-series of tool keypoints by manually signing them on the tool.

To retarget human motion to robot motion, it is necessary to construct an effective transfer function that maps the human world space to the robot world space. Numerous studies [Reference Geng, Lee and Hülse35–Reference Suárez, Rosell and García37] have explored motion retargeting between these two domains, and any established retargeting method can be applied in this process. In our work, we adopt a simple and straightforward approach: direct mapping. While there are significant differences in topology between the human arm and the robot arm, they share corresponding joints, such as the human elbow, wrist, and hand, which align with certain robot joints and the end-effector. By mapping key human joints to their robotic counterparts, we achieve a rough but effective transfer of motion from the human domain to the robot domain.

4.2. Model details

The policy that we used in HMAMP is PPO, the hidden parameters of policy network are of size [512, 256, 128] with exponential linear unit activation layers. The policy outputs the distribution from which the target joint positions are sampled with the representation of mean and standard deviation. Then, the target joint positions are fed to our customized PD controllers to compute the motor torques. The policy is trained on an observation $o_t$ derived from the state, which contains environment information such as hammer position and nail position and robot information such as joint angles, joint velocities, end-effector orientation, and previous actions. The discriminator is an MLP with hidden layers of size [1024, 512] and exponential linear unit activation layers. It takes the orientation of robot joints and the orientation of tools as input. The value of all manually determined parameters is: $\alpha ^g = 0.6,{} \beta ^s = 0.4, \gamma ^d = 0.25, \omega ^f = 10^5, \omega ^d = 1, \omega ^{gp} = 1, F^d = 100$ .

4.3. Simulation

The robot that we choose to train the policy is Kinova Gen3 with the gripper 2f85, the correspondence joint mapping result between Gen3 and human is shown in Figure 3.

Figure 3. Direct mapping between human and robot arm. Some joints and the gripper of a Kinova Gen3 are mapped to act as human hip, elbow, wrist, and hand.

We selected Isaac Gym [38] as our simulation platform due to its ability to accelerate the RL training process using GPU resources. The policy was trained in parallel across 2048 agents, utilizing a single NVIDIA RTX 3090 GPU. The entire training process required 11 h of wall-clock time and spanned 60,000 training epochs. Each RL episode lasted a maximum of 152 steps, corresponding to 3 s of simulated time, and terminated early if the termination criteria were met. The policy operated at a control frequency of 50 Hz during the simulation.

4.4. Termination

An episode terminates and the next one starts when the robot satisfy the termination criteria. It contains task finish signal when the hammer knocks the nail and the collision signal when a force is detected between robot components or between the robot and the table or the hammer and the table.

4.5. Domain randomization and training process

In order to improve the robustness of HMAMP policy and facilitate the transfer of learned policy from simulation to the real world, we apply domain randomization in the training process. In detail, we randomize the coefficient of friction applied to hammer and nail to $[0.5, 1.25]$ and the joint-level PD gains to $[0.9, 1.1]$ . In addition, the same observation noise as in ref. [Reference Zorina, Carpentier, Sivic and Petrík12] is added during the training phase, where the Cartesian position observation noise is $\pm 0.01\,{\rm m}$ and joint position observation noise is $\pm 0.02\,{\rm rad}$ .

From Figure 4, the training curve shows the confrontation and balance between style reward and goal reward. The early stage goal reward has a strong guiding effect, while in the late stage, amp discriminator quickly converges, giving the movements of robot a human style.

Figure 4. Training curves of HMAMP. Left figure shows the evolution of reach reward and knock force reward, while right figure indicates the discriminator loss and gradient in the training process. The two figures show the confrontation and balance between style reward and goal reward. In the early stage goal reward has a strong guiding effect while in the late stage amp discriminator converges quickly, giving the trajectory of robot a human style.

5.1. Comparative experiment in simulation

5.1.4. Results in simulation

We implemented the two baseline control strategies, DPPCP and RL-noAMP, as described in Section 5.1.2, within the simulation environment. Comparative experiments were conducted to evaluate their performance against our proposed method. Each method was tested across 10 trials, and the average values for each evaluation criterion were calculated to ensure stable and reliable performance measurements.

Table I. Comparison results of HMAMP with baselines.

Table I presents a comprehensive comparison of the various approaches, highlighting key performance metrics. The HMAMP consistently outperforms both DPPCP and RL-noAMP in all evaluated aspects. Notably, it has an overwhelming advantage in the impulse received by the nail, which is the most critical task of hammering nails. From the table, we can also obtain that the HMAMP is more efficient in hammering, since both the energy efficiency and vertical force ratio excel the other methods. In addition, HMAMP has the closest Frechet Distance to human manipulation trajectories, which reflects the effectiveness of our method in learning human motion styles. We can experience this more intuitively from Figure 5.

The experimental results clearly demonstrate the effectiveness of the proposed HMAMP framework in learning task-oriented, human-like manipulation skills through the use of AMP. The integration of AMP significantly improves task completion efficiency, knock impulse, and energy efficiency, resulting in superior performance compared to both the direct path-planning approach and traditional reinforcement learning (RL) methods.

Figure 5. Movement trajectory of the end of the robotic arm in Cartesian space. The end-of-arm motion trajectory obtained by HMAMP is the most similar to the human expert trajectory.

5.2. Real robot arm experiment

Figure 6. Experiment in simulation and real world. The first row shows human knocking motion clips that we used as motion priors. The second row shows the policy HMAMP in simulation, the hammer can successfully complete the task with the manipulation trajectory that we desired. The third row shows the HMAMP implemented in real world on Kinova Gen3, and the fourth row is the details about hammering a nail in real world.

We employed the same arm and the same task as the simulation for real-world experiments. The environment setup closely resembled real-world scenarios to ensure the applicability of our approach in practical scenarios.

Our proposed approach, which incorporates AMP into reinforcement learning (RL), was integrated into the control system of the robotic arm. The parameters and settings were optimized based on the training results obtained in the simulation environment. The robotic arm was tasked with performing the manipulation task using the learned skills.

Figure 6 showcases the manipulation effect of HMAMP on the real robot arm. The sequence of images illustrates the robotic arm successfully completing the manipulation task with precision and human-like motion. The trajectory followed by the arm demonstrates smoothness, accuracy, energy efficiency, and human manipulation skills, confirming the benefits of incorporating AMPs.