2.1 UAV model

This work considers a hybrid tricopter UAV whose front two propellers are fixed, forward-facing and provide thrust only when required, while the tail rotor remains idle during the manoeuvre (Fig. 2). This configuration simplifies the mechanism relative to tilt-prop systems and emphasises the gliding dynamics of DS. It also tightens control authority: longitudinal thrust must be scheduled sparingly to preserve energy neutrality; attitude and path shaping must come primarily from aerodynamic loading and crosswind manoeuvring. The UAV under consideration has a wing span ‘l’ of 0.61 m and a mass ‘m’ of 1.5 kg. The rotors are labeled as ${R_1}$ , ${R_2}$ and ${R_3}$ where ${\sigma _1}$ and ${\sigma _2}$ are the tilt angles for front rotors ${R_1}$ and ${R_2}$ , respectively. For simplicity, the rear rotor is assumed to remain inactive, while the front two rotors are modeled as fixed, non-tiltable propellers. The modeling parameters of the UAV are elaborated in Table 1.

Figure 2.

Schematic diagram of UAV considered: coordinate system and forces.

Table 1.

Modeling parameters of tricopter

2.2 Coordinate system

The coordinate system is illustrated in Fig. 2. The body-fixed axes are defined as follows: ${X_B}$ axis points forward from the tricopter’s perspective, ${Y_B}$ axis extends to the right and ${Z_B}$ axis points downward from the tricopter’s centre of gravity. The relationship between the inertial (earth-fixed) frame and the body frame can be described using the rotation matrix R $\left({\phi ,\theta ,\psi } \right)$ . It is the direction cosine matrix (DCM) or coordinate transformation matrix. The vehicle’s orientation is represented by the Euler angles: roll ( $\phi $ ), pitch ( $\theta $ ) and yaw ( $\psi $ ), which correspond to rotations about ${X_B}$ , ${Y_B}$ and ${Z_B}$ axes, respectively. The transformation from the inertial (navigation) frame (X,Y,Z) to the body frame ( ${X_b},{Y_b},{Z_b}$ ) is expressed through the DCM, as shown in Equation (1).

(1) \begin{align}\left[ {\begin{array}{*{20}{c}}{{X_B}}\\{{Y_B}}\\{{Z_B}}\end{array}} \right] = R\left({\phi ,\theta ,\psi } \right)\left[ {\begin{array}{*{20}{c}}X\\Y\\Z\end{array}} \right]\end{align}

The rotation is first applied to the ${Z_B}$ axis with an angle of $\psi $ , then to the ${Y_B}$ axis with an angle of $\theta $ , and finally the ${X_B}$ axis is rotated with an angle of $\phi $ . The rotation matrix $R\left({z,\psi } \right)$ along the z-axis, where $\psi $ is the rolling angle along the z-axis, is given as Equation (2):

(2) \begin{align}R\left({z,\psi } \right) = \left[ {\begin{array}{*{20}{c}}{\cos\psi }&\quad{\sin\psi }&\quad0\\{ - \sin\psi }&\quad{\cos\psi }&\quad0\\0&\quad0&\quad 1\end{array}} \right]\end{align}

The rolling matrix $R\left({y,\theta } \right)$ , where $\theta $ is the rotation angle along the y-axis, is as follows in Equation (3):

(3) \begin{align}R\left({y,\theta } \right) = \left[ {\begin{array}{*{20}{c}}{\cos\theta }&\quad 0&\quad { - \sin\theta }\\0&\quad 1&\quad 0\\{\sin\theta }&\quad 0&\quad {\cos\theta }\end{array}} \right]\end{align}

The rotation matrix $R\left({x,\phi } \right)$ rotates around the x-axis with a rotation angle $\phi $ given as Equation (4):

(4) \begin{align}R\left({x,\phi } \right) = \left[ {\begin{array}{*{20}{c}}1&\quad 0&\quad 0\\0&\quad {\cos\phi }&\quad {\sin\phi }\\0&\quad { - \sin\phi }&\quad {\cos\phi }\end{array}} \right]\end{align}

By combining the defined rotation matrices, the complete transformation from the inertial frame to the body frame can be obtained:

(5) \begin{align}R\left({\phi ,\theta ,\psi } \right) = R\left({z,\psi } \right)R\left({y,\theta } \right)R\left({x,\phi } \right)\end{align}

(6) \begin{align}DCM = \left({\phi ,\theta ,\psi } \right) = \left[\! {\begin{array}{*{20}{c}}{\cos\theta \cos\psi }&\quad {\cos\theta \sin\psi }&\quad { - \sin\theta }\\{\sin\phi \sin\theta \cos\psi - \cos\phi \sin\psi }&\quad {\sin\phi \sin\theta \sin\psi + \cos\phi \cos\psi }&\quad {\sin\phi \cos\theta }\\{\cos\phi \sin\theta \cos\psi + \sin\phi \sin\psi }&\quad {\cos\phi \sin\theta \sin\psi - \sin\phi \cos\psi }&\quad {\cos\phi \cos\theta }\end{array}} \!\right]\end{align}

2.3 Flight dynamics model

The tricopter UAV’s flight dynamics are governed by a six-degree-of-freedom (6-DOF) nonlinear model, which captures the translational and rotational motion of the vehicle in three-dimensional space. These dynamics are expressed in the body-fixed frame, with respect to the centre of gravity of the UAV, and are influenced by aerodynamic forces, propulsive forces (when active) and gravity.

The flight mechanics model of the drone in this article differs from that of conventional drones due to the usage of rotors along with a fixed wing. The rotor at aft is considered to be inactive, thus there is no thrust component in the force acting on the drone and no momentum change caused by the engine rotor in the torque calculation formula. However, the front two rotors provide a limited thrust according to mission requirement.

The model is described in terms of two aspects:

1. 1. the translational dynamics (motion along body axes- ${X_B}$ , ${Y_B}$ , ${Z_B}$ )

2. 2. the rotational dynamics (angular motion about body axes- $\phi $ , $\theta $ , $\psi $ )

The state of the UAV is described by its linear velocities u,v,w, angular rates p,q,r, Euler angles $\phi $ , $\theta $ , $\psi $ and inertial positions x,y,z. The translational dynamics of the UAV are governed by Newton’s Second Law, which relates the sum of external forces acting on the vehicle to the rate of change of its linear momentum.

(7) \begin{align}&{\dot u} = v - qw + \dfrac{{{F_x}}}{m} - g\, {\sin} \, \theta \nonumber\\&{\dot v} = pw - ru + \dfrac{{{F_y}}}{m} - g \, {\cos} \, \theta \, {\sin}\, \phi \nonumber\\&{\dot w} = qu - pv + \dfrac{{{F_z}}}{m} - g \, {\cos} \, \theta\, {\cos} \, \phi \end{align}

where $m$ is mass of the UAV, $g$ is gravitational acceleration and ${F_x},{F_y},{F_z}$ are the total forces in body frame, including aerodynamic ( ${F_a}$ ), propulsive ( ${F_p}$ ) and gravitational forces ( ${F_g}$ ) in each axis.

The rotational dynamics of the UAV are described by Euler’s equations, which relate the applied moments to the rates of change of the vehicle’s angular motion about its principal axes.

(8) \begin{align}&\dot{p} = \dfrac{{{I_{yy}} - {I_{zz}}}}{{{I_{xx}}}}qr + \dfrac{{{M_x}}}{{{I_{xx}}}} \nonumber\\&{\dot q} = \dfrac{{{I_{zz}} - {I_{xx}}}}{{{I_{yy}}}}pr + \dfrac{{{M_y}}}{{{I_{yy}}}} \nonumber\\&{\dot r} = \dfrac{{{I_{xx}} - {I_{yy}}}}{{{I_{zz}}}}pq + \dfrac{{{M_z}}}{{{I_{zz}}}}\end{align}

where ${I_{xx}},{I_{y{\textrm{y}}}},{I_{zz}}$ are the moments of inertia about the three axes in the body. For symmetric UAVs, the moments of inertia about the three planes ( ${I_{xy}},{I_{yz}},{I_{zx}}$ are zero. ${M_x},{M_y},{M_z}$ are the total moments in body frame. They include aerodynamic and propulsive moments.

The kinematics are described by the Euler angle rate equations.

(9) \begin{align}&{\dot \phi }{ = p + {\tan\,}\theta \left({q \, {\sin} \, \phi + r\, {\cos} \, \phi } \right)}\nonumber\\&{\dot \theta }{ = q\,{\cos} \, \,\phi - r\,{\sin}\,\phi }\nonumber\\&{\dot \psi }{ = \dfrac{{q\,{\sin} \,\phi + r\,{\cos} \, \,\phi }}{{{\cos} \, \,\theta }}}\end{align}

The dynamic pressure ( $Q$ ) is a function of aircraft speed ( ${V_a}$ ) and air density ( $\rho $ ):

(10) \begin{align}Q = \dfrac{1}{2}{{\rho }}V_a^2\end{align}

where the velocity ( ${V_a}$ ) is given by the following equation:

(11) \begin{align}{V_a} = \sqrt {{u^2} + {v^2} + {w^2}} \end{align}

The aerodynamic forces and gravitational forces in each of the three axis are given in Equation (13). Propulsive forces are those generated directly by the UAV’s rotors, and they act on its body.

These forces generated by all rotors are summed and expressed in the body frame of the UAV. The following considerations are made for the analysis: the rear rotor remains inactive throughout the flight, while the two front rotors are fixed in a forward-facing configuration without any tilt mechanism. Each rotor generates a thrust force ${T_i}$ (i = 1, 2) in the direction of its local axis and possibly a torque ${\tau _i}$ depending on the rotation direction. Propulsion is utilised only during the ascent and descent phases, whereas the main flight phase consists of passive DS, during which the UAV harnesses wind energy for sustained flight.

The propulsive forces generated by the rotors in the body frame are given by:

(12) \begin{align}\left[ {\begin{array}{*{20}{c}}{{X_p}}\\{{Y_p}}\\{{Z_p}}\end{array}} \right] = \mathop \sum \limits_{i = 1}^2 {T_i}\left[ {\begin{array}{*{20}{c}}{{\sin}\! \left({{\sigma _i}} \right)}\\{{\sin}\! \left({{\sigma _i}} \right)}\\{ - {\cos}\! \left({{\sigma _i}} \right)}\end{array}} \right]\end{align}

where ${T_i}$ is the thrust produced by the $i$ -th rotor and ${\sigma _i}$ is the rotor’s tilt angle. The notations and assumptions for rotor thrust are defined below:

1. 1. ${T_1}$ , ${T_2}$ : Thrust from front-left and front-right rotors, respectively

2. 2. no tilt assumption: ${\sigma _i} = {90^ \circ } \Rightarrow {\sin} \, \left({{\sigma _i}} \right) = 1,\cos\left({{\sigma _i}} \right) = 0$

3. 3. total rotor thrust: ${T_{{\textrm{total}}}} = {T_1} + {T_2}$ where ${T_{{\textrm{total}}}}$ is thrust for ${X_p}$ only (Equation (13))

The total force vector is expressed as the resultant of all external forces acting on the UAV. It comprises aerodynamic forces, gravitational forces and propulsive forces and is given in Equation (13).

(13) \begin{align}\left[ {\begin{array}{*{20}{c}}{{F}}\\{{F_y}}\\{{F_z}}\end{array}} \right] &= \underbrace {Q{S_{{\textrm{ref}}}}\left[ {\begin{array}{*{20}{c}}{ - {C_{{D_\alpha }}}{\cos} \, \alpha + {C_{{L_\alpha }}}{\sin} \, \alpha + ({ - {C_{{D_\alpha }}}{\cos} \, \alpha + {C_{{L_\alpha }}}{\sin} \, \alpha } )\dfrac{c}{{2{V_a}}}q}\\{{C_Y} + {C_{{Y_\beta }}}\beta + \dfrac{b}{{2{V_a}}}p{C_{{Y_p}}} + \dfrac{b}{{2{V_a}}}r{C_{{Y_r}}}}\\{ - {C_{{D_\alpha }}}{\sin} \, \alpha - {C_{{L_\alpha }}}{\cos} \, \alpha + ({ - {C_{{D_\alpha }}}{\sin} \, \alpha - {C_{{L_\alpha }}}{\cos} \, \alpha } )\dfrac{c}{{2{V_a}}}q}\end{array}} \right]}_{{\textrm{Aerodynamic forces}}}\nonumber\\&\quad + \underbrace {mg\left[ {\begin{array}{*{20}{c}}{ - {\sin} \, \theta }\\{{\cos} \, \theta \, {\sin} \, \phi }\\{{\cos} \, \theta \, {\cos} \, \phi }\end{array}} \right]}_{{\textrm{Gravitational forces}}} + \underbrace {\left[ {\begin{array}{*{20}{c}}{{T_1} + {T_2}}\\0\\0\end{array}} \right]}_{{\textrm{Propulsive forces}}}\end{align}

The general formula for a moment (torque) caused by a force applied at some point offset from the axis is:

(14) \begin{align}{\bf{M}} = \mathop \sum \limits_{i = 1}^2 \left({{{\bf{r}}_i} \times {{\bf{F}}_i}} \right)\end{align}

The magnitude of this moment is:

(15) \begin{align}\left\| {\bf{M}} \right\| = \mathop \sum \limits_{i = 1}^2 \left({\left\| {{{\bf{r}}_i}} \right\| \cdot \left\| {{{\bf{F}}_i}} \right\| \cdot {\sin} \, \left({{\sigma _i}} \right)} \right)\end{align}

where $r = l$ is the position vector, $F = T$ = thrust vector and $\sigma $ is the angle between moment arm and thrust. The rotors are located at $ \pm l$ on the y-axis, i.e. the front left rotor is located at ${{\bf{r}}_1} = {[x, + l,0]^T}$ and front right rotor is at ${{\bf{r}}_2} = {[x, - l,0]^T}$ .

(16) \begin{align}\left[ {\begin{array}{*{20}{c}}{Lp}\\{{M_p}}\\{{N_p}}\end{array}} \right] = \left[ {\begin{array}{*{20}{c}}{l \, {\cos} \, \left({{\sigma _1}} \right)}&\quad { - l \, {\cos} \, \left({{\sigma _2}} \right)}\\0&\quad 0\\{l \, {\sin} \, \left({{\sigma _1}} \right)}&\quad { - l \, {\sin} \, \left({{\sigma _2}} \right)}\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{{T_1}}\\{{T_2}}\end{array}} \right] + \left[ {\begin{array}{*{20}{c}}{{\sin} \, {\sigma _1}}&\quad {{\sin} \, {\sigma _2}}\\0&\quad 0\\{ - {\cos} \, {\sigma _1}}&\quad { - {\cos} \, {\sigma _2}}\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{ - {M_1}}\\{{M_2}}\end{array}} \right]\end{align}

Since no tilt is considered for rotors, so ${\sigma _i} = {90^ \circ } \Rightarrow {\sin} \, \left({{\sigma _i}} \right) = 1,{{\cos}}\left({{\sigma _i}} \right) = 0$

(17) \begin{align}\left[ {\begin{array}{*{20}{c}}{Lp}\\{{M_p}}\\{{N_p}}\end{array}} \right] = \left[ {\begin{array}{*{20}{c}}0&\quad 0\\0&\quad 0\\l&\quad { - l}\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{{T_1}}\\{{T_2}}\end{array}} \right] + \left[ {\begin{array}{*{20}{c}}1&\quad 1\\0&\quad 0\\0&\quad 0\end{array}} \right]\left[ {\begin{array}{*{20}{c}}{ - {M_1}}\\{{M_2}}\end{array}} \right]\end{align}

(18) \begin{align}\left[ {\begin{array}{*{20}{c}}{Lp}\\{{M_p}}\\{{N_p}}\end{array}} \right] = \left[ {\begin{array}{*{20}{c}}{ - {M_1} + {M_2}}\\0\\{l\left({{T_1} - {T_2}} \right)}\end{array}} \right]\end{align}

The total moments $\left({{M_x},{M_y},{M_z}} \right)$ are calculated from the aerodynamic moments and propulsive moments, which are given in Equation (19).

(19) \begin{align}\left[ {\begin{array}{*{20}{c}}{{M_x}}\\{{M_y}}\\{{M_z}}\end{array}} \right] = \underbrace {Q{S_{{\textrm{ref}}}}\left[ {\begin{array}{*{20}{c}}{\bar b\left({{C_L} + {C_{{L_\beta }}}\beta + {C_{{L_p}}}\dfrac{{\bar b}}{{2{V_a}}}p + {C_{{L_r}}}\dfrac{{\bar b}}{{2{V_a}}}r} \right)}\\{c\left({{C_M} + {C_{{M_\alpha }}} + {C_{{M_q}}}\dfrac{c}{{2{V_a}}}q} \right)}\\{\bar b\left({{C_N} + {C_{{N_\beta }}}\left( \beta \right) + {C_{{N_p}}}\dfrac{b}{{2{V_a}}}p + {C_{{N_r}}}\dfrac{b}{{2{V_a}}}r} \right)}\end{array}} \right]}_{{\textrm{Aerodynamic moments}}} + \underbrace {\left[ {\begin{array}{*{20}{c}}{ - {M_1} + {M_2}}\\0\\{l\left({{T_1} - {T_2}} \right)}\end{array}} \right]}_{{\textrm{Propulsive moments}}}\end{align}

2.4 Wind shear modeling

DS relies on wind shear, a phenomenon that occurs within the atmospheric boundary layer where wind speed or direction changes sharply over a short vertical distance. This gradient forms a thin interface between adjacent air layers with different wind velocities, making it a critical factor for extracting energy. Wind shear is a fundamental requirement for DS, as it allows a UAV to gain energy by transitioning between these layers [Reference Bousquet, Triantafyllou and Slotine3, Reference Mir, Eisa and Maqsood27, Reference Mir, Taha, Eisa and Maqsood47, Reference Goto, Yoda, Weimerskirch and Sato48]. Therefore, it is important to consider a wind shear model to describe the wind dynamics specifically the variation of wind speed with altitude.

In this study, the logarithmic wind profile [Reference Sachs2, Reference Mir, Eisa and Maqsood27] is considered, which assumes a neutral stability condition, i.e. no significant thermal effects. In this model, the wind speed ${V_z}$ at altitude $z$ is described as:

(20) \begin{align}{V_z} = {V_{{\textrm{ref}}}}\dfrac{{{\textrm{ln}}{\left({\dfrac{z}{{{z_0}}}} \right)}}}{{{\textrm{ln}} {\left({\dfrac{{{z_{{\textrm{ref}}}}}}{{{z_0}}}} \right)}}}\end{align}

where ${V_{ref}}$ is the wind speed measured at the reference altitude ${z_{ref}}$ , ${z_0}$ is the surface correction factor, which reflects the characteristics of terrain like surface roughness and irregularities etc. It has different values for different terrains, for example its value is $\sim 0.0002$ m for water, $\sim $ 0.03 m for forests, and $\sim $ 1–2 m for urban areas. Figure 3 shows the graphical representation of the logarithmic wind profile with the wind shear strength of ${V_{ref}}$ = 8.64 $m/s$ at reference altitude ${z_{ref}}$ = 10 m and ${z_0}$ = 0.03 m. The blue dashed lines show the intensity of wind at a given altitude, and the thick black curve shows the trend of wind velocity with altitude.

Figure 3.

Logarithmic wind profile model.

3.1 Training process

3.1.1 State space

The state space comprises the collection of all variables and parameters that the UAV obtains from the environment. At any given time during the simulation, position of the UAV and motion variables are defined as state. The state space consists of position in the inertial frame, velocity in the body frame, orientation (roll, pitch, yaw angles), wind speed and energy (potential and kinetic).

(22) \begin{align}{\textrm{state}} = {s_t} = {[x,y,z,V,\phi ,\theta ,\psi ,p,q,r,{V_z},E]^T},{\;\;\;\;\;\;\;\;}s{\left(t \right)} \in {{\mathbb R}^{12}}\end{align}

3.1.2 Action space

The action space includes all possible actions that the UAV can take at each time step in a given environment. These actions are adjusted by the policy learned by the respective algorithm. For the UAV to perform a DS manoeuvre, bank angle, flight path angle and thrust are considered the action space.

(23) \begin{align}{\textrm{action}} = {a_t} = {[{\phi _B},\gamma ,T]^T},{{\;\;\;\;\;\;\;\;}}a{\left(t \right)} \in {{\mathbb R}^3}\end{align}

In a DS manoeuvre, the UAV primarily extracts energy from wind gradients thus reducing its reliance on thrust. However, whether the rotors should be completely off or partially active depends on the mission requirements. Existing literature differentiates passive DS, where UAVs repeatedly cycle through wind shear purely via aerodynamic manoeuvres [Reference Zhu, Hou and Ouyang55, Reference Bousquet, Triantafyllou and Slotine56] from active or assisted DS, where onboard propulsion, morphing-wing mechanisms or energy-harvesting systems are employed to augment energy extraction [Reference Wang, An and Song21, Reference Joseph, Golubev and Gudmundsson57, Reference Mir, Maqsood and Akhtar58].

In this study, the primary source of energy gain comes from exploiting wind gradients, but the rotors still need to provide thrust to maintain altitude and manoeuvre efficiently. For our hybrid UAV to perform DS, the thrust of rotors depends on the phase of manoeuvre. During the climb phase, when the UAV flies into the wind, rotors provide additional thrust to maintain altitude. During descent, when the UAV flies with the wind, thrust is reduced. The thrust distribution for the respective tricopter comes from the front two rotors that assist in stabilisation and manoeuvring, while the rear rotor is assumed to be off. The front two rotors are limited to produce thrust up to 7N.

3.1.3 Reward function components

The reward function is critical for learning. For this study, the following factors are considered:

1. a. Energy gain

Rewards the UAV for efficient energy extraction from the wind gradient. It encourages soaring manoeuvres to maximise potential and kinetic energy. Let total energy at time t be:

\begin{align*}E{\left( t \right)} &= PE + KE\nonumber\\E{\left( t \right)} &= mgh{\left(t \right)} + \dfrac{1}{2}m{[v{\left(t \right)}]^2}\end{align*}

Using per-step energy gain so that the agent is rewarded for increasing energy:

\begin{align*}{{\Delta }}E{\left( t \right)} &= E{\left( t \right)} - E\left({t - 1} \right)\nonumber\\\tilde E{\left( t \right)} &= \dfrac{{{{\Delta }}E{\left( t \right)}}}{{{E_{{\textrm{max}}}}}}\nonumber\\{R_{{\textrm{energy}}}}{\left(t \right)} &= {w_e} \cdot \tilde E{\left( t \right)}\end{align*}

1. b. Control effort

Penalises the UAV against the usage of excessive control inputs to ensure smooth operation and energy efficiency.

\begin{align*}{R_{{\textrm{control}}}} &= - {w_{\textrm{c}}} \cdot \mathop \sum \limits_{c = \alpha \beta {\phi _B}} \left| {{e_c}} \right|\nonumber\\{R_{{\textrm{control}}}}{\left(t \right)} &= - {w_{\textrm{c}}} \cdot \left({\dfrac{{\left| {{{\Delta }}\alpha {\left(t \right)}} \right|}}{{{{\Delta }}{\alpha _{{\textrm{max}}}}}} + \dfrac{{\left| {{{\Delta }}\beta {\left(t \right)}} \right|}}{{{{\Delta }}{\beta _{{\textrm{max}}}}}} + \dfrac{{\left| {{{\Delta }}{\phi _B}{\left(t \right)}} \right|}}{{{{\Delta }}{\phi _{B{\textrm{max}}}}}}} \right)\end{align*}

where ${e_c}$ includes the deviations in the angle-of-attack ( ${{\Delta }}\alpha $ ), sideslip ( ${{\Delta }}\beta $ ) and bank angle ( ${{\Delta }}\phi $ ).

1. c. Safety constraints

The reward function penalises the UAV for violating the altitude and velocity limits (excess speed).

\begin{align*}{R_{{\textrm{safety}}}}{\left(t \right)} = - {w_{\textrm{s}}} \cdot \mathop \sum \limits_{j \in {\mathcal J}} \dfrac{{{\textrm{max}}(0,{\textrm{limi}}{{\textrm{t}}_j} - {\textrm{valu}}{{\textrm{e}}_j})}}{{{{\Delta }}j}}\end{align*}

where

\begin{align*}{\mathcal J} = \left\{ {{z_{{\textrm{min}}}},{\;}{z_{{\textrm{max}}}},{\;}{V_{{\textrm{min}}}},{{\;}}{V_{{\textrm{max}}}}} \right\}\end{align*}

and ${{\Delta }}$ j is the normalisation range ( ${z_{{\textrm{max}}}} - {z_{{\textrm{min}}}}$ for altitude and ${V_{{\textrm{max}}}} - {V_{{\textrm{min}}}}$ for velocity).

These can be elaborated as:

\begin{align*}{R_{{\textrm{safety}}}}{\left(t \right)} &= - {w_{zmin}} \cdot \dfrac{{{\textrm{max}}\left({0,{z_{{\textrm{min}}}} - z} \right)}}{{{z_{{\textrm{max}}}} - {z_{{\textrm{min}}}}}} - {w_{zmax}} \cdot \dfrac{{{\textrm{max}}\left({0,z - {z_{{\textrm{max}}}}} \right)}}{{{z_{{\textrm{max}}}} - {z_{{\textrm{min}}}}}}\nonumber\\&\quad- {w_{Vmin}} \cdot \dfrac{{{\textrm{max}}\left({0,{V_{{\textrm{min}}}} - V} \right)}}{{{V_{{\textrm{max}}}} - {V_{{\textrm{min}}}}}} - {w_{Vmax}} \cdot \dfrac{{{\textrm{max}}\left({0,V - {V_{{\textrm{max}}}}} \right)}}{{{V_{{\textrm{max}}}} - {V_{{\textrm{min}}}}}}\end{align*}

where: ${w_{zmn}}$ , ${w_{zmax}}$ , ${w_{Vmin}}$ and ${w_{Vmax}}$ are weighting factors for the respective penalties. $h$ represents the current altitude, ${z_{{\textrm{min}}}}$ is the minimum allowable altitude (1 $m$ ) to avoid ground collision and to maintain its cruising height, and ${z_{{\textrm{max}}}}$ is the ceiling height of 30 $m$ . $V$ represents the current velocity, ${V_{{\textrm{min}}}}$ is the minimum allowable velocity (1 $m/s$ ), and ${V_{{\textrm{max}}}}$ is the maximum allowed velocity of 40 $m/s$ .

1. d. Stable orientation

The stability reward penalises for deviations in roll, pitch and yaw. It is expressed as:

\begin{align*}{R_{{\textrm{stability}}}}{\left(t \right)} &= - {w_{{\textrm{st}}}} \cdot \mathop \sum \limits_{st = \phi \theta \psi } \left| {{e_{st}}} \right|\nonumber\\{R_{{\textrm{stability}}}}{\left(t \right)} &= - {w_{{\textrm{st}}}} \cdot \left({\dfrac{{\left| {{{\Delta }}\phi {\left(t \right)}} \right|}}{{{{\Delta }}{\phi _{{\textrm{max}}}}}} + \dfrac{{\left| {{{\Delta }}\theta {\left(t \right)}} \right|}}{{{{\Delta }}{\theta _{{\textrm{max}}}}}} + \dfrac{{\left| {{{\Delta }}\psi {\left(t \right)}} \right|}}{{{{\Delta }}{\psi _{{\textrm{max}}}}}}} \right)\end{align*}

where ${e_{st}}$ includes deviations in roll ( ${{\Delta }}\phi $ ), pitch ( ${{\Delta }}\theta $ ) and yaw ( ${{\Delta }}\phi $ ).

1. e. Thrust penalty

The thrust penalty discourages excessive rotor usage by normalising the applied thrust against the maximum available thrust.

\begin{align*}{R_{{\textrm{thrust}}}}{\left(t \right)} = - {w_t} \cdot \dfrac{{{T_{{\textrm{total}}}}}}{{{T_{max}}}}\end{align*}

where ${T_{{\textrm{total}}}}$ is the sum of rotor thrusts and ${T_{{\textrm{max}}}}$ is the maximum possible thrust.

The overall reward function is a weighted sum of the individual components as shown in Equation (24), and the values for their weightage are given in Table 2.

(24) \begin{align}reward = {r_t} = {R_{{\textrm{energy}}}}{\left(t \right)} + {R_{{\textrm{control}}}}{\left(t \right)} + {R_{{\textrm{safety}}}}{\left(t \right)} + {R_{{\textrm{stability}}}}{\left(t \right)} + {R_{{\textrm{thrust}}}}{\left(t \right)}\end{align}

Table 2.

Weights for reward function

4.1 Trust region policy optimisation (TRPO)

TRPO, proposed by Schulman et al. [Reference Schulman, Levine, Abbeel, Jordan and Moritz59], is an on-policy RL algorithm designed to enhance training stability by preventing large, abrupt changes to the policy. This is accomplished by enforcing a constraint on the step size taken in the policy space, ensuring that each update remains within a defined trust region. By maintaining updates within this region, TRPO improves learning stability and overall performance. A key element of the trust region constraint is the Kullback-Leibler (KL) divergence, which measures the difference between the new and old policy distributions. The algorithm seeks to maximise the specified objective function (Equation (25)) while satisfying the trust region constraint (Equation (26)). Figure 4 shows the architecture.

(25) \begin{align}{L_{TRPO}}{\left( \theta \right)} = {E_t}\left[ {\dfrac{{{\pi _\theta }({a_t}|{s_t})}}{{{\pi _{{\theta _{old}}}}({a_t}|{s_t})}}{{\hat A}^t}} \right]\end{align}

(26) \begin{align}{E_t}\left[ {{D_{KL}}\left[ {{\pi _{{\theta _{old}}}}({a_t}|{s_t})||{\pi _\theta }({a_t}|{s_t})} \right]} \right] \le \delta \end{align}

where ${D_{KL}}$ is the KL divergence and $\delta $ controls the size of the trust region. By limiting KL divergence during policy updates, TRPO ensures that successive policies remain close in distribution, thus avoiding drastic behavioural changes and promoting smooth learning dynamics.

Figure 4.

Architecture of the TRPO algorithm, showing interactions between the policy network, environment and value function, with policy updates constrained within a trust region for stable learning.

The trust region constraint in TRPO helps maintain stable policy evolution, which can be beneficial in DS tasks where large policy shifts may lead to loss of aerodynamic efficiency and compromise energy efficiency.

4.2 Proximal policy optimisation (PPO)

PPO, proposed by Schulman et al. [Reference Schulman, Wolski, Dhariwal, Radford and Klimov16], is an on-policy RL algorithm developed to improve training by limiting large policy updates. Instead of using a trust region constraint like TRPO, this is achieved using a clipping mechanism that constrains the change between the new and old policies, thus preventing destabilising updates and promoting smoother learning. Due to its balance between simplicity of implementation and sample efficiency, PPO has become a widely adopted algorithm across many RL tasks. Figure 5 illustrates the structure of the algorithm.

Figure 5.

Structure of the PPO algorithm, showing the interaction between the policy network, value function, environment and advantage-based policy updates.

The algorithm optimises the policy parameters $\theta $ by estimating the gradients using the Monte Carlo method. When the agent interacts with the environment, a policy loss function J ( $\theta $ ) (Equation (27)) is calculated. The back-propagation of these gradients within the neural network helps refine the parameters.

(27) \begin{align}{\nabla _\theta }J{\left( \theta \right)} = {E_{\tau \sim {\pi _\theta }{\left( {\tau} \right)}}}\left[ {\mathop \sum \limits_t^T {\nabla _\theta }{\textrm{log}}{\pi _\theta }({a_t} \, | \, {s_t}) \cdot R{\left( {\tau} \right)}} \right]\end{align}

The use of generalised advantage estimation (GAE) [Reference Schulman, Moritz, Levine, Jordan and Abbeel60] in PPO helps achieve a better policy by reducing the variance of the gradients. It is a function of discounted rewards ( ${r_{t{^{\prime}}}}$ ), value function ( ${V_\phi }\left({{s_t}} \right)$ ) and discount factor $\gamma {^{\prime}}$ for time $t{^{\prime}} \gt t$ , given by Equation (28). This function assesses the behaviour of the agent to either reward or penalise it.

(28) \begin{align}{\hat A^t} = \mathop \sum \limits_{t{^{\prime}} \gt t} {\gamma ^{t{^{\prime}} - t}}\left({{r_{t{^{\prime}}}} - {V_\phi }\left({{s_t}} \right)} \right)\end{align}

A surrogate objective function is used to restrict the deviation of new updates in policy from the previous one. It is given by:

(29) \begin{align}{L_{CLIP}}{\left( \theta \right)} = {\hat E_t}\left[ {{\textrm{min}}\left({{r_t}{\left( \theta \right)}{{\hat A}^t},{\textrm{clip}}\left({{r_t}{\left( \theta \right)},1 - \varepsilon ,1 + \varepsilon } \right){{\hat A}^t}} \right)} \right]\end{align}

where ${r_t}{\left( \theta \right)}$ (Equation (30)) is the probability ratio between the new and old policies, ${\hat A^t}$ represents the GAE, and $\varepsilon $ is a hyperparameter that controls the clipping range.

(30) \begin{align}{r_t}{\left( \theta \right)} = \dfrac{{{\pi _\theta }({a_t}\, |\, {s_t})}}{{{\pi _{{\theta _{old}}}}({a_t} \, | \,{s_t})}}\end{align}

The clipped policy update makes PPO robust to unstable learning, which is advantageous when training agents in DS environments where wind gradients introduce significant variability in rewards.

4.3 Deep deterministic policy gradient (DDPG)

The DDPG algorithm is an off-policy RL approach designed specifically for environments with continuous action spaces. Originally proposed by Lillicrap et al. [Reference Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver and Wierstra15], DDPG integrates principles from Deep Q-Networks (DQN) and actor-critic architectures. It adopts a deterministic policy gradient method to avoid discretising the action space, which can otherwise become intractably large and slow to converge. The algorithm operates within an actor-critic framework: the actor learns a parameterised policy function, denoted by $\theta $ , that maps states to optimal actions (see Equation (31)), while the critic assesses the quality of those actions using temporal-difference (TD) learning. Its schematic flow is shown in the Fig. 6.

(31) \begin{align}{\pi _\theta }\left({s,a} \right) = P[a|s,\theta ]\end{align}

The agent employs two neural networks:

1. 1. Actor network ( $\mu (s \, | \, {\theta ^\mu })$ ): Determines the best action $a$ for a given state $s$ .

2. 2. Critic network ( $Q(s,a \, |\,{\theta ^Q})$ ): Evaluates the action $a$ given the state $s$ .

The algorithm uses a target network to improve stability during training. The deterministic target policy can be described as $\mu \, :\, S \to A$ . These target networks are slowly updated. Equation (32) shows the loss function for the critic network while Equation (33) represents the update for the actor network.

Critic network update:

(32) \begin{align}L({{\theta ^Q}} ) = {E_{{s_t},{a_t},{r_t},{s_{t + 1}}}}\left[ {{{\left({{r_t} + \gamma Q{^{\prime}}({s_{t + 1}},\mu {^{\prime}}({s_{t + 1}} \, | \, {\theta ^{\mu {^{\prime}}}})\, | \, {\theta ^{Q{^{\prime}}}}) - Q({s_t},{a_t} \, | \, {\theta ^Q})} \right)}^2}} \right]\end{align}

where $Q{^{\prime}}$ and $\mu {^{\prime}}$ are the target networks for the critic and actor, respectively.

Actor network update:

(33) \begin{align}{\nabla _{{\theta ^\mu }}}J \approx {E_{{s_t}}}\left[ {{\nabla _a}Q(s,a\, | \,{\theta ^Q}){|_{a = \mu (s\, | \, {\theta ^\mu })}}{\nabla _{{\theta ^\mu }}}\mu (s \, | \,{\theta ^\mu })} \right]\end{align}

DDPG is particularly suited for DS because it can directly handle continuous control variables, such as thrust levels or flight path angles, which are essential in energy-efficient trajectory shaping. In this framework, exploration of the state space s(t) is facilitated by adding noise (Ornstein-Uhlenbeck (OU) process) [Reference Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver and Wierstra15, Reference Uhlenbeck and Ornstein61] to actions during training. This type of noise is temporally correlated, making it well suited for continuous control tasks where smoother action variations are beneficial. During early training phases, when the learned policy is still immature, OU noise helps the agent explore state space more thoroughly. As learning progresses, the intensity of noise is gradually decreased to promote more stable and deterministic action selection. The parameters of OU noise process are typically tuned based on system dynamics and computational limitations (Table 4).

Table 4.

Parameters of the Ornstein-Uhlenbeck (OU) noise process

Figure 6.

Schematic representation of the DDPG framework, illustrating the interactions between actor and critic networks, replay buffer, target networks and environment.

4.4 Reinforcement learning architectures and design choices

DDPG operates as an off-policy algorithm and is known for its efficiency in handling high-dimensional, continuous action spaces. PPO, an on-policy algorithm, introduces a clipped objective function to restrict large policy updates and improve training stability, while maintaining simplicity and good performance. TRPO also ensures stable policy updates by imposing a constraint on the Kullback–Leibler divergence between successive policies, making it more computationally intensive but theoretically grounded. These algorithms, when applied to the DS task, enable the UAV to learn energy-efficient manoeuvres in the presence of wind variations by optimising its trajectory and control actions in a data-driven manner.

In this work, the three above-mentioned RL algorithms are employed for the control and trajectory optimisation of a UAV performing DS. All of these algorithms are well suited for continuous control tasks and utilise actor–critic architectures. For the hidden layers of both actor and critic networks, a nonlinear activation function – ReLU – has been used to capture complex relationships within flight dynamics. For the actor output layers, a Tanh activation is used to constrain the action space within physically realisable limits, which is essential in real-world UAV applications (Table 5).

Table 5.

Activation functions used in each algorithm

4.4.1 Hyperparameters and neural network settings

To train the RL algorithm, neural network learning rates, discount factor, batch size, capacity of experience buffer and other parameters are considered. The key settings and values of hyperparameters are given in Table 6. The reward discount factor $\gamma {^{\prime}}$ must be closer to one to emphasise on long-term rewards. For this research, training is based on continuous trajectory control, hence its value is set at 0.99. For DDPG, the experience generated by the interaction of agent in the environment is stored in an experience buffer whose capacity is $1 \times {10^6}$ . During the training process of RL algorithms, the batch size determines how many experience samples are used in each policy update. For DDPG, the batch size refers to the number of samples randomly drawn from the experience replay buffer for each iteration. In this work, a batch size of 64 is used for DDPG. In contrast, for PPO and TRPO, experience is collected using the current policy at each iteration. For PPO, the batch size typically refers to the number of samples gathered per roll-out, and this batch is further divided into mini batches for multiple epochs of gradient updates. A batch size of 128 is adopted for PPO in this study. TRPO, however, uses the entire collected batch for a single constrained optimisation step without splitting into mini-batches. Similarly, a batch size of 128 is used for TRPO, referring to the number of environment steps collected before each update.

Table 6.

Key settings and hyperparameters for each RL algorithm

In this study, all three RL algorithms utilise fully connected feed-forward neural networks to approximate policy and/or value functions. Each network is composed of three hidden layers, with each layer consisting of 256 neurons. The same architecture is adopted across all algorithms to maintain consistency in comparison. This structure provides a sufficient level of representational capacity while keeping the computational complexity manageable. The ReLU activation function is applied between hidden layers to introduce non-linearity and improve learning efficiency. The actor network architecture is shown in Fig. 7.

The learning rate is a crucial hyperparameter in RL algorithms, determining the step size at which the neural network weights are updated during training. A well-chosen learning rate ensures stable convergence and effective policy improvement, while values that are too high may cause divergence and instability. In this study, distinct learning rates are used for each algorithm based on empirical tuning. The learning rates of each network ranges from $1 \times {10^{ - 5}}$ to $1 \times {10^{ - 3}}$ to maintain training stability and prevent overshooting during optimisation.

The GAE method is used in both PPO and TRPO to reduce the variance in the critic’s output. It introduces a hyperparameter, ( ${\lambda _{GAE}}$ ), which balances the trade-off between bias and variance in advantage estimation. To enhance training stability, DDPG leverages a replay buffer and target networks. In contrast, TRPO enforces policy stability by incorporating the Kullback–Leibler (KL) divergence to restrict updates within a trust region. PPO approximates this idea by using a clipped surrogate objective, which limits the deviation between successive policies.

It is important to note that the performance of these algorithms is highly sensitive to the tuning of their hyperparameters. However, currently there is no standardised methodology for selecting these values. As a result, the hyperparameters in this study were fine-tuned through trial-and-error process to effectively address the trajectory optimisation problem under consideration.

5.1 Simulation environment

The DS simulations for the tricopter UAV were implemented in MATLAB and Simulink. The UAV was modeled as a 6-DOF system, incorporating aerodynamic characteristics, environmental wind profiles and thrust dynamics. DRL algorithms, namely DDPG, PPO and TRPO, were deployed within this environment to optimise energy harvesting and trajectory control during soaring. Simulink provided a modular framework for integrating the UAV dynamics with control strategies, while MATLAB was employed for post-processing, data analysis and visualisation of performance metrics such as thrust, energy consumption, velocity and trajectory. This integrated setup ensured a realistic and flexible platform for evaluating the effectiveness of different algorithms in achieving sustained DS.