1. Introduction
Active flow control (AFC) is a technique that leverages actuators to manipulate the external flow field in a way that benefits the overall aerodynamic performance of a target (Gad-el Hak Reference Gad-el Hak2000; Cattafesta III & Sheplak Reference Cattafesta and Sheplak2011). From the perspective of control theory, the essential goal of AFC is to find an optimal control law that maximises the value of a target function (Brunton, Noack & Koumoutsakos Reference Brunton, Noack and Koumoutsakos2020). Traditional model-based control strategies, although promising, rely on reduced-order modelling of the flow and can only be applied to simple geometries such as the bluff body flow and cylinder flow (Pastoor et al. Reference Pastoor, Henning, Noack, King and Tadmor2008). For closed-loop control of complex flows, latest data-driven machine learning approaches represented by deep reinforcement learning (DRL), attract significant attention from the community (Vignon, Rabault & Vinuesa Reference Vignon, Rabault and Vinuesa2023). In figure 1, relevant studies are plotted in a map spanned by the free-stream velocity (
$U_\infty$
) and Reynolds number (
$Re_\infty$
). Note that, for numerical studies where only the Reynolds number is reported, the free-stream velocity is derived by assuming a reference length scale of
$L_{ref}=0.1$
m. Namely,
$U_\infty =Re_\infty \nu /L_{ref}$
, where
$\nu$
denotes the kinematic viscosity. This assumption of the length scale is reasonable for most of the laboratory models.
Summary of recent DRL control studies. The green zone indicates the parameter range relevant to aerospace engineering (AE).Here,
$F_f$
denotes the characteristics frequency of the flowand
$\circ$
,
$\triangle$
and
$\square$
represent 2-D numerical studies, 3-D numerical studies and experimental studies, respectively.
Rabault et al. (Reference Rabault, Kuchta, Jensen, Réglade and Cerardi2019), using a two-dimensional (2-D) low-Reynolds-number numerical environment, demonstrated for the first time that a DRL agent could learn an effective vortex-valley stabilisation strategy in cylinder flow, leading to a drag reduction of DR = 8 % at a free-stream Reynolds number of
$Re_\infty$
= 100. Subsequently, this numerical framework was referenced by other groups, and extensive efforts were made to examine the impacts of hyper-parameters (Rabault & Kuhnle Reference Rabault and Kuhnle2019; Sonoda et al. Reference Sonoda, Liu, Itoh and Hasegawa2023), accelerate the training speed of the agent (Wang et al. Reference Wang, Hua, Aubry, Chen, Wu and Cui2022; Deng, Hu & Chen Reference Deng, Hu and Chen2024; Suárez et al. Reference Suárez, Alcantara-Avila, Miró, Rabault, Font, Lehmkuhl and Vinuesa2024), improve the robustness of learnt control strategies (Ren, Rabault & Tang Reference Ren, Rabault and Tang2021; Tang et al. Reference Tang, Rabault, Kuhnle, Wang and Wang2020; Jia & Xu Reference Jia and Xu2024) and optimise the sensor placement (Paris, Beneddine & Dandois Reference Paris, Beneddine and Dandois2021; Wang et al. Reference Wang, Yan, Hu, Chen, Rabault and Noack2024). Although valuable instructions are obtained by these low-Reynolds-number studies (mostly two-dimensional at
$Re_\infty \lt 1000$
), practical implementations of DRL in experiments remain challenging, particularly for high-speed high-Reynolds-number aeronautical flows. This difficulty mainly arises from the critical time requirement imposed to the control law execution and neural-network training. Specifically, to couple with the flow dynamics, the control should be performed on the characteristic scale of the dominant flow structures. Under this assumption, the control loop frequency (denoted as
$F_c$
) should be at least several times higher than the characteristic frequency of the flow (denoted as
$F_f$
) (Rabault & Kuhnle Reference Rabault and Kuhnle2019). Namely
\begin{equation} \displaystyle F_c\geqslant F_f=\frac {U_\infty }{L_{ref}}=\frac {U_\infty ^2}{Re_\infty \nu }. \end{equation}
For aerospace engineering (AE) applications where
$U_\infty$
and
$Re_\infty$
range typically in 10–1000 m s–1 and
$10^6{-}10^8$
, the characteristic frequency of the flow is estimated to be 100 Hz–10 kHz. This, in combination with (1.1), indicates that the control frequency in an experimental DRL system should at least exceed 1 kHz to be truly meaningful. However, the reality is that, almost all of the existing experimental DRL studies (Fan et al. Reference Fan, Yang, Wang, Triantafyllou and Karniadakis2020; Shimomura et al. Reference Shimomura, Sekimoto, Oyama, Fujii and Nishida2020; Amico, Cafiero & Iuso Reference Amico, Cafiero and Iuso2022; Dong et al. Reference Dong, Hong, Deng, Zhong and Hu2023) used CPU as the main controller, taking charge of both the execution and training of the neural networks. In such a framework, the sensor signals and the control commands are stored in the buffer of a data acquisition card, before transmitting to RAM and being processed by CPU. The corresponding time delay caused by this buffer mechanism can easily reach O(10 ms), which practically limits the maximum control frequency to below 100 Hz, far away from AE applications. Actually, in Fan et al. (Reference Fan, Yang, Wang, Triantafyllou and Karniadakis2020) and Shimomura et al. (Reference Shimomura, Sekimoto, Oyama, Fujii and Nishida2020),
$F_c$
is only 10 Hz. Apart from the above-mentioned real-time execution challenge, in supersonic/hypersonic experiments, timely training and fast convergence of the neural network are also a crucial concern, since most of these wind tunnels operate in transient mode with a typical test time of
$\mathit {O}$
(10 s).
Driven by the above need, a novel high-speed experimental DRL control framework is proposed, using field-programmable gate array (FPGA) as the main controller. The maximum control frequency of this framework reaches
$\mathit {O}$
(1–10 kHz), two orders higher than conventional CPU-based implementation. The feasibility of this framework is tested in a supersonic backward-facing step (BFS) flow, which is rather challenging since the entire DRL process has to be completed in one run of the wind tunnel (10 s). The paper is organised as follows. Section 2 details the control methodology and framework. In § 3, the experimental set-up, including the flow environment and measurement systems, is introduced. In § 4, the DRL learning curves and high-speed schlieren results are presented. Major findings are summarised in § 5.
2. High-speed experimental DRL control framework
Based on the review inthe previous section, it becomes clear that, to realise real-time closed-loop control in high-speed experiments, the key is to cut down the control period. In DRL, the minimum control period a hardware system can implement reads as
\begin{equation} T_c=T_{acq}+T_{nn}+T_{out}+T_{com},\end{equation}
where
$T_{acq}$
,
$T_{nn}$
,
$T_{out}$
and
$T_{com}$
represent, respectively, the time delays caused by data acquisition, neural-network computation, control command output and necessary hardware communication.
To minimise each of the four terms in § 2.1, an FPGA-based experimental DRL (FeDRL) framework is proposed, as shown in § 2. Compared with the traditional CPU-based DRL framework (Fan et al. Reference Fan, Yang, Wang, Triantafyllou and Karniadakis2020; Shimomura et al. Reference Shimomura, Sekimoto, Oyama, Fujii and Nishida2020), three modifications are made. First, the execution and the training processes of the neural network are effectively split, finished respectively by a time-critical FPGA chip and a high-performance CPU chip. Second, the original fully connected neural network (FCNN) used to represent the policy and the Q-function, is substituted by a radial basis function (RBF) network. For universal approximation, the RBF network requires only one hidden layer, thus running and training much faster than a conventional FCNN. Third, the data acquisition (DAQ) devices including an analogue-to-digital converter (ADC) and a digital-to-analogue converter (DAC) necessary for reading the sensor signal and outputting the command signal are hooked directly on the FPGA chip. Thus, no buffer mechanism is required, and the communication time delay (
$T_{com}$
) is eliminated. As a result, the highest control frequency (minimum control period) that can be reached by this novel framework equals to
$F_{c,max}=1/(T_{acq}+T_{nn}+T_{out})$
. Using high-end FPGA chips and DAQ modules, we can easily push
$F_{c,max}$
to O(1–10 kHz).
High-speed experimental DRL framework powered by FPGA and RBF network.
To illustrate the basic workflow of this framework, a classical value-based DRL algorithm, namely the deep Q-network (DQN), is implemented. As show in figure 2, two loops are configured in the FeDRL framework: a real-time control loop with a period of O(10–100
$\unicode{x03BC}$
s), and a slow network training loop running once every hundreds of ms. In each control period, the sensor signal, namely the instantaneous flow state (
$s_t$
), is first sampled by an ADC and then inserted into a first-in first-out (FIFO) sequence (denoted as S-FIFO, “S” for state). This sequence (length:
$l$
), including the current state and its time histories, will be used as the input of the RBF network, namely
$\mathbf {x}=[s_t,s_{t-1},s_{t-2},\ldots,s_{t-l+1}]$
. Previous DRL control studies have shown that, for situations where the observation of a flow system is sparse in space, such an augmentation of the flow states can help the agent find better control strategies, close to that in full observation (Pino et al. Reference Pino, Schena, Rabault and Mendez2023; Wang et al. Reference Wang, Yan, Hu, Chen, Rabault and Noack2024). The outputs of the network are essentially the Q-values of different actions. In figure 2, a simplest case is shown, with only two actions indicating the ON/OFF status of the actuator. Mathematically, the Q-functions represented by the network are as follows:
\begin{equation} \left \lbrace \begin{array}{l}{\mathbf Q}_{\boldsymbol {2\times 1}}= \mathbf {W}_{\boldsymbol{2\times} \mathbf{m}}\cdot \left [ \phi _1({\mathbf x}),\phi _2({\mathbf x}),\ldots,\phi _m({\mathbf x})\right]^{{\mathbf T}}+ {\mathbf {B}_{\boldsymbol{2\times 1}}}\\[5pt]\phi _i={\mathrm {exp}}(-h_i\left |\left | {\mathbf x}-{\mathbf {c}_{\mathbf{i}}}\right |\right |^2),\qquad {\mathrm {for}}\ i=[1,2,\ldots,m] \end{array}\right. \end{equation}
where
$\phi _i$
is the basis function, chosen as a Gaussian kernel. The centre and variance of
$\phi _i$
are denoted as
$\mathbf {c}_{\mathbf {i}}$
and
$h_i$
, respectively. In matrix form,
$\mathbf {C}_{\mathbf{l}\boldsymbol{\times} \mathbf{m}}=[\mathbf {c}_{\boldsymbol 1}^{\mathrm {T}},\mathbf {c}_{\boldsymbol 2}^{\mathrm {T}},\ldots,\mathbf {c}_{\mathbf m}^{\mathrm {T}}]^{\mathrm {T}}$
and
$\mathbf {H}_{\mathbf {m}\boldsymbol{\times 1}}=[h_1,h_2,\ldots,h_m]$
. The weights connecting the hidden layer to the output layer and the biases of the output layer are presented by
$\mathbf {W}_{\boldsymbol{2\times} \mathbf{m}}$
and
$\mathbf {B}_{\mathbf{2} \boldsymbol{\times} \mathbf{1}}$
, respectively.
Using an epsilon-greedy strategy, an action
$a_t$
can be selected from the Q-function, which is then converted into the driving waveform of the actuator via an DAC and a power supply. For training purposes, the experiences collected in each control period are organised as a tuple
$(s_t,a_t,s_{t+1})$
and enqueued into an experience-FIFO (abbreviated as E-FIFO, length:
$n$
). In the training loop, a Labview program is set up, which takes advantage of its NI-FPGA module, acting as a bridge between the controller (FPGA code) and the trainer (Python code). To avoid frequent data requests, the experiences stored in the E-FIFO are read by the Labview program in batch mode. After that, the rewards (
$r_t$
) are assigned and appended into the experience tuple as an extra column. These experiences are transferred to the Python program via a technical data management streaming (TDMS) file, based on which the prediction losses are computed, and the RBF network together with its target is updated. To keep the control law up to date, the weights of the updated network are first sent back to the Labview program via another TDMS file, and then written to the coefficient-FIFO (C-FIFO) of the FPGA.
3. Practical implementation in supersonic BFS flow
3.1. Experimental set-up
(
$a$
) Three-dimensional view of the BFS model. (
$b$
) Cross-sectional view in the
$xy$
-plane. (
$c$
) Structure of the ceramic block.
The supersonic BFS flow, which can be found in the flame holder of scramjets and the tail of rockets, is chosen to test the FeDRL framework, with an aim of mixing enhancement in the shear layer. Experiments are conducted in the supersonic wind tunnel at the Air Force Engineering University. It is a transient suction-type wind tunnel, driven by a 200
${\mathrm m^3}$
vacuum chamber. The free-stream wind speed is Mach 2 (
$U_\infty$
= 510 m s–1), and the maximum test time in one run is approximately 10 s. This wind tunnel has a square test section (200 mm
$\times$
200 mm), providing optical access from the top and side windows. As shown in figure 3, the BFS model is made of a flat plate (material: Polyetheretherketone) with a sharp leading edge. The plate width, thickness and step height (denoted as
$H$
) are 80 mm, 15 mm and 10 mm, respectively. The distance between the step corner and the leading edge is 190 mm. The Reynolds number based on the free-stream velocity and plate length reaches
$Re_\infty =2\times 10^6$
. A coordinate system is established at the middle of the step corner, with x, y and z along the streamwise, wall-normal and spanwise directions, respectively. A hot-wire probe is clamped in the far downstream of the step, acting as the sensor and measuring the velocity fluctuations in the shear layer. This probe is specially designed for supersonic flow, with a wedge front and a stiff steel body. The diameter of the wire is 6.5
$\unicode{x03BC}$
m, corresponding to a frequency response of more than 10 kHz. The hot-wire probe is connected to a constant-temperature anemometry sensor (CTA-4, Hanghua Technology). The operating temperature of the wire (
$T_{wire}$
) is set as 410 K, which translates to an overheat ratio of 2.5 in the cold supersonic flow.
To disturb the supersonic BFS, a spanwise array of five plasma synthetic jet actuators (PSJAs) are installed in front of the step corner, as show in figure 3(
$b$
). This actuator is selected due to the unique combination of high jet velocity (
$O$
(100 m s–1)) and high frequency (
$\gt$
5 kHz). Unlike previous designs, where independent shells and cavities are created for each actuator (Zong & Kotsonis Reference Zong and Kotsonis2017; Zong et al. Reference Zong, Wu, Liang, Su and Li2024), an integrated ceramic block is designed in this study which hosts five pairs of tungsten needles and a slot internal cavity shared by all the actuators (see figure 3
$c$
). This ceramic block has a L-shape cross-section, embedded into the flat plate from the bottom. The corresponding jet orifices are created on the top surface of the flat plate. The jet orifice diameter (
$D$
), spanwise spacing and distance to the step corner are 3 mm, 15 mm and 8 mm, respectively. The width, height and length of the internal cavity are 4 mm, 5 mm and 66 mm, respectively, resulting in total cavity volume of
$V_{ca}=$
1320
$\mathrm{mm}^3$
.
(
$a$
) The sequential discharge circuit used to feed the PSJA array. (
$b$
) Sketch of the three test stages in one run of the wind tunnel.
As shown in figure 4(
$a$
), the sequential discharge circuit developed by Zong & Kotsonis (Reference Zong and Kotsonis2017) is used to feed the PSJA array. All the plasma actuators are connected in serial, such that the same discharge current is shared. In each period, a high-voltage trigger pulse is first produced by a nano second (NS) power supply to create air breakdown in between the electrodes. Then, the electrical energy stored in the capacitor C1 is rapidly deposited into the actuator cavity, by virtue of arc heating. R1 (500
$\Omega$
) is used to limit the charging current of the DC power supply, and the capacitance and DC voltage are set as
$C_0=$
0.1
$\unicode{x03BC}$
F and
$U_0=$
1 kV, respectively. Two high-voltage diodes (D1 and D2) are added for isolation purposes. Different from the original design in Zong & Kotsonis (Reference Zong and Kotsonis2017), a high-voltage electronic switch (Q1) is inserted between R1 and D1. During the capacitive discharge stage, Q1 is switched off to prevent current directly flowing from the DC power supply to the PSJA array. Based on the relevant electrical and geometrical parameters, the dimensionless energy deposition (
$\lambda$
), defined as the ratio of capacitor energy (
$E_c$
) to the total enthalpy of the cavity gas (
$E_g$
), can be calculated
\begin{equation} \lambda =\frac {E_c}{E_g}=\frac {0.5 C_0 U_0^2}{c_p\rho _0V_{ca}T_0}, \end{equation}
where
$c_p$
is the constant-pressure specific heat capacity;
$\rho _0$
and
$T_0$
denote the static density (0.23 kg m
$^-{^3}$
) and temperature (163 K) in the test section, respectively. Consequently,
$\lambda$
is determined as 10, twice higher than that in Zong & Kotsonis (Reference Zong and Kotsonis2018). According to Zong et al. (Reference Zong, Chiatto, Kotsonis and De Luca2018), the maximum jet velocity (
$U_p$
) scales with the cubic root of
$\lambda$
. Therefore, the value of
$U_p$
in the present study is estimated as
$2^{1/3}$
times of that in Zong & Kotsonis (Reference Zong and Kotsonis2018), namely 146 m s–1.
3.2. The DRL settings
Following the framework sketched in figure 2, a closed-loop DRL control system is set up. This system consists of an NI real-time controller which has a built-in FPGA chip (National Instruments, cRIO-9049), a hot-wire sensor and a PSJA array. Two modules are hooked in the chassis of the controller: an analogue input module (NI-9223) for acquiring the hot-wire voltage signal, and an analogue output module (NI-9262) for sending trigger pulses to the actuator power supply. The sampling rate of the ADC device is set as 50 kHz, resulting in a DAQ period of
$T_{acq}=$
20
$\unicode{x03BC}$
s. The depth of the S-FIFO is set as
$l=$
10, which determines the maximum number of input nodes in the RBF network. The PSJA array is controlled in an ON/OFF manner (
$a_t=0$
or 1). If
$a_t=1$
, a trigger pulse with a width of
$5\;\unicode{x03BC}$
s is sent to the high-voltage power supply, which then fires one shot of plasma jet at the flow. Limited by the maximum working frequency of the PSJA (5.2 kHz; see Zong & Kotsonis Reference Zong and Kotsonis2018), the control loop frequency is selected as 5 kHz. Additionally, considering that the input nodes of RBF network (
$\mathrm {l}$
) are no more than 10, the number of hidden neurons is set as
$\mathbf {m}=20$
.
Ideally, we would use the velocity fluctuation in a control period as the instant reward. Practically, due to the compressibility effect, the variations of different flow quantities in the supersonic regime are coupled, and it is impractical to obtain an accurate velocity–voltage calibration curve as that in incompressible subsonic flow. Therefore, we directly use the standard deviation of the normalised voltage signal (
$\bar {s}_t$
) within the subsequent
$l$
steps as a metric to surrogate the instant reward, namely
\begin{equation} \left \lbrace \begin{array}{l} \displaystyle r_t=\sqrt {\frac {1}{l}\sum _{i=1}^{i=l}\bar {s}_{t+i}^2}-1 \\ \bar {s}_t=(s_t-\mu _0)/\sigma _0,\end{array} \right. \end{equation}
where the mean and standard deviation of the undisturbed voltage signal (
$\mu _0$
and
$\sigma _0$
) measured in the baseline condition are used for normalisation. By definition, in the first expression, the first term on the right-hand side has an expectation of 1 when no plasma actuation is applied. Therefore, by subtracting 1 from the expression, a positive (negative) reward would correspond to a increase (decrease) in the measured flow fluctuation.
The Keras API is employed to train the network, and the training loop frequency is 2 Hz. Namely, for every 500 ms, the 2500 fresh experiences collected by the FPGA are piled into a replay buffer (size: 10 000), based on which the RBF network is trained approximately for two episodes. Since we use the classical DQN algorithm, the Bellman equation is used iteratively to update the Q-function. The loss function (
$L$
) is defined as follows (Mnih et al. Reference Mnih2015):
\begin{equation} L(\boldsymbol {\theta })=\mathbb {E}\left [\left (r_t+\gamma \cdot \mathop {\mathrm {max}}_{a_{t+1}^\prime }\mathbf {Q}^{\ast }(\mathbf {x}_{t+1},a_{t+1}^\prime)-\mathbf {Q}(\mathbf {x}_t,a_t)\right)^2\right],\end{equation}
where
$\gamma$
is the discount factor (0.95), influencing the horizon of the agent;
$\theta$
represents the trainable parameters in the RBF network, including the centre matrix
$\mathbf {C}$
, variance matrix
$\mathbf {H}$
, weight matrix
$\mathbf {W}$
and bias matrix
$\mathbf {B}$
. On the right-hand side of (3.3), the sum of the first two terms defines the expected Q-value, while the last term gives the predicated Q-value. Obviously, the expectation of their squared difference is essentially the loss. Note that a separate target network (
$\mathbf{Q}^{\ast }$
) is used to calculate the expected Q-value, and the parameters of the original Q-network are copied to the target Q-network once every five training episodes.
In each training episode, a mini-batch of 1000 experiences is sampled randomly from the replay buffer to compute the loss, and the network parameters are updated according to the following equation:
\begin{equation} \theta ^{i+1}=\theta ^{i}-\alpha \cdot \nabla _{\theta ^i} L(\theta ^i),\end{equation}
where the superscripts ‘i’ and ‘i + 1’ represent the iteration number. Here,
$\alpha$
denotes the learning rate, which is initialised as 0.001 and adjusted dynamically via the Adam optimiser during the course of training (Kingma Reference Kingma2014).
To achieve balanced exploration and exploitation in reinforcement learning (RL), the probability of choosing to explore in the epsilon-greedy strategy, namely the value of
$\varepsilon$
, is assumed as a piecewise function of time (see (3.5)). As sketched in figure 4(
$b$
), in each run of the wind tunnel, the complete operation time (10 s) is divided into three stages. The first stage (
$t\leqslant 0.5 \mathrm {\;s}$
) corresponds to the baseline test without plasma actuation (
$\varepsilon =0$
,
$a_t=0$
). The second stage (
$0.5\mathrm {\;s}\lt t\lt 9 \mathrm {\;s}$
) denotes the training stage with the exploration probability decreasing linearly from 0.5 to 0. The last stage is the evaluation stage (
$t\geqslant 9 \mathrm {\;s}$
) where the final control strategy is executed deterministically without exploration, namely
$a_t=\mathrm {argmax}\{\mathbf {Q}(\mathbf {x}_{\mathbf t},a_t)\}$
. The authors have compared the statistics of learnt strategies computed from this post-training evaluation scheme and that from an independent evaluation run, and marginal differences can be observed. This post-training evaluation scheme allows us to obtain the baseline and actuated statistics simultaneously in one run, eliminating the possible measurement error caused by the drift of wind tunnel operating condition (e.g. ambient temperature).
\begin{equation} \varepsilon = \left \lbrace \begin{array}{ll} 0, & t\leqslant 0.5 \mathrm {\;s}\\ 0.5\times (1-(t-0.5)/8.5), & 0.5\mathrm {\;s}\lt t\lt 9 \mathrm {\;s}\\ 0, & t\geqslant 9 \mathrm {\;s}.\\ \end{array} \right. \end{equation}
4. Results and discussion
4.1. Velocity statistics
In total, three cases with different state dimensionalities and initialisation strategies are selected to perform DRL control, as listed in table 1. In case 1, the input vector of the Q-network is set as
$\mathbf {x}=[s_t,s_{t-1},\ldots,s_{t-9}]$
. The Gaussian kernel centre matrix
$\mathbf {C}_{\mathbf{l}\boldsymbol{\times}\mathbf{m}}$
and the weight matrix
$\mathbf {W}_{\boldsymbol{2\times} \mathbf{m}}$
are initialised by a random uniform initialiser (range: [0, 1]) and an Xavier uniform initialiser in the Keras API, respectively. The variance matrix
$\mathbf {H}_{\mathbf{m}\boldsymbol{\times 1}}$
and the bias matrix
$\mathbf {B}_{\boldsymbol{2\times 1}}$
are directly assigned as 1 and 0, respectively. Case 2 shares the same initialisation strategy as case 1. Nevertheless, the network input is truncated to
$\mathbf {x}=[s_t,s_{t-1}]$
, for the purpose of comparing the impact of state dimensionality. For case 3, the same input vector as case 1 is used. Inspired by Maceda et al. (Reference Maceda, Varon, Lusseyran and Noack2023), where the feedback control law in cavity flow can be well interpreted by a cluster-based visualisation, we decided to initialise the Gaussian kernel centre matrix
$\mathbf {C}_{\mathbf{l}\boldsymbol{\times}\mathbf{m}}$
with the K-means clustering method. Specifically, the input state data acquired under the baseline condition (namely the 2500 samples of
$\mathbf {x}$
acquired at t
$\leqslant$
0.5 s) are divided into
$m$
clusters (
$m$
= 20), resulting in a total of
$m$
cluster centroids. Since each centroid is a vector consisting of
$l$
elements (
$l$
= 10); we have
$l\times m$
elements, ready to be assigned to the Gaussian kernel centre matrix
$\mathbf {C}_{\mathbf{l}\boldsymbol{\times} \mathbf{m}}$
in one-to-one correspondence. By doing so, we hope that the Q-network can leverage some of the existing knowledge in baseline flow and converge faster in training. Other parameters in case 3, including the variance matrix
$\mathbf {H}_{\mathbf{m}\boldsymbol{\times 1}}$
, the weight matrix
$\mathbf {W}_{\boldsymbol{2\times} \mathbf{m}}$
and the bias matrix
$\mathbf {B}_{\boldsymbol{2\times 1}}$
, are initialised with the same strategy as that in case 1.
List of test cases and initialisation methods for different parameters.
Figure 5(
$a$
) shows the time evolution of the low-pass filtered reward (dented as
$\overline {r_t}$
, filter cutoff frequency: 50 Hz), together with the history of network loss in different cases. Regardless of the initialisation strategy and state dimensionality, cases 1–3 show a similar increasing trend of the reward. The most noteworthy feature is that the DQN agent is so efficient that, even with only ten seconds of learning, the filtered reward, signifying the relative increase of velocity fluctuation, is increased from 0 to a peak value of 0.2. The number of training episodes spent in this process is one order less than the policy-based DRL algorithm (Rabault et al. Reference Rabault, Kuchta, Jensen, Réglade and Cerardi2019). In the range 0.5 s
$\leqslant t\leqslant$
1 s, the DQN agent has a high exploration probability of close to 0.5, and random actuation is performed half of the time. Consequently, little improvement of the instant reward can be picked when compared with the baseline stage of
$t\leqslant$
0.5 s, indicating that random actuation is inefficient in promoting the mixing. During the course of learning, the network loss varies approximately inversely with the reward. Specifically, in the initial stage of
$t\leqslant 1.5$
s, a sharp drop of the training loss is observed in cases 1–2, followed by a slow decay afterwards. Comparing case 3 with case 1, the cluster-based initialisation method, for which the authors have high expectations, exhibits a slower decay rate of the training loss than the random initialisation method, possibly due to the different distributions of the states in baseline and actuated conditions.
(
$a$
) Time evolution of the filtered rewards (
$\overline {r_t}$
, solid lines) and the episodic variation of the network loss (dashed lines). (
$b$
) Variation of the activation ratio. (
$c$
) Power spectral densities of the hot-wire anemometry (HWA) voltage. (
$d$
) Classification of the states in case 2 based on the action.
The activation ratio of the plasma actuator (
$\beta$
), defined as the occurrence probability of
$a_t=1$
in a short period of time, is plotted in figure 5(
$b$
). In the baseline test stage of
$t\lt 0.5$
s, the activation ratio is kept unchanged at zero. When stepping into the training stage (
$t=0.5$
s), different activation ratios ranging from 0.2 to 0.4 are exhibited in the three cases. This is reasonable, since this initial value of
$\beta$
depends on both the exploration probability (
$\varepsilon$
) and the initial parameters of the Q-network. As sketched in figure 4(
$b$
),
$\varepsilon$
is initialised as 0.5. Therefore, random control is applied only for half of the time. During the other half, deterministic action is chosen according to the blind prediction of the Q-network. Consequently, the initial activation ratio can vary between 0.25 and 0.75. For the three cases tested in this study, even though the same seed is used by the pseudo-random generator to initialise the Q-network, the initial activation ratio still differs due to the different network input dimensionality and initialisation strategy (see table 1). In the first half-course of learning (
$t\lt 5$
s), the agent collects experience by shooting plasma randomly, and the activation ratio goes down due to the decreasing exploration probability. By contrast, in the last half, the agent has gained sufficient experience, and starts to taste the fruit of deterministic actuation, leading to a fast growth in the activation ratio. The average values of
$\beta$
in the evaluation stage are 0.72, 0.57 and 0.42, respectively, for cases 1–3, corresponding to an equivalent discharge frequency of 3.6 kHz, 2.9 kHz and 2.1 kHz, respectively.
Based on the velocity signals collected in the evaluation stage (
$9\;\mathrm {s}\leqslant t\leqslant 10\;\mathrm {s}$
), the power spectra are computed, as shown in figure 5(
$c$
). For all of the cases, a sharp peak can be found at
$f$
= 7 kHz. Compared with the baseline, DRL control is able to lift the power spectrum as a whole, and the relative increases of the velocity fluctuation in cases 1–3 are 15.6 %, 21.2 % and 14.4 %, respectively. In the best case (case 2), since we only have two state variables, it is possible to visualise the control strategy with a scatter plot (see figure 5
$d$
). Consequently, the joint distribution of
$s_1$
and
$s_2$
exhibits an elliptical shape. This is reasonable, since the dimensionless velocities measured at two neighbouring time steps are correlated, and should not differ too much from each other. Active states with
$a_t=1$
are mostly found in the third quadrant, while inactive states with
$a_t=0$
tend to be distributed in the first quadrant. Such a control strategy can be further simplified as a 2-D threshold actuation strategy. Physically, since the instantaneous velocity fluctuation measured by the hot-wire is correlated with the flapping motion of the shear layer, the above threshold actuation strategy indicates that plasma synthetic jets should be shot when the instantaneous wall-normal location of the shear layer is higher than the baseline mean location (i.e. large flow separation area behind the step).
4.2. Schlieren imaging
To understand, physically, how plasma synthetic jets modify the supersonic BFS flow, high-speed schlieren imaging is conducted, alongside the training process. The sampling frequency and spatial resolution are set as 50 kHz and 512
$\times$
384 pixels
$^2$
, respectively. More information about the set-up can be found in Kong et al. (Reference Kong, Wu, Zong and Guo2022). Figure 6(
$a$
)–(
$d$
) shows the response of the BFS flow to a single shot of plasma synthetic jet (PSJ), where
$t$
denotes the time delay with respect to discharge ignition. Dominant features, including the expansion fan, shear layer and reattachment shock wave, can be clearly identified. Between
$t=20\;\unicode{x03BC}$
s and
$t=80\;\unicode{x03BC}$
s, two shock waves are issued from the exit orifice (see the black arrows), due to the rapid arc heating within the cavity. While interacting with the supersonic cross-flow, they evolve into a bow shape and propagate to the top-right corner along the trajectory specified by the expansion fan. Theoretically, between the two bow shock waves, a jet flow should be present. However, such a jet flow is not observable in the schlieren image, probably being buried by the turbulent boundary layer. The root of the reattachment shock seems to be swept downstream by plasma jets, and the reattachment length is estimated as
$4H$
based on a linear extrapolation of the reattachment shock.
(a−d) Time response of the BFS flow to a single shot of plasma synthetic jet. From top to bottom,
$t=20\;\unicode{x03BC}$
s,
$40\;\unicode{x03BC}$
s,
$60\;\unicode{x03BC}$
s and
$80\;\unicode{x03BC}$
s. The red dotted line indicates the reattachment shock. (
$e$
) Contour of the grey scale fluctuation amplitude in the baseline condition. ( f−h) Variation of the grey scale fluctuation amplitude caused by plasma actuation in cases 1–3.
The standard deviation of the schlieren grey scale images (denoted as
$I_{std}$
), to some extent, can reflect the density fluctuations in the flow field. Figure 6(
$e$
) shows the grey scale fluctuation amplitude in baseline condition. Strong density fluctuations can be picked in the incoming boundary layer, shear layer and reattachment shock zone, which is largely expected. Figure 6(f−h) further compares the variation of grey scale fluctuation amplitude caused by plasma actuation (denoted as
$\Delta I_{std}$
) in cases 1–3. Note that, to compute the standard deviation in actuated cases, only the schlieren images acquired in the last 0.2 s (10 000 samples) are used, corresponding to the final control law sought by DRL. As a result, the grey scale fluctuation amplitude is elevated noticeably in the vicinity of the exit orifice, due to the formation of a bow shock. With regard to the shear layer zone, although the velocity fluctuation is elevated noticeably by plasma actuation, the grey scale fluctuation amplitude remains almost the same as that in the baseline.
4.3. Strategy comparison
(
$a$
) Power spectral densities of the HWA voltage and (
$b$
) relative increase of the HWA voltage fluctuation at different discharge frequencies. The dashed blue line in (b) indicates the relative increase of voltage fluctuation under the optimal DRL control (case 2).
In previous studies of subsonic BFS flow control, plasma actuators are operated ubiquitously in constant-frequency periodic mode (Benard et al. Reference Benard, Pons-Prats, Periaux, Bugeda, Braud, Bonnet and Moreau2016a
,Reference Benard, Sujar-Garrido, Bonnet and Moreau
b
). By tuning the duty cycle and discharge frequency (denoted as
$F_d$
) of the plasma actuator, satisfying reduction of the reattachment length was obtained at
$U_\infty$
= 15 m s–1 by exciting the unsteady shear layer. As such, it is natural for us to compare this periodical control strategy with DRL control in the supersonic BFS flow. While implementing the constant-frequency periodical control strategy, the same experimental set-up as that described in § 3 is used, except that the RBF network in FPGA is replaced by a pulse generation module and the network training is disabled. In total, five discharge frequencies (
$F_d=$
0.5, 1, 3, 5, 8 kHz) are tested in one wind tunnel run (10 s), and the sample duration for each discharge frequency is one second. Between two consecutive discharge frequencies, the PSJAs are switched off for one second in order to recover the flow to baseline state. Figure 7(
$a$
) shows the power spectral densities at increasing discharge frequency. As evidenced, a majority of the discharge frequencies are able to elevate the peak of the power spectrum and the amplitude of low-frequency fluctuations. The only exception is
$F_d=0.5$
kHz, where the power spectrum peak is reduced noticeably by 13.6 %.
To compare the control effectiveness in different cases, the relative increase of raw voltage fluctuations caused by plasma actuation (denoted as
$\Delta \sigma _u$
) is computed as follows:
\begin{equation} \Delta \sigma _u=(\sigma _{u,AT}-\sigma _{u,BL})/\sigma _{u,BL},\end{equation}
where
$\sigma _{u,BL}$
and
$\sigma _{u,AT}$
denote the standard deviation of the raw voltage signals in baseline and actuated conditions, respectively. As shown in figure 7(
$b$
), with increasing discharge frequency,
$\Delta \sigma _u$
first increases and then drops. The greatest increase of the raw voltage fluctuation is achieved at
$F_d$
= 5 kHz, which coincides with the control update frequency used in DRL. However, quantitatively, the peak of
$\Delta \sigma _u$
found by periodic control (10.5 %) is only half of that earned by the optimal DRL control (21.2 %, case 2). This difference in control effectiveness can also be seen by comparing their power spectrum peaks in figures 5(
$c$
) and 7(
$a$
) (
$3.5\times 10^{-5}$
vs.
$2.9\times 10^{-5}$
). Intuitively, the superior control authority of DRL can be explained from its ability to identify the events when plasma actuation is beneficial.
5. Summary
In this study, an FPGA-based experimental DRL control framework is proposed, capable of achieving a control frequency of
$O$
(1–10 kHz). This high control frequency, together with the fast-response plasma actuators and high-bandwidth hot-wire, enables us to challenge a real-time closed-loop control of the supersonic BFS flow at Mach 2.
Three cases with different state dimensionalities and initialisation strategies are tested, with an aim of maximising the mixing in the shear layer. The training efficiency of the DQN is found to be much higher than the widely adopted policy-based DRL algorithms. With only ten seconds of learning, the filtered reward, signifying the relative increase of the velocity fluctuation, is increased from 0 to a peak value of 0.2. Compared with the baseline case, the power spectra in actuated conditions are elevated as a whole. The final control law in case 2 suggests that plasma actuators should be shot when the state lies in the third quadrant. Physically, we believe that the mixing enhancement in the shear layer is correlated with the two bow shock waves issued from the jet orifice. In future, side-by-side comparisons between the different control strategies, including classical proportional integral derivative control and DRL control, are recommended.
Funding.
This work was sponsored by the National Natural Science Foundation of China (Nos. U2341277, 52441602, 52025064).
Declaration of interests.
The authors report no conflict of interest.
Data availability statement.
This study is part of a project that aims to implement different DRL strategies in FPGA for purpose of real-time active flow control. The codes will be made open-source in GitHub after the project finishes.
