Skip to main content Accessibility help
×
Home
Hostname: page-component-768ffcd9cc-kfj7r Total loading time: 0.449 Render date: 2022-11-30T06:33:54.399Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "useRatesEcommerce": false, "displayNetworkTab": true, "displayNetworkMapGraph": false, "useSa": true } hasContentIssue true

Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control

Published online by Cambridge University Press:  20 February 2019

Jean Rabault*
Affiliation:
Department of Mathematics, University of Oslo, 0316 Oslo, Norway
Miroslav Kuchta
Affiliation:
Department of Mathematics, University of Oslo, 0316 Oslo, Norway
Atle Jensen
Affiliation:
Department of Mathematics, University of Oslo, 0316 Oslo, Norway
Ulysse Réglade
Affiliation:
Department of Mathematics, University of Oslo, 0316 Oslo, Norway CEMEF, Mines ParisTech, 06904 Sophia-Antipolis, France
Nicolas Cerardi
Affiliation:
Department of Mathematics, University of Oslo, 0316 Oslo, Norway CEMEF, Mines ParisTech, 06904 Sophia-Antipolis, France
*
Email address for correspondence: jean.rblt@gmail.com

Abstract

We present the first application of an artificial neural network trained through a deep reinforcement learning agent to perform active flow control. It is shown that, in a two-dimensional simulation of the Kármán vortex street at moderate Reynolds number ($Re=100$), our artificial neural network is able to learn an active control strategy from experimenting with the mass flow rates of two jets on the sides of a cylinder. By interacting with the unsteady wake, the artificial neural network successfully stabilizes the vortex alley and reduces drag by approximately 8 %. This is performed while using small mass flow rates for the actuation, of the order of 0.5 % of the mass flow rate intersecting the cylinder cross-section once a new pseudo-periodic shedding regime is found. This opens the way to a new class of methods for performing active flow control.

Type
JFM Papers
Copyright
© 2019 Cambridge University Press 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Rudlur, M., Lovenberg, J., Monga, R., Moore, S., Steiner, B., Tuchu, P., Vasudejon, V., Warden, P., Wicke, M., Yu, Y. & Zheng, X. 2016 Tensorflow: a system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16), vol. 16, pp. 265283.Google Scholar
Barbagallo, A., Dergham, G., Sipp, D., Schmid, P. J. & Robinet, J.-C. 2012 Closed-loop control of unsteadiness over a rounded backward-facing step. J. Fluid Mech. 703, 326362.10.1017/jfm.2012.223CrossRefGoogle Scholar
Barbagallo, A., Sipp, D. & Schmid, P. J. 2009 Closed-loop control of an open cavity flow using reduced-order models. J. Fluid Mech. 641, 150.10.1017/S0022112009991418CrossRefGoogle Scholar
Brunton, S. L. & Noack, B. R. 2015 Closed-loop turbulence control: progress and challenges. Appl. Mech. Rev. 67 (5), 050801.10.1115/1.4031175CrossRefGoogle Scholar
Davis, T. A. 2004 Algorithm 832: UMFPACK v4.3 – an unsymmetric-pattern multifrontal method. ACM Trans. Math. Softw. 30 (2), 196199.10.1145/992200.992206CrossRefGoogle Scholar
Dean, B. & Bhushan, B. 2010 Shark-skin surfaces for fluid-drag reduction in turbulent flow: a review. Phil. Trans. R. Soc. Lond. A 368 (1929), 47754806.10.1098/rsta.2010.0201CrossRefGoogle ScholarPubMed
Duan, Y., Chen, X., Houthooft, R., Schulman, J. & Abbeel, P. 2016 Benchmarking deep reinforcement learning for continuous control. In International Conference on Machine Learning, pp. 13291338.Google Scholar
Duriez, T., Brunton, S. L. & Noack, B. R. 2016 Machine Learning Control – Taming Nonlinear Dynamics and Turbulence. Springer.Google Scholar
Erdmann, R., Pätzold, A., Engert, M., Peltzer, I. & Nitsche, W. 2011 On active control of laminar–turbulent transition on two-dimensional wings. Phil. Trans. R. Soc. Lond. A 369 (1940), 13821395.10.1098/rsta.2010.0364CrossRefGoogle ScholarPubMed
Fransson, J. H. M., Talamelli, A., Brandt, L. & Cossu, C. 2006 Delaying transition to turbulence by a passive mechanism. Phys. Rev. Lett. 96 (6), 064501.10.1103/PhysRevLett.96.064501CrossRefGoogle ScholarPubMed
Gautier, N., Aider, J.-L., Duriez, T., Noack, B. R., Segond, M. & Abel, M. 2015 Closed-loop separation control using machine learning. J. Fluid Mech. 770, 442457.10.1017/jfm.2015.95CrossRefGoogle Scholar
Geuzaine, C. & Remacle, J.-F. 2009 Gmsh: a 3-D finite element mesh generator with built-in pre- and post-processing facilities. Intl J. Numer. Meth. Engng 79 (11), 13091331.10.1002/nme.2579CrossRefGoogle Scholar
Glezer, A. 2011 Some aspects of aerodynamic flow control using synthetic-jet actuation. Phil. Trans. R. Soc. Lond. A 369 (1940), 14761494.10.1098/rsta.2010.0374CrossRefGoogle ScholarPubMed
Goda, K. 1979 A multistep technique with implicit difference schemes for calculating two- or three-dimensional cavity flows. J. Comput. Phys. 30 (1), 7695.10.1016/0021-9991(79)90088-3CrossRefGoogle Scholar
Goodfellow, I., Bengio, Y., Courville, A. & Bengio, Y. 2016 Deep Learning, vol. 1. MIT Press.Google Scholar
Gu, S., Lillicrap, T., Sutskever, I. & Levine, S. 2016 Continuous deep Q-learning with model-based acceleration. In Intl Conference on Machine Learning, pp. 28292838.Google Scholar
Guéniat, F., Mathelin, L. & Hussaini, M. Y. 2016 A statistical learning strategy for closed-loop control of fluid flows. Theor. Comput. Fluid Dyn. 30 (6), 497510.10.1007/s00162-016-0392-yCrossRefGoogle Scholar
He, K., Zhang, X., Ren, S. & Sun, J. 2016 Deep residual learning for image recognition. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 770778.Google Scholar
Hornik, K., Stinchcombe, M. & White, H. 1989 Multilayer feedforward networks are universal approximators. Neural Networks 2 (5), 359366.10.1016/0893-6080(89)90020-8CrossRefGoogle Scholar
Kober, J., Bagnell, J. A. & Peters, J. 2013 Reinforcement learning in robotics: a survey. Intl J. Robotics Res. 32 (11), 12381274.10.1177/0278364913495721CrossRefGoogle Scholar
Krizhevsky, A., Sutskever, I. & Hinton, G. E. 2012 Imagenet classification with deep convolutional neural networks. Adv. Neural Inform. Proc. Syst. pp. 10971105.Google Scholar
Kutz, J. N. 2017 Deep learning in fluid dynamics. J. Fluid Mech. 814, 14.10.1017/jfm.2016.803CrossRefGoogle Scholar
LeCun, Y., Bengio, Y. & Hinton, G. 2015 Deep learning. Nature 521, 436444.10.1038/nature14539CrossRefGoogle ScholarPubMed
Li, R., Noack, B. R., Cordier, L., Borée, J. & Harambat, F. 2017 Drag reduction of a car model by linear genetic programming control. Exp. Fluids 58 (8), 103.10.1007/s00348-017-2382-2CrossRefGoogle Scholar
Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D. & Wierstra, D.2015 Continuous control with deep reinforcement learning. arXiv:1509.02971.Google Scholar
Logg, A., Mardal, K.-A. & Wells, G. 2012 Automated Solution of Differential Equations by the Finite Element Method: The FEniCS Book, vol. 84. Springer.10.1007/978-3-642-23099-8CrossRefGoogle Scholar
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. & Riedmiller, M.2013 Playing Atari with deep reinforcement learning. arXiv:1312.5602.Google Scholar
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Peterson, S., Beatie, C., Sadih, A., Antonoglou, I., King, H., Rumanron, D., Wiersta, D., Legg, S. & Hassabis, D. 2015 Human-level control through deep reinforcement learning. Nature 518 (7540), 529.10.1038/nature14236CrossRefGoogle ScholarPubMed
Pastoor, M., Henning, L., Noack, B. R., King, R. & Tadmor, G. 2008 Feedback shear layer control for bluff body drag reduction. J. Fluid Mech. 608, 161196.10.1017/S0022112008002073CrossRefGoogle Scholar
Rabault, J., Kolaas, J. & Jensen, A. 2017 Performing particle image velocimetry using artificial neural networks: a proof-of-concept. Meas. Sci. Technol. 28 (12), 125301.10.1088/1361-6501/aa8b87CrossRefGoogle Scholar
Rauber, P. E., Fadel, S. G., Falcao, A. X. & Telea, A. C. 2017 Visualizing the hidden activity of artificial neural networks. IEEE Trans. Vis. Comput. Graphics 23 (1), 101110.10.1109/TVCG.2016.2598838CrossRefGoogle ScholarPubMed
Schaarschmidt, M., Kuhnle, A. & Fricke, K.2017 Tensorforce: a tensorflow library for applied reinforcement learning. https://github.com/tensorforce/tensorforce.Google Scholar
Schäfer, M., Turek, S., Durst, F., Krause, E. & Rannacher, R. 1996 Benchmark computations of laminar flow around a cylinder. In Flow Simulation with High-Performance Computers II (ed. Hirschel, E. H.), pp. 547566. Springer.10.1007/978-3-322-89849-4_39CrossRefGoogle Scholar
Schmidhuber, J. 2015 Deep learning in neural networks: an overview. Neural Networks 61, 85117.10.1016/j.neunet.2014.09.003CrossRefGoogle ScholarPubMed
Schoppa, W. & Hussain, F. 1998 A large-scale control strategy for drag reduction in turbulent boundary layers. Phys. Fluids 10 (5), 10491051.10.1063/1.869789CrossRefGoogle Scholar
Schulman, J., Levine, S., Moritz, P., Jordan, M. I. & Abbeel, P.2015 Trust region policy optimization. CoRR abs/1502.05477, arXiv:1502.05477.Google Scholar
Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O.2017 Proximal policy optimization algorithms. arXiv:1707.06347.Google Scholar
Shahinfar, S., Sattarzadeh, S. S., Fransson, J. H. & Talamelli, A. 2012 Revival of classical vortex generators now for transition delay. Phys. Rev. Lett. 109 (7), 074501.10.1103/PhysRevLett.109.074501CrossRefGoogle ScholarPubMed
Siegelmann, H. T. & Sontag, E. D. 1995 On the computational power of neural nets. J. Comput. Syst. Sci. 50 (1), 132150.10.1006/jcss.1995.1013CrossRefGoogle Scholar
Sipp, D. & Schmid, P. J. 2016 Linear closed-loop control of fluid instabilities and noise-induced perturbations: a review of approaches and tools. Appl. Mech. Rev. 68 (2), 020801.10.1115/1.4033345CrossRefGoogle Scholar
Valen-Sendstad, K., Logg, A., Mardal, K.-A., Narayanan, H. & Mortensen, M. 2012 A comparison of finite element schemes for the incompressible Navier–Stokes equations. In Automated Solution of Differential Equations by the Finite Element Method, pp. 399420. Springer.10.1007/978-3-642-23099-8_21CrossRefGoogle Scholar
Verma, S., Novati, G. & Koumoutsakos, P. 2018 Efficient collective swimming by harnessing vortices through deep reinforcement learning. Proc. Natl Acad. Sci. USA; http://www.pnas.org/content/early/2018/05/16/1800923115.full.pdf.10.1073/pnas.1800923115CrossRefGoogle ScholarPubMed
Vernet, J., Örlü, R., Alfredsson, P. H., Elofsson, P. & Scania, A. B. 2014 Flow separation delay on trucks a-pillars by means of dielectric barrier discharge actuation. In First International Conference in Numerical and Experimental Aerodynamics of Road Vehicles and Trains (Aerovehicles 1), Bordeaux, France, pp. 12.Google Scholar
Wang, Z., Xiao, D., Fang, F., Govindan, R., Pain, C. C. & Guo, Y. 2018 Model identification of reduced order fluid dynamics systems using deep learning. Intl J. Numer. Meth. Fluids 86 (4), 255268.10.1002/fld.4416CrossRefGoogle Scholar
129
Cited by

Save article to Kindle

To save this article to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
Available formats
×

Save article to Dropbox

To save this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Dropbox account. Find out more about saving content to Dropbox.

Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
Available formats
×

Save article to Google Drive

To save this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you used this feature, you will be asked to authorise Cambridge Core to connect with your Google Drive account. Find out more about saving content to Google Drive.

Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
Available formats
×
×

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *