Hostname: page-component-8448b6f56d-dnltx Total loading time: 0 Render date: 2024-04-17T16:55:20.027Z Has data issue: false hasContentIssue false
  • English
  • Français

Implementation of 2D Domain Decomposition in the UCAN Gyrokinetic Particle-in-Cell Code and Resulting Performance of UCAN2

Published online by Cambridge University Press:  15 January 2016

Jean-Noel G. Leboeuf ,
Viktor K. Decyk ,
David E. Newman  and
Raul Sanchez
Jean-Noel G. Leboeuf*
Affiliation:
Department of Physics, University of Alaska, Fairbanks, AK 99775-5920, USA
Viktor K. Decyk
Affiliation:
Department of Physics and Astronomy, and Institute for Digital Research and Education (IDRE), University of California, Los Angeles, CA 90095-1547, USA
David E. Newman
Affiliation:
Department of Physics, University of Alaska, Fairbanks, AK 99775-5920, USA
Raul Sanchez
Affiliation:
Departamento de Fsica, Universidad Carlos III, Leganes 28911, Madrid, Spain
*
*Corresponding author. Email addresses:jnlscientific@hotmail.com (J.-N. G. Leboeuf), decyk@physics.ucla.edu (V. K. Decyk), denewman@alaska.edu (D. E. Newman), rsanchez@fis.uc3m.es (R. Sanchez)
Get access

Abstract

The massively parallel, nonlinear, three-dimensional (3D), toroidal, electrostatic, gyrokinetic, particle-in-cell (PIC), Cartesian geometry UCAN code, with particle ions and adiabatic electrons, has been successfully exercised to identify non-diffusive transport characteristics in present day tokamak discharges. The limitation in applying UCAN to larger scale discharges is the 1D domain decomposition in the toroidal (or z-) direction for massively parallel implementation using MPI which has restricted the calculations to a few hundred ion Larmor radii or gyroradii per plasma minor radius. To exceed these sizes, we have implemented 2D domain decomposition in UCAN with the addition of the y-direction to the processor mix. This has been facilitated by use of relevant components in the P2LIB library of field and particle management routines developed for UCLA's UPIC Framework of conventional PIC codes. The gyro-averaging specific to gyrokinetic codes is simplified by the use of replicated arrays for efficient charge accumulation and force deposition. The 2D domain-decomposed UCAN2 code reproduces the original 1D domain nonlinear results within round-off. Benchmarks of UCAN2 on the Cray XC30 Edison at NERSC demonstrate ideal scaling when problem size is increased along with processor number up to the largest power of 2 available, namely 131,072 processors. These particle weak scaling benchmarks also indicate that the 1 nanosecond per particle per time step and 1 TFlops barriers are easily broken by UCAN2 with 1 billion particles or more and 2000 or more processors.

Keywords

Gyrokinetic simulationparticle-in-cell methodmassive parallelization
Type
Research Article
Information
Communications in Computational Physics , Volume 19 , Issue 1 , January 2016 , pp. 205 - 225
Copyright
Copyright © Global-Science Press 2016 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

[1]Brizard, A. J., and Hahm, T. S., Foundations of nonlinear gyrokinetic theory, Reviews of Modern Physics, 79 (2007), 421–468.CrossRefGoogle Scholar
[2]Dimits, A. M., Bateman, G., Beer, M. A., Cohen, B. I., Dorland, W., Hammett, G. W., Kim, C., Kinsey, J. E., Kotschenreuther, M., Kritz, A. H., Lao, L. L., Mandrekas, J., Nevins, W. M., Parker, S. E., Redd, A. J., Shumaker, D. E., Sydora, R., and Weiland, J., Comparisons and physics basis of tokamak transport models and turbulence simulations, Physics of Plasmas, 7 (2000), 969–983.CrossRefGoogle Scholar
[3]Parker, S. E., Lee, W. W., and Santoro, R. A., Gyrokinetic simulation of ion temperature gradient driven turbulence in 3D toroidal geometry, Physical Review Letters, 71 (1993), 2042–2045.Google Scholar
[4]Sydora, R. D., Toroidal gyrokinetic particle simulations of core fluctuations and transport, Physica Scripta, 52 (1995), 474–480.CrossRefGoogle Scholar
[5]Sydora, R. D., Decyk, V. K., and Dawson, J. M., Fluctuation-induced heat transport results from a large global 3D toroidal particle simulation model, Plasma Physics and Controlled Fusion, 38 (1996), A281–A294.CrossRefGoogle Scholar
[6]Lin, Z., Hahm, T. S., Lee, W. W., Tang, W. M., and White, R. B., Turbulent transport reduction by zonal flows: Massively parallel simulations, Science, 281 (1998), 1835–1837.Google Scholar
[7]Idomura, Y., Tokuda, S., and Kishimoto, Y., Global gyrokinetic simulation of ion temperature gradient driven turbulence in plasmas using a canonical Maxwellian distribution, Nuclear Fusion, 43 (2003), 234–243.Google Scholar
[8]Jolliet, S., Bottino, A., Angelino, P., Hatzky, R., Tran, T. M., Mcmillan, B. F., Sauter, O., Ap-pert, K., Idomura, Y., and Villard, L., A global collisionless PIC code in magnetic coordinates, Computer Physics Communications, 177 (2007), 409–425.CrossRefGoogle Scholar
[9]Parker, S. E., and Lee, W. W., A fully nonlinear characteristic method for gyrokinetic simulation, Physics of Fluids B, 5 (1993), 77–86.Google Scholar
[10]Dawson, J. M., Particle simulation of plasmas, Reviews of Modern Physics, 55 (1983), 403–448.Google Scholar
[11]Kniep, J. C., Gyrokinetic Particle-in-Cell Simulations of Strong Poloidal Flow Radial Gradient and Parallel Nonlinearity Effects on Toroidal Ion Temperature Gradient Driven Turbulence, Ph. D. Thesis, University of California, Los Angeles, CA, 2006.Google Scholar
[12]Norton, C. D., Decyk, V. K., Szymanski, B. K., and Gardner, H., The transition and adoption of modern programming concepts for scientific computing in Fortran, Scientific Programming, 15 (2007), 27–44.Google Scholar
[13]Decyk, V. K., and Norton, C. D., UCLA parallel PIC framework, Computer Physics Communications, 164 (2004), 80–85.Google Scholar
[14]Decyk, V. K., UPIC: A framework for massively parallel particle-in-cell codes, Computer Physics Communications, 177 (2007), 95–97.Google Scholar
[15]Lee, W. W., Gyrokinetic particle simulation model, Journal of Computational Physics, 72 (1987), 243–269.Google Scholar
[16]Leboeuf, J. N., Dawson, J. M., Decyk, V. K., Kissick, M. W., Rhodes, T. L., and Sydora, R. D., Effect of externally imposed and self-generated flows on turbulence and magnetohydrodynamic activity in tokamak plasmas, Physics of Plasmas, 7 (2000), 1795–1801.Google Scholar
[17]Rettig, C. L., Staebler, G. M., Rhodes, T. L., Leboeuf, J. N., Peebles, W. A., Doyle, E. J., Burrell, K. H., and Moyer, R. A., Search for the ion temperature gradient mode in a tokamak plasma and comparison with theoretical predictions, Physics of Plasmas, 8 (2001), 2232–2237.Google Scholar
[18]Rhodes, T. L., Leboeuf, J.-N., Sydora, R. D., Groebner, R. J., Doyle, E. J., McKee, G. R., Peebles, W. A., Rettig, C. L., Zeng, L., and Wang, G., Comparison of turbulence measurements from DIII-D low-mode and high-performance plasmas to turbulence simulations and models, Physics of Plasmas, 9 (2002), 2141–2148.Google Scholar
[19]Kniep, J. C., Leboeuf, J.-N., and Decyk, V. K., Gyrokinetic particle-in-cell calculations of ion temperature gradient driven turbulence with parallel nonlinearity and strong flow corrections, Computer Physics Communications, 164 (2004), 98–102.Google Scholar
[20]Sanchez, R., Newman, D. E., Leboeuf, J.-N., Decyk, V. K., and Carreras, B. A., Nature of transport across sheared zonal flows in electrostatic ion-temperature-gradient gyrokinetic plasma turbulence, Physical Review Letters, 101 (2008), 205002.Google Scholar
[21]Sanchez, R., Newman, D. E., Leboeuf, J.-N., Carreras, B. A., and V. K. Decyk, ., On the nature of radial transport across sheared zonal flows in ion-temperature-gradient gyrokinetic tokamak plasma turbulence, Physics of Plasmas, 16 (2009), 055905.Google Scholar
[22]Sanchez, R., Newman, D. E., Leboeuf, J.-N., and Decyk, V. K., On the nature of turbulent transport across sheared zonal flows: insights from gyro-kinetic simulations, Plasma Physics and Controlled Fusion, 53 (2011), 074018.Google Scholar
[23]Hahm, T. S., Nonlinear gyrokinetic equations for turbulence in core transport barriers, Physics of Plasmas, 3 (1996), 4658-4664.CrossRefGoogle Scholar
[24]Naitou, H., Sonoda, T., Tokuda, S., and Decyk, V. K., Parallelization of gyrokinetic particle code and its application to internal kink mode simulation, Journal of Plasma and Fusion Research, 72 (1996), 259–269.Google Scholar
[25]Kim, C. C. and Parker, S. E., Massively parallel three-dimensional toroidal gyrokinetic flux-tube turbulence simulation, Journal of Computational Physics, 161 (2000), 589–604.Google Scholar
[26]Hatzky, R., Domain cloning for a particle-in-cell (PIC) code on a cluster of symmetric-multiprocessor (SMP) computers, Parallel Computing, 32 (2006), 325–330.Google Scholar
[27]Madduri, K., Williams, S., Ethier, S., Oliker, L., Shalf, J., Strohmaier, E., Yelick, K., Memory-efficient optimization of gyrokinetic particle-to-grid interpolation for multicore processors, Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (SC ’09), IEEE Computer Society, Los Alamitos, CA, 2009, pp. 1–12. http://www.computer.org/csdl/proceedings/sc/2009/8744/00/87440048-abs.htmlGoogle Scholar
[28]Adams, M. F., Ethier, S., and Wichmann, N., Performance of particle in cell methods on highly concurrent computational architectures, Journal of Physics: Conference Series, 78 (2007), 012001.Google Scholar
[29]Wang, B., Ethier, S., Tang, W., Williams, T., Ibrahim, K., Madduri, K., Williams, S., and Oliker, L., Kinetic turbulence simulations at extreme scale on leadership-class systems, Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC ’13), Association for Computing Machinery, New York, NY, USA, 2013, Article No. 82. http://dl.acm.org/citation.cfm?doid=2503210.2503258Google Scholar
[30]Oliker, L., Canning, A., Carter, J., Shalf, J., and Ethier, S., Scientific application performance on leading scalar and vector supercomputering platforms, International Journal of High Performance Computing Applications, 22 (2008), 5–20.Google Scholar
[31]Decyk, V. K., and Singh, T. V., Particle-in-cell algorithms for emerging computer architectures, Computer Physics Communications, 185 (2014), 708–719.CrossRefGoogle Scholar