The High Level Support Team — A support unit for petaflop computing; Roman Hatzky, Max-Planck-Institut fuer Plasmaphysik, IPP-EURATOM-Association, Germany

Modern petaflop computer architectures are already so demanding that an optimal exploitation can be only performed with a special support unit. The High Level Support Team (HLST) provides support to scientists from all EFDA (European Fusion Development Agreement) Associates for the development and optimization of codes to be used on the dedicated IFERC-CSC supercomputer located in Rokkasho, Japan. The supercomputer delivers computing power of about 1.52 petaflop/s and is optimally suited for the fusion scientists’ simulation programs. The HLST consists of a core team based at the IPP Garching (Max-Planck-Institut für Plasmaphysik) and of other high level support staff provided by the Associates. The HLST members are all HPC experts with a background in developing large scientific applications and particular expertise in numerical algorithms and and visualization.

Conservation relations and accurate schemes for delta-f Particle-In-Cell simulation; Ben F. McMillan (Centre for Fusion, Space and Astrophysics, Department of Physics, University of Warwick, UK)

The Particle-In-Cell method is a means to solve the advection problems that arise when modelling weakly collisional plasmas and gases, which follows the motion of a reduced number of marker particles along the physical trajectories. Although solving the trajectories accurately is relatively straightforward, inaccuracies arise when computing moments of the distribution functions, due to uneven sampling. These statistical errors can be significantly reduced by using a control variates scheme, where most of the integration can be performed numerically, and only the fluctuating component needs to be evaluated using the Monte-Carlo method; this is effective where the distribution function stays close to some known function, such as a local Maxwellian.

The code of interest, ORB5, uses this method to solve for gyrokinetic plasma turbulence in the core of tokamak devices. The control-variates method is known to work well for this problem, reducing computational cost by several orders of magnitude; refinements to this method have also been implemented which prevent the growth of the marker weights, and optimally sample the distributions. Although the control variates method ensures that sampling noise is reduced, we can no longer guarantee exact conservation properties. By explicitly determining the sources of non-conservation in the system, we determine the errors in the local particle number and momentum transport equations. Momentum transport is sensitive to small errors in particle transport, because small charge sources leads to substantial electric field generation. We therefore have implemented an algorithm which imposes exact satisfaction of the charge transport equation, and this substantially improves the accuracy of the slow system-scale profile evolution.

Using this improved algorithm, we use ORB5 to study the profile evolution in a tokamak forced by external sources of heat and momentum. For certain parameter regimes the tokamak exhibits strong rotation gradients with weak external momentum input, providing a possible resolution to the long-standing puzzle of transport barrier formation in tokamaks.

Simulations of energetic particle driven instabilities in Tokamak; A. Bottino, A. Biancalani, D. Zarzoso (Max Planck Institut für Plasmaphysik, Germany), L. Villard (Centre de Recherches en Physique des Plasmas, EPFL)

Plasma heating is essential for reaching appropriate fusion temperatures in tokamak plasmas, but as a side effect global modes can become unstable by converting thermal energy into kinetic energy. The energetic particle (EP) population produced in the process of heating together with the alpha particles produced in fusion reactions act as important actors in this chain, driving plasma oscillations unstable via resonant interaction. On the other hand, the plasma instabilities redistribute the EP population making the plasma heating less effective. For this reasons, it is important to develop models and numerical tools able to describe and predict the linear and nonlinear behavior of those instabilities. Therefore, a new numerical tool, based on an existing code, has been recently developed. The code NEMORB is a particle-in-cell code, based on gyrokinetic equations, derived as the multispecies, electromagnetic version of the code ORB5 [1], which was written for studies of turbulence in tokamak plasmas. Recently NEMORB has been applied to the study of collective instabilities in the presence of a EP population. It has been successfully validated against analytical theory and benchmarked with other codes. [1] S. Jolliet, A. Bottino, P. Angelino et al., Comput. Phys. Comm., 177 (2007) 409.

Towards a GPU-MIC Particle-In-Cell Code For Plasma Applications; Farah Hariri (Ecole Polytechnique Fédérale de Lausanne, EPFL), T. M. Tran (EPFL), A. Jocksch (Swiss National Supercomputing Centre, CSCS), L. Villard (EPFL), S. Brunner (EPFL), G. Gheller

HPC architectures undergo fast hardware and system software upgrades, hence necessitating onerous rebuild of application codes. Our current work aims to foster and streamline the process of adapting particle-in-cell codes to heterogeneous platforms equiped with GPUs and MICs. Within a Co-Design project, our main focus will be on enhancing the portability of the turbulence simulation code ORB5 [1,2] across multifarious machines in order to simulate microturbulence in ITER-size tokamaks. While ORB5 has already shown good parallel scalability on several HPC platforms, we will sketch the essential modifications that will need to be implemented in order to make efficient usage of today's most performing clusters. To this end, a PIC-engine framework containing the main features of the PIC algorithm is being developed. Different parallel programming models are tested in this framework, aiming at finding the best performing tool to allow for easily portable and smaller time-to-solution codes. The encountered programming and computational challenges are to be discussed in this work. Efficient use of computer resources can further be achieved by taking into account the strong anisotropy of magnetized plasmas. This is possible by choosing an optimized system of coordinates such as the FCI system presented in [3] which could lead to a gain of one or two orders of magnitude in computational efficiency for this class of codes.

References: [1] T. M. Tran, et al., Theory of Fusion Plasmas, Joint Varenna-Lausanne Int. Workshop, 1998. [2] S. Jolliet, et al., Comput. Phys. Commun.,2007. [3] F. Hariri and M. Ottaviani, Comput. Phys. Commun., 2013.

Structure preserving schemes; Roger Käppeli (ETH Zurich)

The equations of (magneto-) hydrodynamics can be solved very efficiently with standard high-resolution shock-capturing schemes. These schemes enable the simulation of complex flows, involving both smooth and discontinuous features, in a robust and accurate manner. Beside the common conserved quantities, such as mass, linear momentum and total energy, these equations also possess certain associated structures that are analytically preserved and are of crucial importance in many applications. Standard numerical schemes generally fail in preserving these structures at the discrete level. We will focus on specific applications in astrophysics and present extensions of standard schemes aiming at preserving two such structures discretely. The first extension deals with the accurate simulation of flows close to a hydrostatic state. We will present a so-called well-balanced scheme which is capable of capturing certain stationary states exactly. The second extension is concerned with the conservation of angular momentum. We will discuss the requirements of a numerical scheme to conserve angular momentum and present a genuinely multi-dimensional scheme fulfilling them. Several numerical examples will be presented to demonstrate the performance of the novel schemes.

Radiative transfer in multi-scale, multi-physics simulations; the big challenge in astrophysics and cosmology; Lucio Mayer (University of Zurich)

The predictive power of astrophysical simulations is often hindered by their inability to properly follow the energy transport via radiation and particle carriers over many orders of magnitude in physical scales. At the same time, 3D multi-scale, massively parallel simulations are needed to elucidate the origin of all astrophysical objects, from planets to stars to galaxies, and even the largest black holes in the Universe that could be used as a novel tracer of the structure of spacetime with future experiments. Such simulatons are in turn relevant to answer the big open questions in astrophysics and cosmology. I will discuss the status of radiative simulations in several areas of astrophysics and the challenges ahead. Those are the challenges that the new PASC project "DIAPHANE" aims at tackling.  

Efficient space-charge calculation on irregular domains using adaptive mesh refinement; Tulin Kaman (ETH Zurich and Paul Scherrer Institute), Andreas Adelmann (Paul Scherrer Institute), Ann Almgren (Lawrence Berkeley National Laboratory), Peter Arbenz (ETH Zurich)

Practical simulations of beam dynamics processes are generally cutoff, meaning that due to the limited grid resolution the accuracy is insufficient to resolve the length scales at places desired the most. Since most physical problems have variations in scale, adaptive discretization should improve the efficiency and/or the accuracy of computation. A suitable technology to achieve extra resolution needed within the regions is Adaptive Mesh Refinement (AMR).

We extend OPAL (Object Oriented Parallel Accelerator Library) high-performance algorithmic and software framework with multi-scale capabilities to study the quantitative and efficient evaluation of the space-charge effects in particle accelerators. For this purpose, we create an interface between the block-structured AMR framework and preconditioned iterative solvers to solve Poisson's equation for the electrostatic potential on irregular bounded three-dimensional domains. This new adaptive multigrid technique for 3D space charge calculation on the multilevel grid hierarchy has an interface between OPAL - BoxLib - Trilinos frameworks. BoxLib, block-structured AMR framework, is the tool used for discretization in the regions that requires higher level of grid resolution and Trilinos is the tool used for the solution of the system using smoothed aggregation based algebraic multigrid preconditioning. The problem discretization takes into account the arbitrary geometries of beam pipe elements and the space-charge forces are computed within the geometry. This assures that the image charge components are taken properly into account in any type of geometry. The new technique for accurate 3D space-charge calculations on irregular domain will be used to investigate the image charge components in spiral inflector to determine the limits of IsoDAR Cyclotron, the dark currents in the low energy part of Free Electron Laser accelerator and halo particles in the existing and future high intensity hadron accelerators.

PKDGRAV2: Minimal data movement and high computation in a parallel fast multipole tree code; Joachim Stadel (University of Zurich)

The PKDGRAV2 code for simulating self gravitating systems in cosmology and planet formation has been in use for many years, but in this talk I will focus on recent insights gained in adapting the algorithms to SIMD computation both within the CPU as well within a GPU accelerator. I will argue that the fast multipole method is one of the best suited algorithms for solving the Poisson equation on modern architectures due to its minimal requirement on data movement. However, using a GPU as a coprocessor also requires extra movement of data to and from the GPU device which is quite slow in current systems. It turns out that the fast multipole algorithm allows one to significantly reduce this data stream thereby achieving a high computational rate despite the slow channel to and from current GPU devices.

Adaptively balanced parallel MLMC solver for acoustic wave propagation with log-normal coefficients; Jonas Sukys (ETH Zurich), Siddhartha Mishra (ETH Zurich), Christoph Schwab (ETH Zurich)

We consider multi-dimensional acoustic wave equation with random log-normally distributed coefficients, exhibiting anisotropically correlated layered structures that are characteristic to geological subsurface data observed by seismic imaging or direct measurements.
As the acoustic wave equation is a linear system of conservation laws, the existence and uniqueness of the weak solutions as well as the estimates for the space-time and statistical regularity of the solution are established. For uncertainty quantification in the solutions of multi-dimensional systems of stochastic conservation laws, such as Euler, ideal magnetohydrodynamics (MHD), and shallow water equations, the Multi-Level Monte Carlo Finite Volume Methods (MLMC-FVM) were shown to be robust and provide optimal error convergence rates. We generalize MLMC-FVM to linear systems of conservation laws with random fluxes, in particular, to acoustic wave equation with random coefficients, and provide error bounds for mean square error of MLMC-FVM vs. expected computational work, which are asymptotically of the same order as error bounds for a single, deterministic FVM solve.

Log-normally distributed material coefficients can be efficiently generated using spectral method and Fast Fourier Transform (FFT). In order to couple the spectral FFT generator with the MLMC framework, an upscaling technique to generate identical (up to discretization error) realizations on several different mesh resolutions is developed. The MLMC-FVM algorithm, together with generation of the coefficients using FFT, is parallelized using a recently developed adaptive load balancing algorithm, which takes into account the sample path dependent complexity of the underlying deterministic solver, arising from the random CFL-restricted time step size due to the random wave speed. Adaptive load balancing is implemented in code ALSVID-UQ and is shown to scale up to 40 000 cores. Numerical experiments for uncertainty quantification in acoustic wave scattering in two and three dimensions are presented

Enabling LHC for HPC; Sigve Haug (University of Bern), Michael Hostettler

The Large Hadron Collider (LHC) at CERN in Geneva is the largest and most powerful particle collider ever built. It provides humanity with latest knowledge about universe's fundamental building blocks, it’s earliest conditions and dark matter.

The computational needs around LHC are in many aspects advanced and challenging. Full simulation of theoretical models, detectors and data acquisition systems are not always possible. Only very small fractions of the collected data can be made persistent. The storage volumes approach the exabyte scale. Several hundred computer centres all over the planet are connected into a distributed computing system serving thousands of collaborators.

So far high-energy physics (HEP) software has been designed around an assumption with one CPU core per computational job with a relatively large amount of RAM available. HEP software now needs to accommodate new hardware architectures by introducing parallelism whenever possible in order to make efficient use of all the available cores.

Another challenge is to adapt the world-wide distributed computing systems of the LHC experiments to the environments on the world's largest HPC systems. The Swiss Piz Daint is currently the European flag-ship HPC system. This talk reports on our investigations and achievements towards usage of many-core systems and Swiss flag-ship HPC resources.

DFT+DMFT study of epitaxially strained perovskite LVO; Gabriele Sclauzero, Krzysztof Dymkowski, Claude Ederer (ETHZ)

LaVO_3 (LVO) is a t_2g perovskite showing a rich phase diagram as a function of temperature, with a paramagnetic Mott-Hubbard insulating state at room temperature. Recently, it has become possible to produce and characterize strained lattices of LVO in thin films and superlattices of high structural quality. In such heterostructures both interface effects and epitaxial strain play an important role in the electronic reconstruction of the material, and it is often difficult to distinguish between these two effects in experiments. Moreover, the epitaxial strain can induce at least two different important modifications of the atomic geometry, namely: (i) a change in the amplitudes of the VO_6 octahedral rotations; (ii) a stretching or compression of V-O bonds. In this work, we perform density functional theory plus dynamical mean field theory (DFT+DMFT) simulations to study how the correlated Mott-Hubbard phase of LVO is affected by epitaxial strain. We separate the effect of bond-length changes from that of octahedral rotations by comparing an idealized structure without rotations and a realistic one derived from the bulk orthorhombic phase. We interpret our findings through the strain-induced changes in the energies of the crystal-field t_2g levels of the local DFT Hamiltonian and in the occupation matrix obtained from DMFT. A comparison with the t_2g perovskite LaTiO_3, which shows an insulator-to-metal transition upon compressive strain, will also be presented.

Radiation hydrodynamics with the AMR code RAMSES; R. Teyssier (University of Zurich)

tbd