Weather and Climate modelling at the Petascale: achievements and perspectives; Pier Luigi Vidale, University of Reading, United Kingdom

Increasing availability of high-performance computing has allowed the climate modeling community to construct a new generation of global coupled models, capable of resolving some of the fundamental processes that govern the climate system. More explicit representation of eddies in the ocean and of weather systems in the atmosphere has demonstrated how they contribute to the general circulation and how it is increasingly possible to rely less on uncertain physical parameterisations. These developments are potentially promising for reducing inter-model disagreement in community multi-model hindcasts and projections. We have demonstrated, for instance, how the representation of the relative importance of the remote and local processes governing the global energy and hydrological cycles converges at resolutions beyond 50km. These so-called “weather-resolving” climate models are also suitable for studying how global teleconnections might influence the long-term regional changes in storm tracks, or in understanding changes in high-impact phenomena, such as Tropical and Extra-Tropical Cyclones, as well as their contributions to the general circulation.

Towards GPU-accelerated Operational Weather Forecasting; Oliver Fuhrer, MeteoSwiss

Higher grid resolution, larger ensembles as well as growing complexity of weather and climate models demand ever-increasing compute power. In 2013, several large hybrid high performance computers that contain traditional CPUs as well as some type of accelerator (e.g. GPUs) are online and available to the user community. Reasons for this development are an increase in computing performance in parallel with decreasing power consumption offered by such accelerators. This trend is foreseen to continue in the coming years, as many compute centers have an upper limit in terms of an acceptable power envelope. Early adopters of this technology trend may have considerable advantages in terms of available resources and energy-to-solution. On the downside, a substantial investment is required in order to adapt applications to such accelerator-based supercomputer. Within the COSMO Consortium and the Swiss HP2C Initiative, a version of the weather and regional climate prediction model able to run on GPUs is being developed. This contribution will give an overview of the approaches used, the status of this version and present a roadmap of further plans. Design criteria such as high performance, retaining maintainability, enforcing a single source code which still compiles and runs on x86-based, led us to opt for different approaches in different parts of the model code. While the physical parameterizations have been ported to GPUs using compiler directives, a more fundamental change of programming model has been used for porting the dynamics. We will discuss our experience and advantages and disadvantages of these two porting approaches. Finally, results for time-to-solution and energy-to-solution for COSMO on a hybrid Cray XC30 (Piz Daint) and comparing its performance against a traditional CPU-based system will be presented.

Using OpenACC compiler directives to achieve performance portable code on CPU and GPU architectures; Xavier Lapillonne (C2SM, ETH Zurich), Oliver Fuhrer (Meteoswiss)

In view of adapting the numerical weather prediction code COSMO to high-performance computing architectures with accelerators, a new version of the code capable of running on graphics processing units (GPUs) has been developed. For large parts of the model, the porting is achieved by using compiler directives, so-called OpenACC directives, which enables to generate specific code for GPUs. In principle, compiler directives enable a non-intrusive porting approach via the insertion of pragmas which are simply ignored when the code is compiled for CPU execution. In order to get good performance on GPUs specific code refactoring and optimizations are nonetheless required. In some cases, this may lead to a decrease in performance when running only on traditional CPUs. Being a community code, it is of crucial importance that the COSMO model performs well on both GPU and CPU architectures while remaining easily maintainable which implies retaining a single source code. Looking at the concrete example of the parametrization of atmospheric radiative transfer, we first present some restructuring techniques necessary to achieve high performance on the GPU. We then show that the performance of specific parts of the radiation code are bound by the available floating point performance on the CPU while they are limited by the available system memory bandwidth on the GPU. This leads to different requirements in terms of optimization. Finally, we discuss various solutions to achieve a portable and maintainable code.

Towards Cloud-Resolving European-Scale Climate Simulations using a fully GPU-enabled Prototype of the COSMO Regional Climate Model; David Leutwyler (Institute for Atmospheric and Climate Science, ETH Zurich), Oliver Fuhrer (MeteoSwiss), Benjamin Cumming (CSCS), Tobias Gysi (ETH Zurich), Xavier Lapillonne (C2SM, ETH Zurich), Daniel Lüthi (Institute for Atmospheric and Climate Science, ETH Zurich), Carlos Osuna (C2SM, ETH Zurich), Christoph Schär (Institute for Atmospheric and Climate Science, ETH Zurich)

The representation of moist convection is a major shortcoming of current global and regional climate models. State-of-the-art global models usually operate at grid spacings of 10-300 km, and therefore cannot fully resolve the relevant upscale and downscale energy cascades. Therefore parameterizations of the relevant sub-grid scale processes is required. Several studies have shown that this approach entails major uncertainties for precipitation processes, which raises concerns about the model’s ability to represent precipitation statistics and associated feedback processes, as well as their sensitivities to large-scale conditions and climate change. Refining the model resolution to the kilometer scale allows omitting the parameterization of deep convection. Precipitation processes are then represented much closer to first principles and thus should allow an improved representation of the water cycle including the drivers of extreme events.

Cloud-resolving simulations have proven to be very useful tools for climate simulations and numerical weather prediction. However, the implied high horizontal resolution and consequently the small time steps needed represent a major challenge for current supercomputers, in particular when simulating on large domains and over long time scales. The recent innovations in the domain of hybrid supercomputers have led to mixed node designs with a conventional CPU and an accelerator such as a graphics processing unit (GPU). GPUs have a larger system memory-bandwidth, which is highly beneficial for codes with low compute intensities, such as atmospheric stencil-based models. However, to efficiently exploit these hybrid architectures, climate models need to be ported and/or redesigned.

Within the framework of the Swiss High Performance High Productivity Computing initiative (HP2C) a project to port the COSMO model to hybrid architectures has recently come to and end. The product of these efforts is a prototype version of COSMO with an improved performance on traditional CPU-based clusters as well as hybrid architectures with GPUs. Here we present a first set of short-term cloud-resolving simulations in climate mode covering the European-scale using the GPU-enabled COSMO prototype. Results show how the approach allows for the representation of interactions between synoptic-scale and meso-scale atmospheric circulations at scales from 1000 to 10 km, such as the formation of sharp cold frontal structures or the organization of convective clouds. We will discuss the added value of cloud-resolving resolution on the European Scale and elaborate on our plans to exploit this new modelling capability. Furthermore we will present our experience and lessons learned from enabling and using climate codes on hybrid supercomputers.

An Application Driven, Power Efficient System Design for 1-km Scale Global Cloud System Resolving Models; David Donofrio, Michael Wehner, Leonid Oliker, John Shalf (Lawrence Berkeley National Labs, USA)

The demand for supercomputing resources continues to accelerate and demand ever more power, both computational and electrical, to accurately model extreme problem spaces such as high-resolution resolving models of the global cloud system. No longer can we rely on the traditional doubling of clock speeds every 18-24 months- clock frequencies remain flat and we now see the doubling of the number of cores for explicit parallelism. During the next decade the amount of parallelism on a single microprocessor will rival the number of nodes in the first massively parallel supercomputers of 1980s. Building ever-larger clusters of commercial off-the-shelf (COTS) hardware will force designers to face problematic power and cooling constraints. Applications and algorithms will need to change and adapt as node architectures evolve. As we look forward to a machine capable of the exaflops performance (1 billion- billion floating point operations per second), projections indicate power consumption upwards of 180 megawatts if current technology is scaled forward. This extreme level of power consumption at low efficiency levels makes the current COTS based HPC design philosophies unsustainable. Rather than rely on the general-purpose, power-hungry processors that make up traditional servers, it makes more sense to leverage the considerable innovation of the low-power processor designs developed for embedded computing markets, such as cell phones, and design a processor tailored for demanding scientific applications. Global climate change has huge economic and political consequences and the models used to forecast this change are driving applications for extreme scale systems. The global climate models used for the IPCC (Intergovernmental Panel on Climate Change) assessment reports ranged in horizontal resolution from about 400 km to as fine as 50 km. These production models must be able to simulate decades to centuries of the climate in a reasonable amount of time to provide input to these reports. This constraint provides a meaningful metric for the minimum computational throughput of a production climate model. Although somewhat arbitrary, if a model can simulate the climate a thousand times faster than real time, the production demands of century scale climate simulation can be met in about three weeks of dedicated machine time. Going further, an increase in resolution to a 1-km scale model would allow for explicit rather than parameterized simulation of cumulus convective processes but demand a machine with 1,000 times the performance of current generation systems. Such a kilometer-scale model would decompose the Earth’s atmosphere into 20 billion individual cells, presenting an opportunity to leverage a machine with massive parallelism. This presentation will present a credible estimate of the computational rates and other hardware attributes necessary to integrate GCSRMs in a manner similar to the production climate models of today but at a significantly higher resolution. We arrive at these results through an analysis of several codes prepared at the Colorado State University in Fort Collins, Colorado. The estimates of overall machine requirements then provide the basis for our alternative path towards exascale scientific computing – a machine we call Green Flash. Green Flash is an application-driven design tailored to the problem of high-resolution global cloud resolving models. Green Flash achieves an order-of-magnitude increase in efficiency of scientific computations relative to a conventional approach and could be brought online well before the exascale machine of 2020.
Michael F. Wehner, Leonid Oliker, John Shalf, David Donofrio, Leroy A. Drummond, Ross Heikes, Shoaib Kamil, Celal Konor, Norman Miller, Huiro Miura, Marghoob Mohiyuddin, David Randall, Woo-Sun Yang (2011) Hardware/Software Co-design of Global Cloud System Resolving Models. Journal of Advances in Modeling Earth Systems 3, M10003, DOI:10.1029/2011MS000073

Performance of semi-implicit compressible and anelastic EULAG model for all-scale atmospheric flows; Zbigniew P. Piotrowski, Andrzej A. Wyszogrodzki (IMGW), Piotr K. Smolarkiewicz (ECMWF)

Usually, evaluation of the computational performance of modern numerical models involves distinct numerical methods solving a particular governing equation set. However, it is uncommon, at least in geophysical flows, to evaluate relative performance of the same numerical methods applied to the given physical problem described by mathematically distinct equation sets. In this presentation we offer the examination of computational performance of consistent compressible-soundproof formulation of EULAG model for all scale flows. The equations are solved with MPDATA advection schemes and Generalized Conjugate Residual iterative solver. Machinery of the implict solver of EULAG handles Helmholtz and Poisson problem for compressible and soundproof equations, respectively. In principle, compressible solver employs a larger set of advected dynamical variables and more complicated differential operators in the implicit solver. However, in practice it offers a competitive performance due to different accuracy requirements We discuss the performance characteristics of idealized global Held-Suarez climate simulation, mesoscale Alpine flow and decaying turbulence experiments for various equation sets. We compare the differences in the cost of advection and implict solver resulting from the different equations solved and we provide the time-to-solution results on Cray XC30. Finally, we discuss the feasibility of vertical MPI decomposition (available in EULAG) for components of the consistent EULAG numerics.

Towards a multi-node OpenACC Implementation of the ICON Model; William Sawyer (Swiss National Supercomputing Centre), Guenther Zaengl (German Weather Service, DWD), Leonidas Linardakis (Max Planck Instititue for Meteorology, MPI-M), Markus Wetzstein (Swiss National Computing Centre), Christian Conti (Swiss Federal Institute of Technology, ETH)

We have ported the Icosahedral Non-hydrostatic (ICON) model's dynamics solver to Graphical Processing Units (GPUs), which is a task within the Partnership for Advanced Computing in Europe (PRACE) Second Implementation Phase (2IP) Work Package 8 (WP8). Initial single-node OpenCL and CUDA Fortran implementations of ICON's non-hydrostatic dynamical core (NHDC) resulted in a maximum factor of two speedup over the latest CPU nodes, e.g., a dual-socket Intel Sandybridge. While this performance was promising, ICON developers viewed neither OpenCL nor CUDA Fortran as viable programming paradigms for the actual production code. They suggested instead the OpenACC standard -- compiler directives which are only utilized in an accelerator setting, and thus are minimally intrusive to the existing CPU-targeted code -- as the proper paradigm for the multi-node GPU implementation, which was then undertaken in the second year of WP8.

We will present the results of the multi-node OpenACC implementation of the ICON NHDC for hybrid multicore platforms. The code baseline is the ICON "DSL" (Domain Specific Language) testbed code, which is essentially a stripped-down version of the ICON model for dynamics simulations only. We will discuss the OpenACC directives used for the port of the computational as well as the communication code to GPUs, and report the resulting GPU performance on NVIDIA K20x as compared to contemporary CPU architectures.

In addition, the roadmap for an accelerated ICON full model will be presented. As a first step, we are now incorporating the OpenACC directives into the ICON NHDC in development trunk, based on the feedback given to us from the ICON developers at the Max Planck Institute for Meteorology (MPI-M) and the German Weather Service (DWD). This effort has revealed deficiencies in the OpenACC standard, which have been reported to the standards committee, in particular the insufficient support of Fortran derived types.

Finally, we have concrete plans to incorporate accelerator-capable components into ICON, such as the Rapid Radiative Transfer Model (RRTM), which have already been ported to GPUs by other groups. Moreover, we have initiated the port of other ICON Climate physical parameterizations stemming from the ECHAM and COSMO models to OpenACC. This step should enable ICON for certain scientific configurations to run on many-core platforms which support OpenACC. We will discuss planned efforts to port ICON extensions to accelerators, such as the HAMMOZ module for aerosol-cloud interactions and atmospheric chemistry. The resulting accelerated components should benefit climate researchers world-wide who plan to transition from ECHAM to ICON in the coming years.

Big data in climate modeling; Reto Knutti (ETH Zurich)

Recent coordinated efforts, in which numerous global general circulation climate models have been run for a common set of experiments, have produced large datasets of projections of future climate for various scenarios. Those multi-model ensembles sample initial condition, parameter as well as structural uncertainties in the model design. The most recent Coupled Model Intercomparison Phase 5 (CMIP5) has produced data for hundreds of variables and simulations, exceeding a total of 2 Petabytes, which are stored in a distributed archive with centralized search capability. Here we discuss the concepts behind such data archives, the issues with version control, scientific quality control, the technical challenges of storing and transferring data in distributed systems, and in analyzing the data. We argue that the next generation of such a data archive, which is already planned, will only be possible with significant investments in both technical infrastructure as well as in data management.

Distributed Memory GPU-enabled Compression of Very Large Climate Time Series Data; William Sawyer (CSCS), Benjamin Cumming (CSCS), Illia Horenko (University of Lugano), Timothy Moroney (Queensland University of Technology), John Biddiscombe (CSCS)

Platforms containing multiple nodes with Graphics Processing Units (GPUs) offer huge computational potential for high performance applications. Unfortunately, they require new programming paradigms, such as CUDA or OpenACC directives, which add overhead to the development effort. Moreover, most applications require portability to existing CPU platforms and must run in a distributed memory context, adding additional levels of complexity.

The Swiss National Supercomputing Centre (CSCS) is investigating techniques to minimize the application programming impact while maximizing the portability and achieving good performance of distributed memory GPU clusters, such as the recently installed Cray XC30 "Piz Daint". In this paper, the MINLIN library, which allows linear algebra and matrix manipulations to run transparently on CPU or GPU nodes, is combined with the Message Passing Interface (MPI), to allow the portable parallelization of recently developed time series analysis techniques. These techniques, which are loosely related to Principle Component Analysis (PCA), allow the investigations of correlations and trends in very large data sets and are computational intensive with numerous dense linear algebra operations. They are only computationally viable if the problem is distributed over many nodes, and the matrix operations on each node are efficiently implemented.

The implementation discussed here utilizes these data analysis techniques to compress time series of three-dimensional fields from a climate model to minimize the size of the output files. This lossy compression software would be envisioned to run on tens to hundreds of "I/O nodes" foreseen for the asynchronous postprocessing output of climate simulations running on still larger number of "compute nodes". From an initial MATLAB implementation provided by the developers, the algorithm was first implemented in C++ with the Eigen library, which currently provides only multi-threaded CPU support. It was then extended to distributed memory with MPI by utilizing an algorithmic formulation which inherently allows for a distribution of time series. Finally, Eigen code were replaced with the MINLIN templates, which provide a subset of Eigen functionality but utilizes NVIDIA's public-domain Thrust library to ensure portability to both CPU and GPU architectures.

Performance of the multi-node multi-threaded Eigen (CPU) and MINLIN (CPU/GPU) versions are compared in the realm of tens to hundreds of nodes. Compression ratios will also be presented, but without a comprehensive analysis of the quality of the reconstructed climate data.

Statistical regression analysis of extreme events in HPC context; Olga Kaiser, Illia Horenko (Institute of Computational Science, Università della Svizzera Italiana)

Data-based analysis of extreme events is a prominent problem in climate and weather research. Thereby, the main objective is the identification of (i) the statistical/stochastic models describing the dynamics of extremes, (ii) the spatio-temporal structure of model parameters and (iii) the most significant covariates that impact the probabilities of extremes. Assuming that all relevant covariates are known and observed, statistical regression analysis is often deployed to study the changes in the model parameters. If some of the relevant model covariates are systematically missing the involved regression model becomes non-stationary. Standard approaches for modeling the dependence structure imply a priori assumptions, e.g., Gaussian, local stationary or isotropic behavior. We present an HPC-scalable, non-stationary, semi-parametric framework for statistical regression analysis of extremes with systematically missing covariates and a non-parametric, non-stationary and anisotropic description of the dependence structure. Based on extreme value theory (EVA) and Finite Element time series analysis methods (FEM), the resulting FEM-EVA approach allows a well-posed problem formulation and goes beyond probabilistic a priori assumptions of methods for analysis of extremes based on, e.g., non-stationary Bayesian mixture models, smoothing kernel methods or neural networks. Scalable numerical solution strategies based on gradient-free MCMC methods and numerical solvers for constrained, structured quadratic and linear problems will be presented and compared with standard methods of extreme value regression analysis (e.g., based on Artificial Neuronal Networks). Comparison to standard approaches will be shown both wrt. robustness to missing data and complexity/scalability.