Fluids and Supercomputers: The Billion Particle Eraby Petros Koumoutsakos A century of advances in numerical methods, integrated with advances in software and hardware, provide us today with unprecedented tools to study and control flow as it pertains to key problems of our society. This includes energy (wind turbines, aircraft wakes), health (microfluidics, hematodynamics) and nanotechnology (nanofluidics, nanomedicine). At the Chair of Computational Science at ETH Zurich, we are developing multiscale particle methods for simulating diverse physical systems [1], with an emphasis on their implementation in high-performance computing (HPC) architectures. Multiscale simulations often require a level of algorithmic complexity that is difficult to translate into effective parallel algorithms. A key aspect of our approach is the development of numerical methods with an emphasis not only on their accuracy and stability but also on their software engineering and implementation on massively parallel computer architectures. In one such example from the last two years, we have developed particle methods for the study of vortex-dominated flows. We collaborated with researchers from the IBM Zurich Research Laboratory to port these methods onto massively parallel computer architectures. This enabled state-of-the-art direct numerical simulations of aircraft vortex wakes [2], which employed billions of vortex particles and exhibited excellent scalability for up to 16,000 nodes of the IBM Blue Gene/L (BG/L). Resurrecting Vortex Methods for HPC  Figure 1: Visualisation of aircraft vortex wakes undergoing a long wavelength instability. We first resurrected a numerical method (vortex methods) with Lagrangian adaptivity and a minimal use of computational elements for vortical flows. Vortex methods were in fact the first computational fluid dynamic (CFD) technique employed to study fluids in the 1930s (vortex sheets were computed by hand by Rosenhead and his human 'computers'), and were also used in the first digital computer flow simulations in the 60s and 70s. Vortex methods have since fallen out of favour due to their inaccuracy, which is largely attributed to the distortion of the Lagrangian computational elements. We then addressed the low memory available in massively parallel computer architectures such as the BG/L. This architecture was considered unsuitable for flow simulations requiring large per-processor memory allocations and extensive communications when solving for elliptic differential equations, as imposed here by our velocity-vorticity formulation. The solution to this multiobjective problem was found by reformulating several aspects of vortex methods: coupling particles with grids to regularize their locations, developing novel data structures for the distribution of the particles, and adopting fast Poisson solvers effectively implemented in the BG/L architecture. Last but not least, the development of software libraries and open-source software (see our lab Web site) has enabled the sustained development of these tools over a number of years. How do you look at two billion particles? At the same time, a new challenge has emerged, namely, the visualization and processing of the large data sets that become available through the simulations. Each snapshot of the flow field results in about 100 Gbytes of data, meaning the usual techniques such as flow animation and walkthrough visualization in turn become computationally intensive tasks. The continuum flow field needs to be reconstructed from the particle properties, and high-performance visualization techniques are necessary in order to process the data. We are pursuing this approach in collaboration with scientists at the Swiss National Supercomputing Center (CSCS) in Manno, aiming to provide a unique look at the intricacies of vortex structures in aircraft wakes. A parallel approach involves the use of multiresolution techniques such as wavelets, leading to in situ analysis and structure-informed visualization. From Insight to Optimization Next Steps Links: Please contact: |
ethz.ch