Researchers at University College London (UCL) have been using the UK’s leading supercomputer to model blood flow in the human brain. In the long term, the team hopes that its work might help surgeons decide on appropriate treatment for victims of stroke -- while the patient is on the operating table. The shorter-term goal is to learn how to scale up complex scientific codes to more than 50,000 cores and thus contribute to the co-design of systemware for next-generation Exascale machines.
The UCL team used HemeLB software, a specialised predictive tool for fluid flows in complex geometries, to model the blood flow in the cerebral arterial network and, due to the complexity of their simulations, ran it on ARCHER, the UK's flagship Cray XC30 system. With over 76,000 cores and a performance 1,367.5 TFlop/s, ARCHER is operated by Edinburgh Parallel Computing Centre (EPCC) on behalf of UK researchers.
This particular study formed part of CRESTA – the European Collaborative Research into Exascale Systemware, Tools and Applications. HemeLB employs a lattice Boltzmann (LB) algorithm. In an article for the CRESTA website, ‘HemeLB within CRESTA: computational haemodynamics en route to patient-specific treatment planning using the Exascale’, Timm Kreuger wrote: ‘Due to its locality, the LB algorithm is intrinsically easy to parallelise and therefore a method of choice for HPC. A sophisticated communication approach based on the Message Passing Interface (MPI) allows HemeLB to be run on large supercomputers. Therefore, the HemeLB project fits perfectly into CRESTA, as it stimulates the development of technologies leading to the next generation of supercomputers on the Exascale.’
The CRESTA project focuses on building and exploring appropriate systemware for Exascale platforms – this is achieved largely by partnerships between HPC manufacturers, software providers, and HPC centres so that applications can be co-designed for Exascale.
Among some of the CRESTA partners are the EPCC, UCL and Allinea who were brought together by the EU CRESTA project to solve the challenges of scaling the HemeLB code to 50,000 cores.
The UCL team used Allinea Software's performance profiling tool, Allinea MAP, to increase the performance of HemeLB on the system. Having already improved performance on some test cases by more than 25 per cent they wanted to run a larger simulation.
UCL Post-doctoral Researcher Derek Groen said: ‘We'd never been able to look at this many cores - and get a clear view of how the time was being used - we were keen to see it in Allinea MAP.’
Even though the HemeLB code is designed to be scaled effectively, when scaling the software to 50,000 cores on ARCHER the application would unexpectedly crash. This prompted the use of another tool from Allinea, Allinea DDT.
Groen said: ‘The crash was totally unexpected. I didn't know how I would diagnose or fix it at that scale - it was beyond anything I had tried to do before. Allinea Software helped us straight away - they knew that if we could run the simulation with their debugger, we would find the problem.’
Professor Peter Coveney, Director, Centre for Computational Science, UCL said: ‘Getting HemeLB to scale to 50,000 ARCHER cores is a real achievement. We are thankful for the productive collaborations we enjoy that have allowed us to reach these intoxicating heights, which are enabling us to study hemodynamics within the Circle of Willis for the first time.’
The arrangement of the brain's arteries into the Circle of Willis creates redundancies in the cerebral circulation. If one of the arteries becomes blocked or narrowed, blood flow from the other blood vessels can often preserve the cerebral perfusion well enough to avoid the symptoms of stroke. However, there is great anatomical variation from one individual to another and the redundancies that the Circle of Willis introduce can also lead to reduced cerebral perfusion. Hence the interest in modelling what is going on and in developing a way of providing personalised treatments.
