ONERA: Moving Up the TOP500* List

Intel® technologies boost the performance of France's National Office for Aerospace Studies and Research HPC systems.

At a glance:

  • The mission of France's Office National d'Etudes et de Recherches Aerospatiales (ONERA) is to find ways to advance industrial competitiveness, protect the environment, and enhance the safety and security of air and spacecraft.

  • The new production system is built on Intel® Xeon® Scalable processors interconnected with Cornelis Networks1 Products and provides a powerful solution for specialized research.

author-image

By

Executive Summary

With its mission to find ways to advance industrial competitiveness, protect the environment and enhance the safety and security of air and spacecraft, France’s Office National d’Etudes et de Recherches Aérospatiales (ONERA) relies on a host of high performance computing (HPC) resources. Yet despite the fact that researchers from across seven scientific departments tap into the systems, two departments (the Aerodynamics, Aeroelasticity, Acoustics Department and the Multi-Physics for Energy Department) dealing with computational fluid dynamics and energetics (CFDE) are responsible for nearly 95% of the organization’s CPU usage. What’s more, all of the CPUs at ONERA run proprietary code, so the French aerospace lab is particularly keen on understanding emerging technologies in pursuit of a better software architecture. These considerations came to a head in 2015 when ONERA began considering replacing its main production machine, known as Stelvio, as part of a regular upgrade cycle.

Challenge

ONERA has relied on HPC systems since their early days. “We’ve had all of the vector supercomputers in house, including CRAY 1S, X-MP, Y-MP and C90 systems, as well as NEC SX-4 and SX-8 systems,” explained Alain Refloch, special advisor for HPC at ONERA. “Our first superscalar supercomputer was a Novascale Bull computer with Intel® Itanium® processors [528 cores, 3.4 Tflops] in 2006. We even have an association with Intel going all the way back to 1996 with an Intel® Paragon XP system with i860 processors.” In 2015, ONERA put out a request for proposal (RFP) to replace its Stelvio system, which was an Intel® Xeon® processor-based SGI ICE 8200 supercomputer.

"Our first superscalar supercomputer was a Novascale Bull computer with Intel® Itanium® processors [528 cores, 3.4 Tflops] in 2006. We even have an association with Intel going all the way back to 1996 with an Intel® Paragon XP system with i860 processors.” —Alain Refloch, special advisor for HPC at ONERA

When considering how to best address user needs and processing demands on its new production system, ONERA was primarily interested in additional processing power and architectural consistency. “On the one hand, we were oriented toward an x86 architecture to minimize porting costs as much as possible. Our industry partners using ONERA codes are also on x86 architectures, so it was really an easy decision. We were mostly focused on standard price/performance considerations as we evaluated options,” said Refloch.

Refloch explained that researchers across all seven of ONERA’s scientific departments use its systems for all kinds of research. “For example, the Electromagnetism and Radar Department relies on our systems to study things like electromagnetic compatibility (EMC) and stealth, and our Physics Department performs a lot of lightning studies,” said Refloch. “By far the biggest users of the systems, however, are the CFDE teams”.

The new computer is oriented toward multi-physics studies. For this, it is necessary to work on numerical methodologies to couple multi-physics systems in a stable and conservative way to produce reliable physical results. Important progress has to be made on multi-physics tools and processes in the next ten years.

So the infrastructure of the ONERA multi-physics simulation platform should allow an exploitation of commonality among different physics operators in terms of mesh management, coupling technology, parallelism, communication schedulers, interpolation, error estimation and UQ software, tools for data management, visualization, and I/O and also resilience.

Given the diversity of research projects at ONERA and specialized needs of different departments, ONERA was also looking to deploy a new centralized development system with the same architecture as the new supercomputer that also included specialized nodes.

Solution

In acquiring new systems, ONERA follows French public procurement rules by issuing RFPs. The lab makes its final choice based on a combination of technical considerations, the results of benchmarks, and the price of support and service. “It’s not a big surprise to say that all of the proposals were based on Intel® processors,” commented Refloch. “The main differences were in the number of cores in the processor and the frequency of the processor.” When all was said and done, it took roughly a year from the first meeting of the technical group for drafting the RFPs to getting the new systems in place.

“It’s not a big surprise to say that all of the proposals were based on Intel® processors. The main differences were in the number of cores in the processor and the frequency of the processor.” —Alain Refloch, special advisor for HPC at ONERA

For the centralized development system dubbed SPIRO, ONERA chose an HPE SGI system. This system includes specialized nodes for different ONERA user groups2:

  • 132 nodes running Intel® Xeon® Scalable processors (E5-2650V4 bi-sockets (3168 cores) with 128 GB of memory
  • Six nodes running Intel® Xeon Phi™ 7230 processors
  • One node running two Nvidia* Pascal* GPU processors
  • Three nodes running Nvidia Tesla* K80 GPU processors
  • Five nodes with 256 GB, two nodes with 512 GB, and one with 1 TB memory

For the production system, which was named Sator, ONERA chose NEC HPC1812 Rg 2 servers with Intel® Xeon® processors with 14 cores each. The system includes a total of 17,360 cores as well as a 100Gbps Cornelis Networks high-performance fabrics. In June 2017, the system achieved a Linpack Performance of 579.2 TFlops and a theoretical peak of 667 TFlops, making it the number 341 system on the TOP500 list.3 (Sator moved to 473 in the November 2017 edition of the TOP500 list.4) “The .2 in the Linpack number is important because there are currently three sites between 579 and 580 Tflops,” said Refloch.

Fig 1. ONERA’s NEC HPC1812 Rg 2 system, which it named Sator.

In addition to its in-house resources, ONERA also uses the HPC resources from GENCI (Grand équipement national de calcul intensif) for research, has access to PRACE (Partnership for Advanced Computing in Europe) resources through a preparatory project for exascale computing, and also has access to the CCRT (Center for Computing Research and Technology).

The development system also uses the Cornelis Networks Products. In this case Cornelis Networks Products connects 38 of the compute nodes, supporting up to 100 Gbps per port. A 1 Gbps Ethernet* connects another 94 compute nodes.

ONERA is also in the process of adding six nodes for the network file server and seven for connections, in addition to adding new nodes each year to monitor the evolution of processors.

The new centralized development machine benefits ONERA in several ways, including reducing maintenance requirements and costs. Above all, it provides all ONERA’s development teams common access to the specialized nodes, giving them sufficient computing power to test validation cases, in addition to performing non-regression testing. For the CFDE development teams, sharing the same machine and development tools brings everyone closer together, which is important for advancing ONERA’s roadmap for its CFDE software and its goal of moving toward a common platform for fluid dynamics and energetics.

While convenience and performance were top considerations for the new systems, the objective of having a more centralized development machine also included:

  • Reducing the cost of maintenance
  • Giving access to sufficient computing power to test developments on validation cases and not only on non-regression test
  • Above all, allowing all development teams common access to specialized nodes (large memory, new processors)

While the machine is intended for all ONERA developers and not just CFDE codes, at the same time it was a way to bring together different CFDE development teams by sharing the same machine and the same development tools. This is in line with the ONERA CFDE software roadmap to move towards a common platform.

ONERA’s heritage of CFDE code development is important to the organization’s own research as well as to the work of its industry partners. The lab’s elsA and CEDRE codes, which exceed a million lines, are used by a many of ONERA’s partners, including Airbus, Safran, and ArianeGroup for CFDE-based research. “Today, on one hand, the development time for large codes is longer than the lifetime of the machines. On the other hand, there are many unknowns in anticipating electrical consumption reduction and other factors associated with processor evolution,” explained Refloch. “That’s why, in addition to having cutting-edge machines on hand, we need access as early as possible to new processors. And given these considerations, we also applied to become an Intel® Parallel Computing Center (Intel® PCC) in 2016”.

Result

As before with the Stelvio system, the two CFDE teams will be the primary users of Sator, the NEC machine. Refloch has said that the transition from the SGI Stelvio system to the NEC system is going very smoothly thanks to the continued use of the Intel® processors and compiler. “It’s enough to recompile the part of the codes in direct link with MPI, since we used MPT, the SGI MPI library, on the previous system,” explained Refloch.

Although ONERA is still ramping up usage of the Sator system, Refloch said that the early experiences with the system are very encouraging. “The combination of the Intel® Xeon® processors and Intel® Omni-Path Fabric (Intel® OP Fabric) has proven ideal for the workloads we’ve run so far,” noted Refloch. In one of its first uses of the NEC system, ONERA tested a prototype of a new architecture for elsA, known as Flexible Aerodynamic Solver Technology or FAST. “We showed a 95% scalability on more than 17,000 cores, so our optimization efforts are clearly paying off as the scalability was easy to achieve.” Refloch said that the work ONERA did on vectorizing its FAST application for many cores as part of the IPCC program has helped it accelerate code modernization efforts. He added that Intel’s tools, including Intel® Advisor, have also been a big help. “When it comes to software, profiling is one thing, but interpreting the results and modifying the code accordingly is something else. The Roofline Analysis capability makes it much easier, so it’s a very good thing for developers,” Refloch said.

“The combination of the Intel® Xeon® processors and Intel® Omni-Path Fabric (Intel® OP Fabric) has proven ideal for the workloads we’ve run so far.” —Alain Refloch, special advisor for HPC at ONERA

Overall, the ONERA team is excited about the time and energy savings the NEC system is expected to deliver. “Most users will have access to two times more cores for their usual applications, so in most instances our researchers can expect a significant increase in application performance,” said Refloch.5

Solution Summary

With its new systems, ONERA has the “best of both worlds” from development and production perspectives. The production machine, Sator, with its Intel® Xeon® processors and Cornelis Networks Products, is just what ONERA needed from a price/performance standpoint. It provides a powerful solution for the organization’s specialized research that is a natural step forward from the previous system. And the development machine, SPIRO, with its specialized Intel® Xeon® Scalable processor and Intel® Xeon Phi™ processors, and GPU nodes, will help ensure developers are staying abreast of the latest technologies.

Solution Ingredients

  • SPIRO—HPE SGI system with more than 3,000 Intel® Xeon® processor cores, as well as specialized nodes that include Intel® Xeon Phi™ processors, Intel® Xeon® Scalable processors, and GPUs
  • Sator—NEC HPC1812 Rg 2 servers with Intel® Xeon® processors E5-2680 v4 14c 2.4Ghz processors, 17,360 cores, and Intel® Omni-Path Fabric (Intel® OP Fabric)

Download the PDF ›

Product and Performance Information

1

Intel has spun out the Omni-Path business to Cornelis Networks, an independent Intel Capital portfolio company. Cornelis Networks will continue to serve and sell to existing and new customers by delivering leading purpose-built high-performance network products for high performance computing and artificial intelligence. Intel believes Cornelis Networks will expand the ecosystem of high-performance fabric solutions, offering options to customers building clusters for HPC and AI based on Intel® Xeon™ processors. Additional details on the divestiture and transition of Omni-Path products can be found at www.cornelisnetworks.com.

2

Supercomputing facilities, ONERA.fr, 2017.

3

Sator - NEC HPC1812-Rg-2, Intel® Xeon® E5-2680v4 processor 14C 2.4GHz, 100 Gbps Intel® Omni-Path Architecture (Intel® OPA). The system includes a total of 17,360 cores. TOP500.org, June 2017.

4

Sator - NEC HPC1812-Rg-2, Intel® Xeon® E5-2680v4 processor 14C 2.4GHz, Intel® Omni-Path Architecture (Intel® OPA). The system includes a total of 17,360 cores. TOP500.org, November 2017.

5

Performance varies by use, configuration and other factors. Learn more at www.Intel.com/PerformanceIndex.