Hokkaido University: HPC Intercloud Infrastructure

Intel® Xeon® Scalable Processors and Cornelis Networks products1 help boost HPC computational capability.

At a Glance:

  • Hokkaido University Information Initiative Center provides HPC and cloud computing services to researchers at the institution and across Japan as part of the country’s HPC Infrastructure Initiative.

  • To support advanced research projects employing large-scale distributed systems with high-speed networks across Japan, Hokkaido University built a distributed cloud infrastructure with Intel® Xeon® Scalable processors and Cornelis Networks products.

author-image

By

Executive Summary

Japan’s Hokkaido University Information Initiative Center provides High Performance Computing (HPC) and cloud computing services to researchers at the institution and across Japan. The university’s HPC resources are connected with other supercomputers around the country as part of its High Performance Computing Infrastructure (HPCI) initiative. To continue to support insight and innovation in computational research, the Information Initiative Center increased research supercomputing capacity 23X with installation in December 2018 of Grand Chariot (3.08 petaFLOPS) based on Intel® Xeon® Gold 6148 processors and Polaire (0.87 petaFLOPS) built on Intel® Xeon Phi™ 7250 processors—both interconnected by Cornelis Networks fabric.

Challenge

Hokkaido University promotes world-class computational research to solve the problems confronting humankind. The University’s Information Initiative Center takes a major role to support the activity by providing large-scale computing and networking services to researchers. The typical research run on Hokkaido University’s supercomputers includes ocean analysis, electromagnetic field analysis, weather simulation, computational chemistry, and others.

Since 2011, the university has supported scholarly studies in cloud services to promote research for the practical use of intercloud systems. Combining the results of this research and its need to expand supercomputing capacity, in 2018 it developed an interdisciplinary large-scale computer system called the High Performance Intercloud. The Intercloud includes a supercomputer system and a cloud system.

Solution

Hokkaido University’s interdisciplinary supercomputer comprises two clusters totaling nearly four petaFLOPS of theoretical performance.2 To support supercomputing applications in various scientific domains, Hokkaido University deployed a 3.08 petaFLOPS cluster called Grand Chariot. The supercomputer uses 1,004 nodes of Fujitsu PRIMERGY* CX2550 with Intel Xeon Gold 6148 processors (40 cores/node). Grand Chariot ranked 95 in the November 2018 Top500.3 Furthermore, considering the trend of HPC technology, the Information Initiative Center installed a 288-node cluster called Polaire. It uses Fujitsu PRIMERGY* servers with Intel Xeon Phi 7250 processors. Polaire will be used to develop an advanced simulation code, which efficiently utilizes many-core processors.

Both clusters are interconnected by 100 Gbps Cornelis Networks fabric and supported by a Data Direct Networks 16 petabyte Lustre* storage system. Each node in Grand Chariot includes two Cornelis Networks host fabric adapters to support increased injection bandwidth and to ensure continued job execution if a switch fails.

Interdisciplinary supercomputer clusters: Grand Chariot and Polaire

To support advanced research projects employing large-scale distributed systems with high-speed networks across Japan, Hokkaido University built a distributed cloud infrastructure stretching from the Information Initiative Center’s data center to Kyushu. The distributed Intercloud system comprises groups of servers installed at Hokkaido University, University of Tokyo, Osaka University, and Kyushu University. Archive storage was installed at Kitami Institute of Technology. Internally, a 25 Gbps Ethernet* network connects the cloud servers. The entire cloud is connected to Japan’s Science Information Network (SINET5).

The cloud system uses 71 servers built on Intel® Xeon® Gold 6138 processors. Each group of servers is configured to support different types of workloads and use cases. Users can request a bare-metal server with all the resources on the platform to run their applications. Virtual environments are also available to create isolated virtual machines. Users can select the amount of resources they need, and the Intercloud system will deploy the environment for their use. For GPU-based applications, researchers and students can request a physical server with two GPUs. The variety of configurable environments gives users a wide range of possible platforms on which to run their workloads.

Hokkaido University intercloud servers

Results
With the supercomputer and cloud systems located close to each other, the infrastructure has special characteristics appropriate for collaboration with Internet of Things (IoT) technology and analysis of sensor data. The Information Initiative Center expects advanced research to leverage such collaboration in IoT, analytics, and machine learning.

Solution Summary

  • Intel Xeon Gold 6148 processors (40,160 cores total)
  • Intel Xeon Phi 7250 processor (19,584 cores total)
  • Intel Xeon Gold 6138 processor (2,840 cores total)
  • Cornelis Networks fabric (100 Gbps)

Explore Related Products and Solutions

Product and Performance Information

1

Intel has spun out the Omni-Path business to Cornelis Networks, an independent Intel Capital portfolio company. Cornelis Networks will continue to serve and sell to existing and new customers by delivering leading purpose-built high-performance network products for high performance computing and artificial intelligence. Intel believes Cornelis Networks will expand the ecosystem of high-performance fabric solutions, offering options to customers building clusters for HPC and AI based on Intel® Xeon™ processors. Additional details on the divestiture and transition of Omni-Path products can be found at www.cornelisnetworks.com.

2Testing completed by Hikkaido University on December 22, 2018. Based on theoretical peak performance of 3.08 +0.87 pF listed at https://www. hucc.hokudai.ac.jp/en/supercomputer/sc-overview/ for both systems and Top500 performance of previous system (0.168907 pF) shown at https://www.top500.org/system/177462. Previous computer configuration listed at http://www.hitachi.co.jp/New/cnews/month/2011/10/1031.html. Grand Chariot system configuration, 3.08 pFLOPS: 1,004 nodes. PRIMERGY CX2550 M4, PRIMERGY CX400 M4, Intel Xeon Gold 6148 pro¬cessors (20 cores/40 cores per node), 384 GB, 240GB SSD, Intel OPA fabric at 100 Gbps. Polaire system configuration, .87 pFLOPS: 288 nodes. PRIMERGY CX1640 M1, PRIMERGY CX600 M1, Intel Xeon Phi 7250, 16 GB MCDRAM (96 GB total), 64 GB SATA, Intel OPA at 100 Gbps. Previous system configuration, 170 teraFLOPS: 114 nodes. Hitachi BS2000 servers, Intel® Xeon® processor E7-8870, 128GB, 10 Gbps Ethernet; Hitachi SR16000 M1 server, Power7* processor.