We’re on the forefront of converging artificial intelligence (AI), analytics, simulation and modeling, and other high-performance computing (HPC) workloads that will drive the industry toward the next era in supercomputing.
Intel’s HPC platforms deliver exceptional performance and scale with industry-first innovations in memory, computational performance, fabric and storage. In fact, a record 95 percent of all systems in the February 2018 Top500 list of the world’s leading supercomputers are powered by Intel® Xeon® processors. And next-generation Cornelis Networks coming in 2019 will feature data rate speeds up to 200 Gb/second, doubling current performance levels.
Those high-performance/low-latency capabilities at scale provide system architects the ability to use tens of thousands of nodes, while visualization allows systems to deliver greater insights with faster turnaround when using large-scale data sets.
New HPC Challenges for Developers
Today’s HPC challenges stem from burgeoning scale, greater heterogeneity, mounting energy costs and explosive new uses in AI and commercial analytics. With all of that growing complexity, it is not surprising that a recent IDC survey of HPC users1 found that the software stack is the biggest pain point of any technical category—specifically, updating and using system software, including operating systems and middleware.
Programmers and system administrators need more automation and intelligence built-in to the HPC software stack, but the growing scope and rate of change makes it extremely difficult for individual OEMs or other vendors to keep up.
Open Source Software Solutions for HPC
Fortunately for the developer community, the Linux* model maps well to addressing these challenges: an open source software (OSS) community that contributes source code upstream, and specialized downstream distributors that support specific customer needs. The OSS model helps the community:
- Keep pace with user requirements by delivering more new capabilities to users more quickly than proprietary commercial software.
- Provide developers faster access to new capabilities, including features that automate labor-intensive work.
- Enable vendors to select open source components for baseline functions while focusing more R&D at providing competitive differentiation and supporting growing customer demands.
The OSS community represents the most effective way to advance the software stack beyond the limitations of any one vendor.
Several open source HPC software components are used by the HPC community today: Open MPI, OpenSFS, OpenFOAM, OpenStack, and others. And nearly all major OEMs are participants in the OpenHPC community, as are many key HPC independent software vendors (ISVs). Dozens of the world’s top HPC user sites have also joined this collaboration as participants.
Intel Contributions across OSS Community
As the leading contributor to the Linux* kernel and many OSS projects and initiatives, Intel has been a catalyst for the OSS community in general, and specifically for the widely-supported OpenHPC initiative. Intel donated hardware to the Texas Advanced Computing Center (TACC) to build a continuous integration environment for developing OpenHPC open source versions. This build-and-test environment integrates new elements into the foundational layer of OpenHPC and uses the OpenHPC test suite to ensure that new elements are compliant and performant.
This collaborative development and testing model helps advance the ease-of-use of HPC systems, making them more productive and useful in a broader scope of uses, such as commercial analytics. By addressing these requirements in the software stack, Intel has paved the way for broader adoption of HPC for high performance data analytics (HPDA)—a market IDC forecasts to grow at about three times the rate of the overall HPC market through 20201 —and for AI solutions that promise to solve some of the biggest challenges in healthcare, scientific discovery and industrial innovation.
Intel’s HPC platform supports these and other HPC solutions with Intel Xeon Scalable processors, Cornelis* Omni-Path interconnect fabric and other elements. Intel® Select Solutions for HPC offer tested and verified configurations specifically designed for use cases such as professional visualization and simulation & modeling.
Opening the Door to AI with HPC
In her recent article for Data Center Knowledge, Intel’s general manager of Rack Scale Design, Dr. Figen Ülgen noted that, “…many aspects of model training—a foundational tool for many deep learning and AI use cases—require the computational horsepower and throughput of HPC.” But she points out that “…for many data scientists and domain experts who want to incorporate AI capabilities into their business or research projects, HPC resources are too difficult to access.” That was a key motivation for Dr. Ülgen to get involved with the establishment of OpenHPC. Since its launch by the Linux Foundation in November 2015, OpenHPC has grown to include more than 30 organizations working together on an open, vendor-neutral software stack for HPC infrastructure. OpenHPC fosters innovation as a virtual operating system for HPC software on multiple platforms running Linux. The OpenHPC repository offers pre-built software ingredients, as well as validated recipes. The stack includes Linux operating systems, I/O libraries and services, numerical and scientific libraries, provisioning and management tools, and other elements that reduce the work of creating and maintaining HPC software systems while exposing the parallel power of the hardware. As Dr. Ülgen concludes, “This is an exciting time for the AI and HPC communities. By embracing HPC’s power to advance AI and by using the OpenHPC stack to facilitate access to HPC resources, we can help create a smarter world and realize the vision of the AI revolution.” |
Figen Ulgen PhD,Showcasing AI & HPC Innovations at SC18
At the Supercomputing 2018 conference (SC18), being held Nov. 11-15, 2018, in Dallas, Intel will showcase the convergence of AI and HPC on Intel Xeon Scalable processor-based systems and other Intel technologies in the following demos:
- 3D AI: Because the World Isn’t Flat
- Platform-Independent AI Inferencing with Intel® OpenVINO™ Toolkit
- First Look: Cascade Lake with Intel® DLBoost™ Technology
- DIY Workload Acceleration with Intel® FPGAs
- First Look: Apache* Spark* on Intel® Optane™ DC Persistent Memory
- Cloudified AI with Containers
- Versatile Memory & Storage with Intel® Optane™ SSDs
- Intel® Rendering Framework in Action
- Look Ahead: A Whole New Level of Performance
- Quantum Leap: Intel® Quantum & Neuromorphic Computing
In addition, Intel HPC experts and AI specialists will be giving technical talks with our ecosystem partners on a variety of HPC and AI-related topics in the Intel booth theater (#3223) throughout the conference. Our experts will also be featured in more than 50 paper presentations, workshops, tutorials and Birds of a Feather discussions (full list of sessions available here). Be sure to catch the following AI and open source sessions:
And, join us at the One Intel Station developer community event taking place Nov. 12-14 at the historical Union Station, just a five-minute walk from the conference. Experience demos, tutorials and technical sessions, as well as free refreshments all day long. No registration is required.
We look forward to meeting with you at SC18 and discussing how the latest open source and AI software development tools for HPC can accelerate your path to deeper discovery and insights. For more information on Intel® HPC platform software and Intel Select Solutions for HPC, click the links below:
Editor's Note: Reese Baird contributed to this article.
1 From IDC Technology Spotlight June 2016: The Open HPC Stack Initiative Hits a Milestone