Intel® MPI Library

Developer Guide for Linux* OS

Download PDF
ID 768728
Date 6/30/2025
Public
Document Table of Contents
Document Table of Contents x
Developer Guide for Linux* OS
Developer Guide for Linux* OS x
Introduction Installation and Prerequisites Compiling and Linking Running Applications Debugging Applications Analysis and Tuning Troubleshooting Additional Supported Features Examples Notices and Disclaimers
Introduction x
Introducing Intel® MPI Library Conventions and Symbols Related Information
Installation and Prerequisites x
Installation Prerequisite Steps
Compiling and Linking x
Compiling an MPI Program Compilers Support ILP64 Support
Running Applications x
Running Intel® MPI Library in Containers Selecting a Library Configuration Running an MPI Program Running an MPI/OpenMP* Program MPMD Launch Mode Fabrics Control Job Schedulers Support Controlling Process Placement Java* MPI Applications Support
Running Intel® MPI Library in Containers x
Build a Singularity* Container for an MPI Application Run the Application with a Container
Fabrics Control x
Selecting Fabrics Libfabric* Support OFI* Providers Support
Debugging Applications x
Debugging Using -gtool for Debugging
Analysis and Tuning x
Displaying MPI Debug Information Tracing Applications Interoperability with Other Tools Through -gtool MPI Tuning
Troubleshooting x
Error Message: Bad Termination Error Message: No such file or Directory Error Message: Permission Denied Error Message: Fatal Error Error Message: Bad File Descriptor Error Message: Too Many Open Files Error Message: Too Many Communicators Problem: High Memory Consumption Readings Problem: MPI Application Hangs Problem: Password Required Problem: Cannot Execute Binary File Problem: MPI limitation for Docker*
Additional Supported Features x
Asynchronous Progress Control Device-Initiated Communications Notified One-Sided Communications Multiple Endpoints Support
Multiple Endpoints Support x
MPI_THREAD_SPLIT Programming Model Threading Runtimes Support Program Examples Code Change Guide
Examples x
async_progress_sample.c thread_split.cpp thread_split_omp_for.c thread_split_omp_task.c thread_split_pthreads.c
Developer Guide for Linux* OS

Error Message: Too Many Communicators

Error Message

Abort(136403471) on node 0 (rank 0 in comm 0): Fatal error in internal_Comm_dup: Other MPI error, error stack:
internal_Comm_dup(24268)............: MPI_Comm_dup(MPI_COMM_WORLD, newcomm=0x75fe1d097008) failed
MPIR_Comm_dup_impl(738).............:
MPII_Comm_dup(1207).................:
MPII_Comm_copy(1249)................:
MPIR_Get_contextid_sparse_group(609): Too many communicators (0/32768 free on this process; ignore_id=0)

Cause

MPI has a limitation on how many communicators can be created. The maximum number of communicators that can be created is 32766, with MPI_COMM_WORLD and MPI_COMM_SELF consuming allocations. Remote Memory Access (RMA) type windows will also affect this number because each window that has been created consumes a communicator for internal needs. The actual number of communicators that can be created may be less if an application creates communicators with similar groups but varying communicator IDs.

NOTE:
This is one example of an error caused by no free communicators. If you have overridden error-handling logic, you may only receive an error code but no backtrace.

Solution

To fix this issue, it is recommended that any unused MPI objects be freed using the function MPI_COMM_FREE.

Parent topic: Troubleshooting
Level Two Title