Intel® oneAPI DPC++/C++ Compiler
Developer Guide and Reference
Improving Vectorization of OpenMP* Collapse Loop Nests
The collapse clause on an OpenMP* worksharing loop is handled by the compiler for CPU by merging a loop nest into a single iteration space so that more iterations are available to distribute across threads. This usually increases the amount of parallelism and improves load balancing, but it can work against vectorization in some cases: after the loops are merged, the compiler code generation may introduce divide and modulo operations inside the loop that uses the collapsed loop index to recover the loop-indices of the original loops. This may result in less efficient vectorization if the collapsed loop becomes the innermost vectorized loop.
The -fopenmp-vector-aware-collapse (Linux) or /Qopenmp-vector-aware-collapse (Windows) option lets the compiler collapse one loop level less than the requested collapse level, leaving the innermost loop untouched by collapsing so that it can be efficiently vectorized. Threading is still applied to the remaining collapsed loops and may reduce the total amount of parallelism. Program results are unchanged; only the distribution of iterations across threads and the shape of the loop nest in the generated code change.
The option is off by default because its profitability depends on whether the loop nest with reduced collapse level can still be effectively parallelized across the available threads on the CPU. The recommended approach is to evaluate the option and confirm that it leads to a performance gain.
A Simple Example: 3-Level Collapsed Loop Nest
The testcase below evaluates a degree-4 polynomial over a three-dimensional array, with collapse(3) on a three-deep nest. The kernel is repeated so the timing is stable, and the result array is summed at the end so that no part of the computation can be optimized away.
C source, collapse_poly.c:
#include <stdio.h>
#include <omp.h>
#define NK 16
#define NJ 64
#define NI 1024
#define NREP 9600
static float a[NK][NJ][NI];
static float b[NK][NJ][NI];
int main(void) {
for (int k = 0; k < NK; k++)
for (int j = 0; j < NJ; j++)
for (int i = 0; i < NI; i++)
b[k][j][i] = (float)((i + j + k) % 17) * 0.125f;
const float c0 = 1.0f, c1 = 0.5f, c2 = 0.25f, c3 = 0.125f, c4 = 0.0625f;
/* Warm up so the timed loop is not measuring first-touch page faults. */
for (int r = 0; r < 5; r++) {
#pragma omp parallel for collapse(3)
for (int k = 0; k < NK; k++)
for (int j = 0; j < NJ; j++)
for (int i = 0; i < NI; i++) {
float x = b[k][j][i];
a[k][j][i] = (((c4 * x + c3) * x + c2) * x + c1) * x + c0;
}
}
double t0 = omp_get_wtime();
for (int r = 0; r < NREP; r++) {
#pragma omp parallel for collapse(3)
for (int k = 0; k < NK; k++)
for (int j = 0; j < NJ; j++)
for (int i = 0; i < NI; i++) {
float x = b[k][j][i];
a[k][j][i] = (((c4 * x + c3) * x + c2) * x + c1) * x + c0;
}
}
double t1 = omp_get_wtime();
double sum = 0.0;
for (int k = 0; k < NK; k++)
for (int j = 0; j < NJ; j++)
for (int i = 0; i < NI; i++)
sum += a[k][j][i];
printf("threads = %d\n", omp_get_max_threads());
printf("time = %.2f ms\n", 1.0e3 * (t1 - t0));
printf("checksum= %.6e\n", sum);
return 0;
}
In this nest, when the option is enabled, the compiler will leave the innermost loop out of the collapsed iteration space. The outer two loops (k, j) still supply 16 x 64 = 1024 iterations to distribute across threads while the 1024-iteration innermost i-loop is vectorized.
Compiling and Running the Example
The testcase is built twice, once with the option disabled and once with the option enabled.
Linux:
icx -O2 -qopenmp -xCORE-AVX512 -fno-openmp-vector-aware-collapse collapse_poly.c -o c_off
icx -O2 -qopenmp -xCORE-AVX512 -fopenmp-vector-aware-collapse collapse_poly.c -o c_on
Windows:
icx /O2 /Qopenmp /QxCORE-AVX512 /Qopenmp-vector-aware-collapse- collapse_poly.c -o c_off
icx /O2 /Qopenmp /QxCORE-AVX512 /Qopenmp-vector-aware-collapse collapse_poly.c -o c_on
Each binary is run with the same thread count and thread placement, so that the performance difference is mainly due to the difference in the compiler code generation:
export OMP_NUM_THREADS=16
export KMP_AFFINITY=granularity=fine,compact,1,0
numactl --cpunodebind=0 --membind=0 ./c_off
numactl --cpunodebind=0 --membind=0 ./c_on
The reported checksum values can be compared to confirm that the results are unchanged.
Measured Results
Total time for 9600 iterations of the kernel was measured. The time reported for each binary is the fastest of seven alternating runs:
| Language | With option disabled | With option enabled | Speedup |
|---|---|---|---|
| C (icx) | 469.32 ms | 72.87 ms | 6.4x |
Hardware and Software Configuration
| Processor | 2 x Intel® Xeon® Platinum 8480L |
| Microarchitecture | Sapphire Rapids |
| Cores and threads | 56 cores per socket, 2 threads per core, 224 logical processors |
| Max turbo | 3.80 GHz |
| Caches | 48 KB L1d and 2 MB L2 per core; 105 MB L3 per socket |
| Vector ISA | Intel® AVX-512 (F, DQ, IFMA, CD, BW, VL, BF16, VBMI, VBMI2, VNNI, BITALG, VPOPCNTDQ, FP16) |
| Memory | 503 GiB, 4 NUMA nodes |
| Operating system | Ubuntu 24.04 LTS, kernel 6.8.0-59-generic |
| Compiler | Intel® oneAPI DPC++/C++ Compiler 2026.2.0 |
| Compiler options | -O2 -qopenmp -xCORE-AVX512 |
| Threads | OMP_NUM_THREADS=16 KMP_AFFINITY=granularity=fine,compact,1,0 |
| Working set | two float arrays of 16 x 64 x 1024 elements, 4 MB each |
The 16 threads and the data were bound to a single socket, so the measurement is not affected by cross-socket memory traffic. Performance varies by use, configuration, and other factors.
Confirming That the Optimization Was Applied
Add the optimization report at level 2 or higher to the same compilation:
icx -O2 -qopenmp -xCORE-AVX512 -fopenmp-vector-aware-collapse \
-qopt-report=2 -qopt-report-file=stdout -c collapse_poly.c
When the option is enabled, the report shows the collapse count specified by the user, the reason it was adjusted down and the properties of the vectorized innermost loop that is now not included as part of the collapsing. For the timed loop of the C example, at line 33:
OMP PARALLEL FOR BEGIN at collapse_poly.c (33, 1)
remark #30014: OpenMP: 2 loops collapsed
LOOP BEGIN at collapse_poly.c (33, 1)
remark #30042: OpenMP: Collapse clause argument reduced from 3 to 2 to preserve innermost loop for efficient vectorization
remark #30000: OpenMP: Outlined parallel loop
remark #15541: loop was not vectorized: outer loop is not an auto-vectorization candidate.
LOOP BEGIN at collapse_poly.c (33, 1)
remark #15300: LOOP WAS VECTORIZED
remark #15305: vectorization support: vector length 8
remark #25438: Loop unrolled without remainder by 2 by LLVM community unroller
LOOP END
LOOP END
OMP PARALLEL FOR END
Compiling the same source without the option reports the full collapse of the 3-level loop nest:
OMP PARALLEL FOR BEGIN at collapse_poly.c (33, 1)
remark #30014: OpenMP: 3 loops collapsed
LOOP BEGIN at collapse_poly.c (33, 1)
remark #30000: OpenMP: Outlined parallel loop
remark #15300: LOOP WAS VECTORIZED
remark #15305: vectorization support: vector length 8
LOOP END
LOOP BEGIN at collapse_poly.c (33, 1)
<Remainder loop for vectorization>
LOOP END
OMP PARALLEL FOR END
Guidance on When to Use It
The best way to apply OpenMP* collapse clause is to apply it only on outer loops so that the innermost loops can be efficiently vectorized. If this is not possible then the user may want to evaluate the use of this option to improve application performance.
Leaving the innermost loop out of the collapsed iteration space reduces the number of iterations available for threading. If the remaining loops do not supply enough iterations to keep all threads busy, enabling this option can hurt performance rather than help it. Nests whose outer loops have small trip counts and whose innermost loop is long are the cases to check most carefully.
Notes
Applies to host code generation only; device code for OpenMP* offload is unaffected.
Requires OpenMP* to be enabled, with -qopenmp or /Qopenmp.
Does not apply to every collapse nest. Remark #30042 in the optimization report tells you which nests it affected.
See Also
fopenmp-vector-aware-collapse, Qopenmp-vector-aware-collapse compiler option
fopenmp-targets, Qopenmp-targets compiler option
fiopenmp, Qiopenmp compiler option
fopenmp compiler option