Improving Vectorization of OpenMP* Collapse Loop Nests

Intel® Fortran Compiler

Developer Guide and Reference

Download PDF

ID 767251

Date 10/15/2026

Version

Public

Improving Vectorization of OpenMP* Collapse Loop Nests

The collapse clause on an OpenMP* worksharing loop is handled by the compiler for CPU by merging a loop nest into a single iteration space so that more iterations are available to distribute across threads. This usually increases the amount of parallelism and improves load balancing, but it can work against vectorization in some cases: after the loops are merged, the compiler code generation may introduce divide and modulo operations inside the loop that uses the collapsed loop index to recover the loop-indices of the original loops. This may result in less efficient vectorization if the collapsed loop becomes the innermost vectorized loop.

The -fopenmp-vector-aware-collapse (Linux) or /Qopenmp-vector-aware-collapse (Windows) option lets the compiler collapse one loop level less than the requested collapse level, leaving the innermost loop untouched by collapsing so that it can be efficiently vectorized. Threading is still applied to the remaining collapsed loops and may reduce the total amount of parallelism. Program results are unchanged; only the distribution of iterations across threads and the shape of the loop nest in the generated code change.

The option is off by default because its profitability depends on whether the loop nest with reduced collapse level can still be effectively parallelized across the available threads on the CPU. The recommended approach is to evaluate the option and confirm that it leads to a performance gain.

A Simple Example: 3-Level Collapsed Loop Nest

The testcase below evaluates a degree-4 polynomial over a three-dimensional array, with collapse(3) on a three-deep nest. The kernel is repeated so the timing is stable, and the result array is summed at the end so that no part of the computation can be optimized away.

Fortran source, collapse_poly.f90:


program collapse_poly
  use omp_lib
  implicit none

  integer, parameter :: ni = 1024, nj = 64, nk = 16
  integer, parameter :: nrep = 9600
  real, allocatable :: a(:,:,:), b(:,:,:)
  real, parameter :: c0 = 1.0, c1 = 0.5, c2 = 0.25, c3 = 0.125, c4 = 0.0625
  real :: x
  real(kind=8) :: t0, t1, s
  integer :: i, j, k, r

  allocate(a(ni,nj,nk), b(ni,nj,nk))

  do k = 1, nk
    do j = 1, nj
      do i = 1, ni
        b(i,j,k) = real(mod(i + j + k, 17)) * 0.125
      end do
    end do
  end do

  ! Warm up so the timed loop is not measuring first-touch page faults.
  do r = 1, 5
    !$omp parallel do collapse(3) private(x)
    do k = 1, nk
      do j = 1, nj
        do i = 1, ni
          x = b(i,j,k)
          a(i,j,k) = (((c4 * x + c3) * x + c2) * x + c1) * x + c0
        end do
      end do
    end do
    !$omp end parallel do
  end do

  t0 = omp_get_wtime()
  do r = 1, nrep
    !$omp parallel do collapse(3) private(x)
    do k = 1, nk
      do j = 1, nj
        do i = 1, ni
          x = b(i,j,k)
          a(i,j,k) = (((c4 * x + c3) * x + c2) * x + c1) * x + c0
        end do
      end do
    end do
    !$omp end parallel do
  end do
  t1 = omp_get_wtime()

  s = sum(real(a, kind=8))

  print '(a,i0)',     'threads = ', omp_get_max_threads()
  print '(a,f9.2,a)', 'time    = ', 1.0d3 * (t1 - t0), ' ms'
  print '(a,es14.6)', 'checksum= ', s

  deallocate(a, b)
end program collapse_poly

In this nest, when the option is enabled, the compiler will leave the innermost loop out of the collapsed iteration space. The outer two loops (k, j) still supply 16 x 64 = 1024 iterations to distribute across threads while the 1024-iteration innermost i-loop is vectorized.

Compiling and Running the Example

The testcase is built twice, once with the option disabled and once with the option enabled.

Linux:


ifx -O2 -qopenmp -xCORE-AVX512 -fno-openmp-vector-aware-collapse collapse_poly.f90 -o f90_off
ifx -O2 -qopenmp -xCORE-AVX512 -fopenmp-vector-aware-collapse collapse_poly.f90 -o f90_on

Windows:


ifx /O2 /Qopenmp /QxCORE-AVX512 /Qopenmp-vector-aware-collapse- collapse_poly.f90 -o f90_off
ifx /O2 /Qopenmp /QxCORE-AVX512 /Qopenmp-vector-aware-collapse collapse_poly.f90 -o f90_on

Each binary is run with the same thread count and thread placement, so that the performance difference is mainly due to the difference in the compiler code generation:


export OMP_NUM_THREADS=16
export KMP_AFFINITY=granularity=fine,compact,1,0

numactl --cpunodebind=0 --membind=0 ./f90_off
numactl --cpunodebind=0 --membind=0 ./f90_on

The reported checksum values can be compared to confirm that the results are unchanged.

Measured Results

Total time for 9600 iterations of the kernel was measured. The time reported for each binary is the fastest of seven alternating runs:

Language	With option disabled	With option enabled	Speedup
Fortran (ifx)	758.18 ms	63.96 ms	11.9x

NOTE:

This kernel does a lot of arithmetic on a working set that fits in cache. A loop whose speed is limited by memory bandwidth cannot gain much from better vector code and may limit the performance gain due to this option. The performance gain shown here is for demonstration purposes only. The real gain will depend on the application characteristics.

Hardware and Software Configuration

Processor	2 x Intel® Xeon® Platinum 8480L
Microarchitecture	Sapphire Rapids
Cores and threads	56 cores per socket, 2 threads per core, 224 logical processors
Max turbo	3.80 GHz
Caches	48 KB L1d and 2 MB L2 per core; 105 MB L3 per socket
Vector ISA	Intel® AVX-512 (F, DQ, IFMA, CD, BW, VL, BF16, VBMI, VBMI2, VNNI, BITALG, VPOPCNTDQ, FP16)
Memory	503 GiB, 4 NUMA nodes
Operating system	Ubuntu 24.04 LTS, kernel 6.8.0-59-generic
Compiler	Intel® Fortran Compiler 2026.2.0
Compiler options	-O2 -qopenmp -xCORE-AVX512
Threads	OMP_NUM_THREADS=16 KMP_AFFINITY=granularity=fine,compact,1,0
Working set	two float arrays of 16 x 64 x 1024 elements, 4 MB each

The 16 threads and the data were bound to a single socket, so the measurement is not affected by cross-socket memory traffic. Performance varies by use, configuration, and other factors.

Confirming That the Optimization Was Applied

Add the optimization report at level 2 or higher to the same compilation:


ifx -O2 -qopenmp -xCORE-AVX512 -fopenmp-vector-aware-collapse \
    -qopt-report=2 -qopt-report-file=stdout -c collapse_poly.f90

When the option is enabled, the report shows the collapse count specified by the user, the reason it was adjusted down and the properties of the vectorized innermost loop that is now not included as part of the collapsing. For the timed loop of the Fortran example, at line 39:


OMP PARALLEL DO BEGIN at collapse_poly.f90 (39, 11)
    remark #30014: OpenMP: 2 loops collapsed

    LOOP BEGIN at collapse_poly.f90 (41, 7)
        remark #30042: OpenMP: Collapse clause argument reduced from 3 to 2 to preserve innermost loop for efficient vectorization
        remark #30000: OpenMP: Outlined parallel loop
        remark #15541: loop was not vectorized: outer loop is not an auto-vectorization candidate.

        LOOP BEGIN at collapse_poly.f90 (42, 9)
            remark #15300: LOOP WAS VECTORIZED
            remark #15305: vectorization support: vector length 8
            remark #25438: Loop unrolled without remainder by 2 by LLVM community unroller
        LOOP END
    LOOP END
OMP PARALLEL DO END

The default Fortran compilation is the clearest illustration of why LOOP WAS VECTORIZED alone is not enough to judge a collapsed nest. It reports the loop as vectorized at vector length 16, twice the vector length of the version with the option enabled, and it is the slower of the two binaries measured above:


OMP PARALLEL DO BEGIN at collapse_poly.f90 (39, 11)
    remark #30014: OpenMP: 3 loops collapsed

    LOOP BEGIN at collapse_poly.f90 (41, 7)
        remark #30000: OpenMP: Outlined parallel loop
        remark #15300: LOOP WAS VECTORIZED
        remark #15305: vectorization support: vector length 16
    LOOP END
    ...
OMP PARALLEL DO END

Guidance on When to Use It

The best way to apply OpenMP* collapse clause is to apply it only on outer loops so that the innermost loops can be efficiently vectorized. If this is not possible then the user may want to evaluate the use of this option to improve application performance.

Leaving the innermost loop out of the collapsed iteration space reduces the number of iterations available for threading. If the remaining loops do not supply enough iterations to keep all threads busy, enabling this option can hurt performance rather than help it. Nests whose outer loops have small trip counts and whose innermost loop is long are the cases to check most carefully.

Notes

Applies to host code generation only; device code for OpenMP* offload is unaffected.
Requires OpenMP* to be enabled, with -qopenmp or /Qopenmp.
Does not apply to every collapse nest. Remark #30042 in the optimization report tells you which nests it affected.

Select Your Language

Using Intel.com Search

Quick Links

Recent Searches

Advanced Search

Only search in

Developer Guide and Reference

Improving Vectorization of OpenMP* Collapse Loop Nests

A Simple Example: 3-Level Collapsed Loop Nest

Compiling and Running the Example

Confirming That the Optimization Was Applied

Guidance on When to Use It

Notes

See Also