RELEASED: HPE Cray Supercomputer Programming Environment 24.03 General Availability Release for HPE Cray EX and HPE Cray Supercomputer Systems with CSM 1.5 COS 23.11 SLES15 SP5 – May 2024

PRODUCT DESCRIPTION

The HPE Cray Programming Environment 24.03 release for HPE Cray EX and HPE Cray Supercomputer systems is now available for CSM 1.5 COS 23.11 SLES15 SP5 systems.

NOTE: COS 2.4 (based on SLES 15 SP4) is not supported in this release. COS 2.4 users must use the CPE 23.09 (or earlier) release.

SOFTWARE OVERVIEW

This software announcement applies to HPE Cray EX and/or HPE Cray Supercomputer CSM 1.5 COS 23.11 SLES15 SP5 systems.

HPE Cray Programming Environment licensed customers may download the release materials by submitting a request to HPE PointNext for the package below (depending on entitlement):

HPE Cray Programming Environment 24.03 for CSM

CPE support for specific HPE Cray EX recipes may be found in the following documents which are maintained for each recipe on the HPE Support Center:

HPE Cray EX System Software Getting Started Guide (S-8052)
HPE Cray Software Compatibility Matrixes Reference Guide (S-8042)

See “module help <product>” for the corresponding PE product release notes.

Content for CSM 1.5 COS 23.11 SLES15 SP5 Systems:

cpe-24.03-sles15-sp4-csm-24.3.2.tar.gz

NOTE: The single tarfile(s) above contain base PE as well as optional installation of other third-party compiler support.

REQUIRED: Base PE
OPTIONAL:

AMD AOCC 4.1.0

AMD ROCm 6.0

Intel OneAPI 2024.0.0

NVIDIA HPC SDK 23.11

CUDA 12.3

GCC 12 & 13

MOFED 5.1

Documentation

(NOTE: Documentation**, including revision documentation,** may be found on support.hpe.com; plus many CPE docs are also released in a tar file as part of the official release package as well. To find a specific document on support.hpe.com, simply do a search on “pubno 24.03” without the quotation marks. For example, search on S-8003 24.03 to find the CPE 24.03 version of the S-8003 document.)

Installation Guide

HPE Cray Programming Environment Installation Guide: CSM on HPE Cray EX Systems (24.03) S-8003

HPE_CPE_Installation_Guide_CSM_on_HPE_Cray_EX_Systems_24.03_S-8003.pdf

User Guide

HPE Cray Programming Environment User Guide: CSM on HPE Cray EX Systems (24.03) S-8005

HPE_CPE_User_Guide_CSM_on_HPE_Cray_EX_Systems_24.03_S-8005.pdf

Software Supported

The following software is supported with this release:

Cluster System Manager (CSM) 1.5 - managed systems running SLES15 SP5 on the compute/login nodes.
COS 23.11 components comprises:
- COS Base
- HPE Cray Supercomputing User Services Software (USS)
- HPE SUSE Linux Enterprise Server

IMPORTANT NOTES

NOTE: Hidden symbol errors when linking Fortran with CCE 17.0.1 Linking Fortran applications may fail with an error message of the form “hidden symbol `<SYMBOL>’ in <LIB> is referenced by DSO”. If this error message is seen, it can usually be worked around by adding ‘-lgcc_s’ to your link line. This is known to affect use of craypat and cray-parallel-netcdf but may be seen without them as well.

NOTE: NVIDIA HPC SDK releases include three versions of CUDA. For each new SDK version supported by HPE, Cray PE supports only the latest CUDA version included in the respective NVIDIA HPC SDK release (i.e., the older two CUDA versions are not supported by CPE when a new SDK is supported).

NOTE: CUDA 12.0 is compatible with GCC 11 (and below).

NOTE: Starting with CPE 23.12, cray-gcc packages will no longer be provided for SLES and COS based systems. Instead, SLES and COS based systems must use SLES-provided gcc*, gcc*-c++, and gcc*-fortran packages (available from the SLES Development Tools Module). Customers who wish to continue to use cray-gcc should continue to use this release (CPE 23.09) or earlier CPE releases.

NOTE: Starting with CPE 23.12, the PrgEnv-nvhpc and nvhpc module files were deprecated in favor of PrgEnv-nvidia and nvidia, respectively, and may be removed in a later release.

NOTE: Starting with CPE 23.12, the ROCm modulefile can now be created with craypkg-gen. The generated implementation of the Lmod ROCm modulefile works with the CPE amd modulefile.

NOTE: Perftools 24.03.0 and PAPI 7.1.0.1 included with CPE 24.03 are not compatible with ROCm <6.0.0; they require the use of ROCm 6.0.0 or later.

NOTE: Starting with CPE 24.03, On systems with aarch64 CPUs and NVIDIA GPUs, if
MPICH_GPU_SUPPORT_ENABLED=1 is set, HPE Cray MPI automatically chooses to disable the use of XPMEM for intra-node, inter-process MPI data movement operations that involve memory regions that are managed by system allocators (e.g., mmap, malloc, and new). This is done to guard against potential interactions between XPMEM and the GPU runtime layer that may cause node failures to occur. Instead of using XPMEM, HPE Cray MPI uses Linux Cross Memory Attach (CMA) to optimize these MPI operations. It is important to note that GPU Peer2Peer IPC will continue to be used for intra-node, inter-process data movement operations involving memory regions managed via GPU memory allocators (e.g., cudaMalloc, cudaFree).

NOTE: CPE 24.03 is not supported on XD670.

NOTE: CPE 24.03 is compatible with AMD ROCm 6.0 and not backwards compatible with previous versions of ROCm. Use of AMD ROCm 5.7 would require CPE 23.12.

NOTE: For machines with CSM & Blanca Peak nodes and CPE 24.03. The following workaround is needed to successfully deploy CPE 24.03 on the Blanca Peak nodes.

In the local VCS cpe-config-management repo, update ‘roles/cray.pe_deploy/files/squashfsmount.sh’ file, making the following change:

          -numa_nodes=$(numactl -H | grep available: | awk '{print $2}')
          -numa_interleave="0-$(( numa_nodes - 1 ))"
          +numa_cpu_nodes=$(grep -l . /sys/devices/system/node/node*/cpulist | 
                                                               sed -E -e 's/^.*([0-9]+).*$/\1/')
          +numa_interleave=$(echo $numa_cpu_nodes | sed -e 's/\s/,/g')

NOTE: The Workload Manager (WLM) was removed with the CPE 23.09 release but is now included with COS 23.11.

As applicable, use one of three options to install PBS Professional (PBS Pro) and Slurm on the system:

Install it from COS 23.11.

Install WLM RPMs from an earlier CPE release, such as CPE 23.05. WLM packages are included in the main CPE tarball. Contact the HPE Support Center to open an HPE support ticket for updated RPMs or additional assistance.

Obtain a newer WLM version directly from a third-party vendor. For example, you can install PBS Pro from Altair and Slurm from SchedMD

CPE 24.03 Product Versions:

NOTE: Use of **** indicates a new or updated component version compared to CPE 23.12.

HPE Cray Programming Environment for Shasta with CSM


Cray Compiling Environment – CCE
cce 17.0.1 ****

Cray Message Passing Toolkit - CMPT
cray-mpich 8.1.29 ****
cray-mpixlate 1.0.4 ****
cray-dsmml 0.3.0 ****
cray-pmi 6.1.14 ****
cray-openshmemx 11.7.1 ****

Application Launch Tools - ALT
cray-pals 1.3.2 ****

Cray Debugging Support Tools – CDST
cray-cti 2.18.3 ****
gdb4hpc 4.16.1 ****
cray-ccdb 5.0.3 ****
cray-stat 4.12.2
atp 3.15.3 ****
valgrind4hpc 2.13.2
sanitizers4hpc 1.1.2
cray-dyninst 12.3.1
cray-mrnet 5.1.2

Cray Performance Measurement & Analysis Tools – CPMAT
perftools 24.03.0 ****
cray-papi 7.1.0.1 ****

Cray Scientific and Math Libraries - CSML
cray-libsci 24.03.0 ****
cray-libsci-acc 24.03.1 ****
cray-fftw 3.3.10.7 ****

Cray Deep Learning Tools
craype-dl-plugin-py3 24.03.1 ****
craype-dl-plugin-ftr 22.06.1.2

Cray Environment Setup and Compiling support – CENV
craypkg-gen 1.3.32 ****
craype 2.7.31 ****
cpe-prgenv 8.5.0
cray-lmod 8.7.32 ****
cray-modules 3.2.11.7

Third party products
cray-hdf5 1.12.2.11 ****
cray-netcdf 4.9.0.11 ****
cray-parallel-netcdf 1.12.3.11 ****
cray-python 3.11.7 ****
cray-R 4.3.2 ****

Third-party products supported
Totalview 2023.4.16
Forge 23.1

*******************************************************************************************

Certain components, files or programs contained within this package or product are Copyright - 2024 Hewlett Packard Enterprise Development LP. All trademarks used in this document are the property of their respective owners.