CodeEntropy.core.dask_clusters module

Helpers for setting up Dask clusters on HPC using SLURM.

class CodeEntropy.core.dask_clusters.HPCDaskManager(args)

Bases: object

Manage SLURM-backed Dask clusters and submission utilities for HPC environments.

check_slurm_env() → None

Remove inherited SLURM environment variables that can break nested srun calls.

This is important when the master CodeEntropy process itself is already running inside a SLURM allocation and then launches Dask worker jobs.

configure_cluster() → Client

Configure a SLURM-backed Dask cluster.

Returns:

Dask distributed client connected to the SLURMCluster.

resolve_conda_settings() → None

Fill missing conda/mamba settings from the active environment.

Explicit user-provided values are preserved. Auto-detection is only used when values are missing.

slurm_directives() → tuple[list[str], list[str]]

Process additional SLURM directives and directives to skip.

Returns:

Tuple containing extra directives and skipped directives.

slurm_prologues() → list[str]

Build environment setup commands for the SLURM worker job script.

Returns:

List of shell commands executed before the Dask worker starts.

submit_master() → None

Submit a SLURM job that runs the master CodeEntropy process.

This generates a temporary SLURM script and submits it via sbatch.

system_network_interface() → str

Get the best candidate for the HPC network interface.

This deliberately follows the WaterEntropy-style behaviour and only selects from known HPC-safe interfaces. It avoids selecting arbitrary interfaces such as eno1, which may exist on the master node but not on worker nodes.