Warning, /AID2E-framework/docs/scheduler-configuration.md is written in an unsupported language. File is not indexed.
0001 # Scheduler/Runner Configuration Guide
0002
0003 ## Overview
0004
0005 The scheduler/runner configuration defines how AID2E executes optimization jobs. The framework supports three execution backends:
0006
0007 1. **JobLibRunner** - Local parallel execution using Python's joblib library
0008 2. **SlurmRunner** - HPC cluster execution via SLURM workload manager
0009 3. **PanDAiDDSRunner** - Distributed execution across multiple sites via PanDA iDDS
0010
0011 ## Configuration Structure
0012
0013 The `scheduler` section in your configuration file defines runner settings:
0014
0015 ```yaml
0016 scheduler:
0017 runner_type: "JobLibRunner" # or "SlurmRunner" or "PanDAiDDSRunner"
0018 parameters:
0019 # Runner-specific settings
0020 # Common settings
0021 max_retries: 3
0022 output_location: "./scheduler_output"
0023 monitor_interval: 30
0024 ```
0025
0026 ## JobLibRunner Configuration
0027
0028 **Best for:** Single-machine parallel execution with multiple CPU cores
0029
0030 ### Parameters
0031
0032 | Parameter | Type | Default | Description |
0033 |-----------|------|---------|-------------|
0034 | `n_jobs` | int | -1 | Number of jobs to run in parallel. -1 = use all available processors |
0035 | `backend` | str | "loky" | Execution backend: "loky", "threading", "processes" |
0036 | `timeout` | int/None | None | Timeout in seconds for each job. None = no timeout |
0037 | `verbose` | int | 0 | Verbosity level (0-11) for joblib logging |
0038
0039 ### Example
0040
0041 ```yaml
0042 scheduler:
0043 runner_type: "JobLibRunner"
0044 parameters:
0045 n_jobs: 8 # Use 8 CPU cores
0046 backend: "loky" # Robust multiprocessing backend
0047 timeout: 3600 # 1 hour timeout per job
0048 verbose: 1 # Some logging output
0049 max_retries: 3
0050 output_location: "./joblib_output"
0051 monitor_interval: 30
0052 ```
0053
0054 ### When to Use
0055
0056 - Development and testing on local machines
0057 - Single-node multi-core systems
0058 - Quick prototyping with manageable problem sizes
0059 - Integration testing in CI/CD pipelines
0060
0061 ## SlurmRunner Configuration
0062
0063 **Best for:** HPC cluster execution with job queuing and resource allocation
0064
0065 ### Parameters
0066
0067 | Parameter | Type | Default | Description |
0068 |-----------|------|---------|-------------|
0069 | `partition` | str/None | None | SLURM partition/queue name |
0070 | `ntasks` | int | 1 | Number of tasks to run |
0071 | `cpus_per_task` | int/None | None | CPU cores per task |
0072 | `mem` | str/None | None | Memory request (e.g., "4GB", "8000MB") |
0073 | `time` | str/None | None | Wall clock time limit (HH:MM:SS) |
0074 | `gres` | str/None | None | Generic resource (e.g., "gpu:1" for one GPU) |
0075 | `job_name_prefix` | str | "aid2e" | Prefix for generated SLURM job names for tracking |
0076
0077 ### Example
0078
0079 ```yaml
0080 scheduler:
0081 runner_type: "SlurmRunner"
0082 parameters:
0083 partition: "gpu"
0084 ntasks: 4
0085 cpus_per_task: 8
0086 mem: "32GB"
0087 time: "12:00:00"
0088 gres: "gpu:1"
0089 max_retries: 3
0090 output_location: "./slurm_output"
0091 monitor_interval: 60
0092 ```
0093
0094 ### When to Use
0095
0096 - Production runs on HPC clusters (NERSC, XSEDE, etc.)
0097 - Large-scale optimization problems
0098 - GPU-accelerated evaluations
0099 - Jobs requiring specific resource constraints
0100 - When you need job queuing and fair scheduling
0101
0102 ### SLURM Tips
0103
0104 - Check available partitions: `sinfo`
0105 - Monitor submitted jobs: `squeue`
0106 - View available GPUs: `sinfo --gres`
0107 - Check resource limits: `slurm_limits`
0108
0109 ## PanDAiDDSRunner Configuration
0110
0111 **Best for:** Distributed execution across multiple computing sites with automated load balancing
0112
0113 ### Parameters
0114
0115 | Parameter | Type | Default | Description |
0116 |-----------|------|---------|-------------|
0117 | `name` | str/None | auto-generated | PanDA job name for tracking, must start with `user.` |
0118 | `job_name_prefix` | str | "aid2e_job" | Prefix used when auto-generating the PanDA job name |
0119 | `task_type` | str/None | "AID2E" | Type of processing (PanDA classification) |
0120 | `cloud` | str/None | None | Target cloud/region |
0121 | `max_walltime` | int/None | None | Maximum walltime in seconds |
0122
0123 ### Example
0124
0125 ```yaml
0126 scheduler:
0127 runner_type: "PanDAiDDSRunner"
0128 parameters:
0129 job_name_prefix: "aid2e_dtlz2"
0130 cloud: null # Auto-select best site
0131 task_type: "optimization"
0132 max_walltime: 7200 # 2 hours
0133 max_retries: 5
0134 output_location: "./panda_output"
0135 monitor_interval: 120
0136 ```
0137
0138 ### When to Use
0139
0140 - Large-scale distributed optimization across multiple institutions
0141 - Federated computing environments (e.g., ATLAS collaboration)
0142 - Load-balanced execution across geographically distributed sites
0143 - When individual sites may have intermittent availability
0144 - Projects requiring centralized job tracking and monitoring
0145
0146 ### PanDA iDDS Tips
0147
0148 - Monitor jobs via PanDA dashboard: `https://panda.cern.ch/`
0149 - Check campaign status: `idds show --id <campaign_name>`
0150 - View worker logs: `idds logs --id <task_id>`
0151 - Troubleshoot: `idds status --id <campaign_name>`
0152
0153 ## Common Parameters
0154
0155 These apply regardless of runner type:
0156
0157 | Parameter | Type | Default | Description |
0158 |-----------|------|---------|-------------|
0159 | `max_retries` | int | 3 | Global maximum retries for failed jobs |
0160 | `output_location` | str | "./scheduler_output" | Base directory for scheduler output files |
0161 | `monitor_interval` | int | 30 | Interval (seconds) for job status checks |
0162
0163 ## Configuration Examples
0164
0165 ### Minimal JobLib Configuration
0166 ```yaml
0167 scheduler:
0168 runner_type: "JobLibRunner"
0169 parameters:
0170 n_jobs: -1
0171 ```
0172
0173 ### Production SLURM Setup
0174 ```yaml
0175 scheduler:
0176 runner_type: "SlurmRunner"
0177 parameters:
0178 partition: "gpu"
0179 ntasks: 8
0180 cpus_per_task: 16
0181 mem: "64GB"
0182 time: "24:00:00"
0183 gres: "gpu:4"
0184 job_name_prefix: "aid2e_large_scale"
0185 max_retries: 5
0186 output_location: "./results/slurm"
0187 monitor_interval: 300
0188 ```
0189
0190 ### Distributed PanDA Setup
0191 ```yaml
0192 scheduler:
0193 runner_type: "PanDAiDDSRunner"
0194 parameters:
0195 job_name_prefix: "aid2e_large_scale"
0196 cloud: null # Distributed across all sites
0197 max_walltime: 7200
0198 max_retries: 7
0199 output_location: "./results/panda"
0200 monitor_interval: 300
0201 ```
0202
0203 ## Python API Usage
0204
0205 ```python
0206 from aid2e.utilities.configurations import (
0207 load_config,
0208 SchedulerConfiguration,
0209 )
0210
0211 # Load complete config including scheduler
0212 config = load_config("examples/basic/full_example_slurm.yml")
0213
0214 # Access scheduler configuration
0215 scheduler_cfg = config.scheduler
0216 print(f"Runner type: {scheduler_cfg.runner_type}")
0217 print(f"Max retries: {scheduler_cfg.max_retries}")
0218
0219 # Get validated runner-specific config
0220 runner_cfg = scheduler_cfg.parse_runner_params()
0221 print(f"Runner parameters: {runner_cfg}")
0222
0223 # Register custom runner types if needed
0224 from aid2e.utilities.configurations import register
0225 from pydantic import BaseModel
0226
0227 class CustomRunnerConfig(BaseModel):
0228 # Your custom fields
0229 pass
0230
0231 register("CustomRunner", CustomRunnerConfig)
0232 ```
0233
0234 ## Migration from Legacy Scheduler Configuration
0235
0236 If upgrading from an older AID2E version without scheduler configuration:
0237
0238 1. **JobLib (local execution)** - New default behavior
0239 ```yaml
0240 scheduler:
0241 runner_type: "JobLibRunner"
0242 parameters:
0243 n_jobs: -1
0244 ```
0245
0246 2. **SLURM (HPC)** - Replace old SLURM template with:
0247 ```yaml
0248 scheduler:
0249 runner_type: "SlurmRunner"
0250 parameters:
0251 # ... parameters from old slurm.template file
0252 ```
0253
0254 3. **No scheduler specified** - Defaults to JobLibRunner with 1 job (sequential execution)
0255
0256 ## Troubleshooting
0257
0258 ### Issue: "runner_type must be one of JobLibRunner, SlurmRunner, PanDAiDDSRunner"
0259
0260 **Solution:** Check spelling and case sensitivity. Valid values are exactly:
0261 - `JobLibRunner`
0262 - `SlurmRunner`
0263 - `PanDAiDDSRunner`
0264
0265 ### Issue: JobLib jobs timeout
0266
0267 **Solution:** Increase the `timeout` parameter:
0268 ```yaml
0269 scheduler:
0270 runner_type: "JobLibRunner"
0271 parameters:
0272 timeout: 7200 # Increase to 2 hours
0273 ```
0274
0275 ### Issue: SLURM job not starting
0276
0277 **Solution:** Check partition availability and resource limits:
0278 ```bash
0279 sinfo # Check available partitions
0280 sinfo --gres # Check GPU availability
0281 ```
0282
0283 ### Issue: PanDA task failures
0284
0285 **Solution:** Increase scheduler retries and the runner walltime:
0286 ```yaml
0287 scheduler:
0288 runner_type: "PanDAiDDSRunner"
0289 parameters:
0290 max_walltime: 7200
0291 ```
0292
0293 ## See Also
0294
0295 - [Full Configuration Guide](../user-guide/overview.md)
0296 - [Optimization Configuration](optimization_config.md)
0297 - [Problem Configuration](problem_config.md)
0298 - [Example Configurations](../../examples/basic/)
