Warning, /AID2E-framework/docs/user-guide/workflow-improvements.md is written in an unsupported language. File is not indexed.
0001 # Workflow and Objective Improvements Guide
0002
0003 This guide documents the improvements made to AID2E's workflow and objective specification system.
0004
0005 ## Table of Contents
0006
0007 1. [Objective Plan Terminology](#objective-plan-terminology)
0008 2. [Scheduler Cascade](#scheduler-cascade)
0009 3. [Combined Objectives](#combined-objectives)
0010 4. [Multi-Step Plans](#multi-step-plans)
0011 5. [Migration Guide](#migration-guide)
0012 6. [Examples](#examples)
0013
0014 ---
0015
0016 ## Objective Plan Terminology
0017
0018 ### What Changed?
0019
0020 The term **"computation"** has been renamed to **"objective_plan"** throughout the codebase for clarity. An objective plan represents the executable specification of how to compute an objective value.
0021
0022 ### Key Classes
0023
0024 - **`ObjectiveDefinition`**: Top-level objective specification
0025 - `name` (str): Objective name (e.g., "f1", "f2")
0026 - `direction` (ObjectiveDirection): MINIMIZE or MAXIMIZE
0027 - `objective_plan` (ObjectivePlanSpec): How to compute this objective
0028 - `scheduler` (Optional[SchedulerConfiguration]): Objective-level scheduler default
0029 - `metrics_keys` (List[str]): Keys to extract from objective output
0030
0031 - **`ObjectivePlanSpec`**: Canonical specification for computing an objective
0032 - Uses `steps`: `StepPlanSpec` with one or more stages
0033 - Each stage defines either:
0034 - `script`: Path to executable script
0035 - `inline`: Entrypoint to Python function
0036
0037 ### Single-Step and Multi-Step Plans
0038
0039 By design, all objective plans use the same `steps` structure. A simple
0040 objective can define one stage, while more complex objectives can define
0041 multiple dependent stages. This provides:
0042 - Unified handling of simple and complex computations
0043 - Support for preprocessing, evaluation, and postprocessing stages
0044 - Clear separation of concerns
0045
0046 Example (single script stage):
0047
0048 ```python
0049 ObjectiveDefinition(
0050 name="f1",
0051 direction=ObjectiveDirection.MINIMIZE,
0052 objective_plan=ObjectivePlanSpec(
0053 steps=StepPlanSpec(
0054 stages=[
0055 StepStage(
0056 name="f1_stage_0",
0057 script=ScriptObjective(
0058 path="scripts/dtlz2.py",
0059 output_file="f1.json",
0060 ),
0061 produces_objective=True,
0062 )
0063 ]
0064 )
0065 )
0066 )
0067 ```
0068
0069 ---
0070
0071 ## Scheduler Cascade
0072
0073 ### Overview
0074
0075 Instead of a single global scheduler, AID2E now supports **scheduler cascading** at multiple levels:
0076
0077 ```
0078 Cascade Precedence (highest to lowest):
0079 1. Stage-level scheduler (override)
0080 2. Branch-level scheduler (branch default)
0081 3. Workflow-level scheduler (workflow default)
0082 4. Objective-level scheduler (objective default)
0083 5. Global scheduler (global default)
0084 ```
0085
0086 This allows fine-grained control while maintaining sensible defaults.
0087
0088 ### Cascade Resolution
0089
0090 Use the `resolve_scheduler_cascade()` utility to determine the effective scheduler:
0091
0092 ```python
0093 from aid2e.utilities.configurations import resolve_scheduler_cascade
0094
0095 effective_scheduler = resolve_scheduler_cascade(
0096 stage_scheduler=stage_config.scheduler,
0097 branch_scheduler=branch_config.scheduler,
0098 workflow_scheduler=workflow_config.scheduler,
0099 objective_scheduler=objective_config.scheduler,
0100 global_scheduler=global_config.scheduler,
0101 )
0102 ```
0103
0104 ### Example
0105
0106 YAML configuration with scheduler cascade:
0107
0108 ```yaml
0109 problem:
0110 name: "DTLZ2"
0111 type: "toy"
0112 design_space:
0113 path: "design.params"
0114
0115 # Global/workflow-level scheduler
0116 scheduler:
0117 runner_type: "JobLibRunner"
0118 parameters:
0119 n_jobs: 8
0120 backend: "loky"
0121
0122 workflows:
0123 - name: "dtlz2_eval"
0124
0125 # Workflow-level scheduler (overrides global)
0126 scheduler:
0127 runner_type: "JobLibRunner"
0128 parameters:
0129 n_jobs: 4
0130 backend: "threading"
0131
0132 branches:
0133 - name: "main"
0134
0135 # Branch-level scheduler (overrides workflow)
0136 scheduler:
0137 runner_type: "JobLibRunner"
0138 parameters:
0139 n_jobs: 2
0140 backend: "loky"
0141
0142 stages:
0143 - name: "evaluate"
0144
0145 # Stage-level scheduler (overrides branch)
0146 scheduler:
0147 runner_type: "SlurmRunner"
0148 parameters:
0149 partition: "gpu"
0150 nodes: 1
0151
0152 jobs:
0153 - name: "compute_objective"
0154 command: "python eval.py"
0155
0156 objectives:
0157 - name: "f1"
0158 direction: "minimize"
0159
0160 # Objective-level scheduler (applies to this objective)
0161 scheduler:
0162 runner_type: "JobLibRunner"
0163 parameters:
0164 n_jobs: 1
0165
0166 objective_plan:
0167 steps:
0168 stages:
0169 - name: "evaluate_f1"
0170 script:
0171 path: "scripts/dtlz2.py"
0172 output_file: "f1.json"
0173 produces_objective: true
0174 ```
0175
0176 ### Precedence Resolution
0177
0178 For the objective "f1" in the above example:
0179 1. Check objective-level scheduler → Found: `JobLibRunner` with `n_jobs=1`
0180 2. This is the effective scheduler for objective execution
0181
0182 For a stage without explicit scheduler:
0183 1. Check stage-level scheduler → Not found
0184 2. Check branch-level scheduler → Found: `JobLibRunner` with `n_jobs=2`
0185 3. This is the effective scheduler for the stage
0186
0187 ---
0188
0189 ## Combined Objectives
0190
0191 ### Motivation
0192
0193 Sometimes a single computation produces **multiple objective metrics**. For example:
0194 - A DTLZ2 evaluation script outputs both `f1` and `f2`
0195 - A surrogate model prediction outputs multiple target values
0196 - A simulation produces both efficiency and quality scores
0197
0198 Instead of running the same plan twice, **combined objectives** allow one execution to produce multiple metrics.
0199
0200 ### Key Classes
0201
0202 - **`CombinedObjectivePlan`**: Bundle of a plan with multiple metric definitions
0203 - `name` (str): Combined objective name
0204 - `objective_plan` (ObjectivePlanSpec): The plan to execute
0205 - `metrics` (List[CombinedObjectiveMetric]): Metrics extracted from the output
0206 - `scheduler` (Optional[SchedulerConfiguration]): Scheduler for this plan
0207
0208 - **`CombinedObjectiveMetric`**: A single metric from a combined plan
0209 - `name` (str): Metric name (e.g., "f1", "f2")
0210 - `direction` (ObjectiveDirection): MINIMIZE or MAXIMIZE
0211 - `metric_key` (str): Key to extract from plan output (e.g., "f1" from `{"f1": 0.5, "f2": 0.3}`)
0212
0213 ### Usage in Workflows
0214
0215 Add `combined_objectives` to a `WorkflowDefinition`:
0216
0217 ```yaml
0218 workflows:
0219 - name: "dtlz2_multi"
0220
0221 branches:
0222 - name: "main"
0223 stages:
0224 - name: "evaluate"
0225 jobs:
0226 - name: "dtlz2"
0227 command: "python scripts/dtlz2.py"
0228 outputs:
0229 - path: "objectives.json"
0230 format: "json"
0231
0232 # Combined objectives: one plan produces multiple metrics
0233 combined_objectives:
0234 - name: "dtlz2_pareto"
0235
0236 objective_plan:
0237 steps:
0238 stages:
0239 - name: "evaluate_objectives"
0240 script:
0241 path: "scripts/dtlz2.py"
0242 output_file: "objectives.json"
0243 produces_objective: true
0244
0245 metrics:
0246 - name: "f1"
0247 direction: "minimize"
0248 metric_key: "f1" # Extract {"f1": ...} from output
0249
0250 - name: "f2"
0251 direction: "minimize"
0252 metric_key: "f2" # Extract {"f2": ...} from output
0253
0254 - name: "efficiency"
0255 direction: "maximize"
0256 metric_key: "efficiency" # Extract {"efficiency": ...}
0257 ```
0258
0259 ### Output Format
0260
0261 The objective plan script should output a JSON/YAML file with the metric values:
0262
0263 ```json
0264 {
0265 "f1": 0.45,
0266 "f2": 0.67,
0267 "efficiency": 0.92
0268 }
0269 ```
0270
0271 Each key in this object becomes extractable via `metric_key` in `CombinedObjectiveMetric`.
0272
0273 ---
0274
0275 ## Step Plans
0276
0277 ### Structure
0278
0279 A step plan contains one or more stages, each with a distinct computation:
0280
0281 ```python
0282 ObjectiveDefinition(
0283 name="complex_eval",
0284 direction=ObjectiveDirection.MINIMIZE,
0285 objective_plan=ObjectivePlanSpec(
0286 steps=StepPlanSpec(
0287 stages=[
0288 StepStage(
0289 name="preprocess",
0290 script=ScriptObjective(path="preprocess.py", output_file="prep.json"),
0291 inputs=["design_params.json"],
0292 outputs=["preprocessed.json"],
0293 produces_objective=False,
0294 ),
0295 StepStage(
0296 name="evaluate",
0297 script=ScriptObjective(path="evaluate.py", output_file="eval.json"),
0298 inputs=["preprocessed.json"],
0299 outputs=["objectives.json"],
0300 produces_objective=True, # This stage produces the final objective
0301 ),
0302 ],
0303 produces_from_stage="evaluate", # Which stage's output is the objective value
0304 )
0305 )
0306 )
0307 ```
0308
0309 ### StepStage Fields
0310
0311 - `name` (str): Stage name (must be unique within a plan)
0312 - `script` or `inline` (exactly one): The execution mode
0313 - `script`: Path to executable script
0314 - `inline`: Python function entrypoint
0315 - `inputs` (List[str]): Input artifacts required
0316 - `outputs` (List[str]): Output artifacts produced
0317 - `extra_args` (Dict[str, Any]): Additional arguments to pass to the stage
0318 - `produces_objective` (bool): Whether this stage produces the objective value
0319 - `depends_on` (List[str]): Names of preceding stages this depends on
0320
0321 ### Validation
0322
0323 The model automatically validates:
0324 - **Mutual exclusivity**: Each stage has exactly one of `script` or `inline`
0325 - **Unique names**: All stage names are unique within a plan
0326 - **DAG integrity**: Dependencies form a valid DAG (no cycles)
0327 - **Single producer**: Exactly one stage marks `produces_objective=True`
0328 - **Dependency satisfaction**: All dependencies reference existing stages
0329
0330 ---
0331
0332 ## Migration Guide
0333
0334 ### Old Terminology → New Terminology
0335
0336 | Old Term | New Term | Notes |
0337 |----------|----------|-------|
0338 | "Computation" | "Objective Plan" | More descriptive; plan indicates it's an executable specification |
0339 | `multi_steps` | `steps` | Objective plans now use a neutral name for one or more stages |
0340
0341 ### Updating Your Configurations
0342
0343 #### Before (old terminology):
0344
0345 ```yaml
0346 objectives:
0347 - name: "f1"
0348 direction: "minimize"
0349 computation: # Old field name
0350 script:
0351 path: "scripts/dtlz2.py"
0352 output_file: "f1.json"
0353 ```
0354
0355 #### After (new terminology):
0356
0357 ```yaml
0358 objectives:
0359 - name: "f1"
0360 direction: "minimize"
0361 objective_plan: # New field name
0362 steps:
0363 stages:
0364 - name: "evaluate_f1"
0365 script:
0366 path: "scripts/dtlz2.py"
0367 output_file: "f1.json"
0368 produces_objective: true
0369 ```
0370
0371 ### Python Code Updates
0372
0373 ```python
0374 from aid2e.utilities.configurations import ObjectivePlanSpec, StepPlanSpec, StepStage
0375
0376 spec = ObjectivePlanSpec(
0377 steps=StepPlanSpec(
0378 stages=[
0379 StepStage(...),
0380 ],
0381 ),
0382 )
0383 ```
0384
0385 ---
0386
0387 ## Examples
0388
0389 ### Example 1: Single Objective with Scheduler Cascade
0390
0391 File: [`examples/complete/workflow_example_single_objective.yml`](../examples/complete/workflow_example_single_objective.yml)
0392
0393 Shows:
0394 - Single objective with `objective_plan`
0395 - Scheduler cascade from global → workflow → branch → stage
0396 - Job factory for parameter sweeps
0397 - Clear comments on precedence resolution
0398
0399 ### Example 2: Combined Objectives
0400
0401 File: [`examples/complete/workflow_example_combined_objectives.yml`](../examples/complete/workflow_example_combined_objectives.yml)
0402
0403 Shows:
0404 - One plan producing multiple metrics (f1, f2)
0405 - Metric extraction via `metric_key`
0406 - Combined objective in workflow
0407 - Optional scheduler override at combined objective level
0408
0409 ### Example 3: Multi-Step Objective Plan
0410
0411 ```python
0412 # Three-stage pipeline: preprocess → evaluate → aggregate
0413 ObjectivePlanSpec(
0414 steps=StepPlanSpec(
0415 stages=[
0416 StepStage(
0417 name="preprocess",
0418 script=ScriptObjective(path="preprocess.py"),
0419 inputs=["raw_design.json"],
0420 outputs=["design_preprocessed.json"],
0421 produces_objective=False,
0422 ),
0423 StepStage(
0424 name="evaluate",
0425 script=ScriptObjective(path="evaluate.py"),
0426 inputs=["design_preprocessed.json"],
0427 outputs=["raw_objectives.json"],
0428 produces_objective=False,
0429 ),
0430 StepStage(
0431 name="aggregate",
0432 inline=InlineObjective(entrypoint="my_module:aggregate_objectives"),
0433 inputs=["raw_objectives.json"],
0434 outputs=["final_objectives.json"],
0435 produces_objective=True,
0436 ),
0437 ],
0438 produces_from_stage="aggregate",
0439 )
0440 )
0441 ```
0442
0443 ---
0444
0445 ## Summary of Benefits
0446
0447 1. **Clarity**: "Objective plan" is more intuitive than "computation"
0448 2. **Flexibility**: Scheduler cascade allows both global consistency and local overrides
0449 3. **Efficiency**: Combined objectives avoid redundant computations
0450 4. **Composability**: Multi-step plans support complex workflows within a single objective
0451 5. **Backward Compatibility**: Old code continues to work with alias imports
0452
0453 ---
0454
0455 ## Questions & Support
0456
0457 For more information:
0458 - See [`docs/api-reference/`](../docs/api-reference/) for detailed API docs
0459 - Check [`tests/test_utilities/test_workflows/`](../tests/test_utilities/test_workflows/) for integration tests
0460 - Review [YAML examples](../examples/complete/) for real-world use cases
