Warning, /containers/docs/build-pipeline.md is written in an unsupported language. File is not indexed.
0001 # Build Pipeline
0002
0003 The container build pipeline is implemented as a GitHub Actions workflow. This document describes the workflow structure and job dependencies.
0004
0005 ## Workflow Overview
0006
0007 ```mermaid
0008 flowchart TB
0009 subgraph "Triggers"
0010 T1[Schedule<br/>Every 6 hours]
0011 T2[Push to master]
0012 T3[Pull Request to master]
0013 T4[Manual Dispatch]
0014 end
0015
0016 T1 & T2 & T3 & T4 --> W[build-push workflow]
0017
0018 subgraph "Jobs"
0019 W --> B1[base amd64]
0020 W --> B2[base arm64]
0021 B1 & B2 --> BM[base-manifest]
0022 BM --> E1[eic_ci amd64]
0023 BM --> E2[eic_ci arm64]
0024 BM --> E3[eic_xl amd64]
0025 BM --> E4[eic_xl arm64]
0026 E1 & E2 --> EM1[eic-manifest ci]
0027 E3 & E4 --> EM2[eic-manifest xl]
0028 end
0029 ```
0030
0031 ## Job Details
0032
0033 ### Base Image Job
0034
0035 Builds the `debian_stable_base` image with Spack and compilers installed.
0036
0037 ```mermaid
0038 sequenceDiagram
0039 participant GH as GitHub Actions
0040 participant R as Registry (ghcr.io)
0041 participant C as Cache
0042
0043 GH->>GH: Checkout repository
0044 GH->>GH: Load spack versions<br/>from *.sh files
0045 GH->>GH: Setup Docker Buildx
0046 GH->>C: Restore build mount caches<br/>apt, spack source
0047 GH->>R: Login to registry
0048 GH->>GH: Build Dockerfile
0049 Note over GH: containers/debian/Dockerfile
0050 GH->>R: Push image by digest
0051 GH->>R: Push layer cache
0052 GH->>C: Save build mount caches
0053 GH->>GH: Upload digest artifact
0054 ```
0055
0056 **Key Build Arguments:**
0057 - `SPACK_ORGREPO`, `SPACK_VERSION`, `SPACK_SHA`, `SPACK_CHERRYPICKS`
0058 - `SPACKPACKAGES_ORGREPO`, `SPACKPACKAGES_VERSION`, `SPACKPACKAGES_SHA`, `SPACKPACKAGES_CHERRYPICKS`
0059 - `KEY4HEPSPACK_ORGREPO`, `KEY4HEPSPACK_VERSION`, `KEY4HEPSPACK_SHA`
0060 - `EICSPACK_ORGREPO`, `EICSPACK_VERSION`, `EICSPACK_SHA`
0061
0062 ### Base Manifest Job
0063
0064 Creates a multi-architecture manifest from the per-architecture digests.
0065
0066 ```mermaid
0067 sequenceDiagram
0068 participant GH as GitHub Actions
0069 participant R as Registry (ghcr.io)
0070
0071 GH->>GH: Download digest artifacts
0072 GH->>R: Login to registry
0073 GH->>GH: Compute metadata tags
0074 GH->>R: Create manifest list<br/>combining amd64 + arm64
0075 ```
0076
0077 **Output Tags:**
0078 - `pipeline-{run_id}` - Internal tag for CI chaining
0079 - `unstable-pr-{number}` - For pull requests
0080 - Version tags for releases
0081
0082 ### EIC Image Job
0083
0084 Builds the full EIC environment image on top of the base image.
0085
0086 ```mermaid
0087 sequenceDiagram
0088 participant GH as GitHub Actions
0089 participant R as Registry (ghcr.io)
0090 participant C as Cache
0091
0092 GH->>GH: Free disk space
0093 GH->>GH: Checkout repository
0094 GH->>GH: Resolve benchmark SHAs
0095 GH->>GH: Resolve campaign SHAs
0096 GH->>GH: Generate mirrors.yaml
0097 GH->>GH: Setup Docker Buildx
0098 GH->>C: Restore build mount caches<br/>ccache, spack source
0099 GH->>R: Login to registry
0100 GH->>GH: Build multi-stage Dockerfile
0101 Note over GH: containers/eic/Dockerfile
0102 GH->>R: Push image by digest
0103 GH->>R: Push layer cache
0104 GH->>C: Save build mount caches
0105 GH->>GH: Upload digest artifact
0106 ```
0107
0108 **Key Build Arguments:**
0109 - `ENV` - Environment type (`ci` or `xl`)
0110 - `INTERNAL_TAG` - Base image tag to build from
0111 - Benchmark SHAs for common_bench, detector_benchmarks, etc.
0112 - Campaign SHAs for simulation_campaign_hepmc3, job_submission_*, etc.
0113
0114 **Secret Mounts:**
0115 - `mirrors.yaml` - Spack buildcache configuration with credentials
0116
0117 ### EIC Manifest Job
0118
0119 Creates multi-architecture manifests for each environment variant.
0120
0121 ## Caching Strategy
0122
0123 ### Docker Layer Cache
0124
0125 Stored in the registry using the `cache-to` and `cache-from` build options:
0126
0127 ```yaml
0128 cache-from: |
0129 type=registry,ref=ghcr.io/eic/buildcache:{image}-{branch}-{arch}
0130 cache-to: type=registry,ref=ghcr.io/eic/buildcache:{image}-{branch}-{arch},mode=max
0131 ```
0132
0133 ### Build Mount Cache
0134
0135 Uses [buildkit-cache-dance](https://github.com/reproducible-containers/buildkit-cache-dance) to persist mount caches:
0136
0137 | Cache | Path | Contents |
0138 |-------|------|----------|
0139 | `var-cache-apt` | `/var/cache/apt` | APT package cache |
0140 | `var-lib-apt` | `/var/lib/apt` | APT lists cache |
0141 | `var-cache-spack` | `/var/cache/spack` | Spack source tarballs |
0142 | `ccache` | `/ccache` | Compiler cache |
0143
0144 ### Spack Buildcache
0145
0146 Pre-built binaries are stored in OCI registries:
0147
0148 - **Read-only**: `oci://ghcr.io/eic/spack-{version}` - Public buildcache
0149 - **Write**: Configured via secret `mirrors.yaml` mount during builds
0150
0151 ## Workflow Triggers
0152
0153 | Trigger | Behavior |
0154 |---------|----------|
0155 | Schedule (cron) | Every 6 hours - nightly builds |
0156 | Push to master | Build and push with `pipeline-*` tag |
0157 | Pull Request | Build with `unstable-pr-*` tag |
0158 | Manual Dispatch | Allows overriding EDM4EIC, EICRECON, JUGGLER versions |
0159
0160 ## Environment Matrix
0161
0162 The EIC job builds the following matrix:
0163
0164 | ENV | Architecture | Description |
0165 |-----|--------------|-------------|
0166 | ci | amd64 | CI environment for x86_64 |
0167 | ci | arm64 | CI environment for ARM64 |
0168 | xl | amd64 | Full environment for x86_64 |
0169 | xl | arm64 | Full environment for ARM64 |
0170
0171 ## Concurrency Control
0172
0173 ```yaml
0174 concurrency:
0175 group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
0176 cancel-in-progress: false
0177 ```
0178
0179 Workflows are grouped by PR number or branch, but **not cancelled** when new commits are pushed (builds are expensive and take hours).
