Warning, /iDDS/PYPROJECT_MIGRATION.md is written in an unsupported language. File is not indexed.
0001 # iDDS Build System Migration to pyproject.toml
0002
0003 This document describes the migration from `setup.py` to modern `pyproject.toml` configuration.
0004
0005 ## Overview
0006
0007 The iDDS project has been migrated from the legacy `setup.py` approach to modern `pyproject.toml` configuration (PEP 517/518/621). This provides:
0008
0009 - **Standard configuration**: All metadata in declarative `pyproject.toml` files
0010 - **Better tooling**: Native support in pip, build, and modern Python tools
0011 - **Reproducible builds**: Isolated build environments
0012 - **Development workflow**: Simplified installation and dependency management
0013
0014 ## Structure
0015
0016 The repository contains multiple packages:
0017
0018 - `common/` - Core utilities and common code
0019 - `main/` - Server components
0020 - `client/` - Client library
0021 - `atlas/` - ATLAS-specific extensions
0022 - `doma/` - DOMA extensions
0023 - `workflow/` - Workflow management
0024 - `prompt/` - Prompt agent
0025 - `website/` - Web interface
0026 - `monitor/` - Monitoring components
0027
0028 Each package now has its own `pyproject.toml` file.
0029
0030 ## Installation
0031
0032 ### Option 1: Using the build script (recommended)
0033
0034 ```bash
0035 # Install all packages in development mode
0036 python build_all.py develop
0037
0038 # Or install normally
0039 python build_all.py install
0040
0041 # Build wheel distributions
0042 python build_all.py build
0043 ```
0044
0045 ### Option 2: Using make
0046
0047 ```bash
0048 # Install in development mode
0049 make develop
0050
0051 # Or install normally
0052 make install
0053
0054 # Build wheels
0055 make build
0056
0057 # Clean build artifacts
0058 make clean
0059 ```
0060
0061 ### Option 3: Manual installation
0062
0063 Install packages individually in dependency order:
0064
0065 ```bash
0066 # Install common package first (required by others)
0067 cd common && pip install -e . && cd ..
0068
0069 # Then install other packages
0070 cd main && pip install -e . && cd ..
0071 cd client && pip install -e . && cd ..
0072 # ... etc
0073 ```
0074
0075 ## Development Workflow
0076
0077 ### Setting up for development
0078
0079 ```bash
0080 # Clone the repository
0081 git clone https://github.com/HSF/iDDS.git
0082 cd iDDS
0083
0084 # Install in development/editable mode
0085 python build_all.py develop
0086
0087 # Or use make
0088 make develop
0089 ```
0090
0091 ### Running tests
0092
0093 ```bash
0094 # Run all tests
0095 pytest
0096
0097 # Or use make
0098 make test
0099
0100 # Run tests for a specific package
0101 cd common && pytest
0102 ```
0103
0104 ### Code quality
0105
0106 ```bash
0107 # Run linting
0108 make lint
0109
0110 # Format code with black
0111 make format
0112 ```
0113
0114 ## Building Distributions
0115
0116 ### Build wheels for all packages
0117
0118 ```bash
0119 python build_all.py wheel
0120 # Wheels will be in each package's dist/ directory
0121 ```
0122
0123 ### Build a specific package
0124
0125 ```bash
0126 cd common
0127 python -m build
0128 # or
0129 pip wheel .
0130 ```
0131
0132 ## Migration Notes
0133
0134 ### What Changed
0135
0136 1. **Configuration files**: Each package now has `pyproject.toml` instead of `setup.py`
0137 2. **Build command**: Use `python build_all.py <command>` instead of `python setup.py <command>`
0138 3. **Dependencies**: Declared in `dependencies` array in `pyproject.toml`
0139 4. **Custom commands**: Removed custom install/wheel commands (no longer needed)
0140
0141 ### Backwards Compatibility
0142
0143 The old `setup.py` files are kept for reference but are deprecated. Use the new `pyproject.toml` approach.
0144
0145 ### Key Differences
0146
0147 | Old (setup.py) | New (pyproject.toml) |
0148 |---------------|---------------------|
0149 | `python setup.py install` | `pip install .` or `python build_all.py install` |
0150 | `python setup.py develop` | `pip install -e .` or `python build_all.py develop` |
0151 | `python setup.py bdist_wheel` | `python -m build --wheel` or `python build_all.py wheel` |
0152 | `python setup.py clean` | `python build_all.py clean` |
0153
0154 ## Package Dependencies
0155
0156 ```
0157 idds-common (base)
0158 ├── idds-server (main)
0159 ├── idds-client
0160 ├── idds-atlas
0161 ├── idds-doma
0162 ├── idds-workflow
0163 ├── idds-prompt
0164 ├── idds-website
0165 └── idds-monitor
0166 ```
0167
0168 All packages depend on `idds-common`. Install it first if installing manually.
0169
0170 ## Troubleshooting
0171
0172 ### Build package not found
0173
0174 ```bash
0175 pip install build
0176 ```
0177
0178 ### Import errors after migration
0179
0180 Reinstall in development mode:
0181
0182 ```bash
0183 python build_all.py clean
0184 python build_all.py develop
0185 ```
0186
0187 ### Namespace package conflicts
0188
0189 The migration handles namespace packages correctly. Each subpackage (e.g., `idds.common`, `idds.core`) is properly configured to work together.
0190
0191 ## Tools Configuration
0192
0193 The root `pyproject.toml` contains shared tool configurations:
0194
0195 - **Black**: Code formatter (line-length: 100)
0196 - **pytest**: Test runner with coverage
0197 - **mypy**: Type checking
0198 - **flake8**: Linting
0199
0200 These apply to all packages in the monorepo.
0201
0202 ## CI/CD Integration
0203
0204 Update your CI/CD pipelines to use the new build system:
0205
0206 ```yaml
0207 # Example GitHub Actions
0208 - name: Install dependencies
0209 run: |
0210 python -m pip install --upgrade pip build
0211
0212 - name: Install iDDS packages
0213 run: python build_all.py develop
0214
0215 - name: Build distributions
0216 run: python build_all.py wheel
0217 ```
0218
0219 ## References
0220
0221 - [PEP 517](https://peps.python.org/pep-0517/) - Build system interface
0222 - [PEP 518](https://peps.python.org/pep-0518/) - pyproject.toml
0223 - [PEP 621](https://peps.python.org/pep-0621/) - Project metadata
0224 - [Setuptools Pyproject Guide](https://setuptools.pypa.io/en/latest/userguide/pyproject_config.html)
