swf-testbed/docs/monitor.md

0001 # Monitor Integration
0002
0003 The SWF Monitor provides web interface and API services for the testbed.
0004
0005 ## Starting the Monitor
0006
0007 ### Dual Server Configuration
0008
0009 The monitor runs on two ports for different purposes:
0010
0011 ```bash
0012 # Start monitor with both HTTP and HTTPS
0013 ../swf-monitor/start_django_dual.sh
0014 ```
0015
0016 This creates:
0017 - **HTTP Server (port 8002)**: REST logging from agents (no authentication required)
0018 - **HTTPS Server (port 8443)**: Authenticated API calls (STF files, workflows, runs)
0019
0020 ### Environment Configuration
0021
0022 Required environment variables in `~/.env`:
0023
0024 ```bash
0025 # Monitor URLs for different purposes (development, local Django)
0026 export SWF_MONITOR_URL=https://localhost:8443      # Authenticated API calls
0027 export SWF_MONITOR_HTTP_URL=http://localhost:8002  # REST logging (no auth)
0028
0029 # Authentication token for API calls
0030 export SWF_API_TOKEN=your_token_here
0031 ```
0032
0033 ## API Endpoints
0034
0035 **Get Next Run Number** (used by DAQ simulator):
0036 ```bash
0037 POST /api/state/next-run-number/
0038 # Returns: {"run_number": 100015, "status": "success"}
0039 ```
0040
0041 **Get Next Agent ID** (used by all agents at startup):
0042 ```bash
0043 POST /api/state/next-agent-id/
0044 # Returns: {"agent_id": 42, "status": "success"}
0045 ```
0046
0047 Both endpoints are atomic and thread-safe for concurrent access.
0048
0049 ### Production Monitor URL for SSE example
0050
0051 The Remote SSE example agent is intended to connect to the production Apache-hosted monitor only. Set this in your `~/.env` (or rely on the built-in default):
0052
0053 ```bash
0054 # Canonical production monitor base (Apache path-aware)
0055 export SWF_MONITOR_PROD_URL=https://pandaserver02.sdcc.bnl.gov/swf-monitor
0056 ```
0057
0058 Notes:
0059 - The example receiver will use `SWF_MONITOR_PROD_URL` if set; otherwise it falls back to the canonical URL above.
0060 - It does not support the local Django dev server.
0061
0062 ### Production SSL trust (required)
0063
0064 Python requests must trust the production monitor certificate chain. Add this to your `~/.env`:
0065
0066 ```bash
0067 # Use the same full chain as production Apache
0068 export REQUESTS_CA_BUNDLE=/eic/u/wenauseic/github/swf-monitor/full-chain.pem
0069 # Alternative copy:
0070 # export REQUESTS_CA_BUNDLE=/eic/u/wenauseic/github/swf-common-lib/config/full-chain.pem
0071 ```
0072
0073 This ensures the receiver can validate the server certificate when calling:
0074 - `${SWF_MONITOR_PROD_URL}/api/messages/stream/status/`
0075 - `${SWF_MONITOR_PROD_URL}/api/messages/stream/`
0076
0077 ## Web Interface Access
0078
0079 - **Dashboard**: http://localhost:8002/ (main monitoring interface)
0080 - **STF Files**: http://localhost:8002/stf-files/ (file tracking)
0081 - **Workflows**: http://localhost:8002/workflows/ (processing status)
0082 - **Admin Interface**: http://localhost:8002/admin/ (Django admin)
0083
0084 ## API Endpoints
0085
0086 ### Authenticated APIs (HTTPS - port 8443)
0087
0088 Require `Authorization: Token <token>` header:
0089
0090 - `POST /api/runs/` - Create run records
0091 - `POST /api/stf-files/` - Register STF files
0092 - `POST /api/workflows/` - Create workflow records
0093 - `POST /api/workflow-stages/` - Track processing stages
0094
0095 ### REST Logging (HTTP - port 8002)
0096
0097 No authentication required:
0098 - `POST /api/logs/` - Agent logging endpoint
0099
0100 ## Getting API Token
0101
0102 From the monitor server:
0103 ```bash
0104 cd ../swf-monitor/src
0105 python manage.py get_token <username>
0106 ```
0107
0108 ## SSL Certificate Setup
0109
0110 The startup script automatically creates self-signed certificates for development:
0111 - `ssl_cert.pem` - SSL certificate
0112 - `ssl_key.pem` - Private key
0113
0114 For production, replace with proper certificates.
0115
0116 ## Troubleshooting
0117
0118 ### Common Issues
0119
0120 1. **"Connection refused"** - Monitor not started
0121    ```bash
0122    ../swf-monitor/start_django_dual.sh
0123    ```
0124
0125 2. **"HTTPS timeout"** - Wrong port or SSL issues
0126    - Check HTTPS server is running on port 8443
0127    - Verify SSL certificates exist
0128
0129 3. **"401 Unauthorized"** - Missing or invalid API token
0130    - Set `SWF_API_TOKEN` environment variable
0131    - Get new token with `manage.py get_token`
0132
0133 4. **"Empty STF files page"** - Agents not calling API
0134    - Verify monitor is serving HTTPS on 8443
0135    - Check agent logs for API call failures
0136    - Ensure `SWF_MONITOR_URL` points to HTTPS endpoint
0137
0138 ## Production SSE Receiver (example_agents)
0139
0140 This example is intended to run only against the production monitor under Apache.
0141
0142 - Required environment variables (can be placed in `~/.env`):
0143    - `SWF_API_TOKEN` — DRF token for the API
0144    - `SWF_MONITOR_PROD_URL` — Base URL of the production monitor (defaults to `https://pandaserver02.sdcc.bnl.gov/swf-monitor` if not set)
0145
0146 tcsh example:
0147
0148 ```tcsh
0149 setenv SWF_API_TOKEN your_token
0150 setenv SWF_MONITOR_PROD_URL https://pandaserver02.sdcc.bnl.gov/swf-monitor
0151 ```
0152
0153 The receiver connects to:
0154 - Stream: `${SWF_MONITOR_PROD_URL}/api/messages/stream/`
0155 - Status: `${SWF_MONITOR_PROD_URL}/api/messages/stream/status/`
0156
0157 ### Port Configuration Issues
0158
0159 **Problem**: STF files not appearing despite agents running
0160 **Cause**: Port/protocol mismatch between agent configuration and monitor
0161 **Solution**: Ensure environment variables match actual server configuration:
0162 - `SWF_MONITOR_URL` → HTTPS port (8443) for API calls
0163 - `SWF_MONITOR_HTTP_URL` → HTTP port (8002) for logging
0164
0165 ## Integration with Agents
0166
0167 Agents automatically use the correct endpoints:
0168 - **REST logging**: Uses `SWF_MONITOR_HTTP_URL` (HTTP, no auth)
0169 - **API calls**: Uses `SWF_MONITOR_URL` (HTTPS, token auth)
0170 - **Proxy bypass**: Automatic for localhost connections
0171
0172 See [Agent System](agents.md) for more details on agent configuration.