swf-testbed/docs/prompt-processing-workflow.md

0001 # Prompt Processing Workflow
0002
0003 This document describes the prompt processing workflow for STF (Super Time Frame) files via PanDA/iDDS workers.
0004
0005 ## Overview
0006
0007 The prompt processing workflow enables rapid processing of detector data by:
0008 1. Simulating DAQ data taking (STF generation)
0009 2. Processing STF files through the Data Agent
0010 3. Submitting STF files to PanDA for batch processing
0011
0012 ### Pipeline Overview
0013
0014 ![Prompt Processing Pipeline](images/prompt-processing-workflow.png)
0015
0016 ## Data Products
0017
0018 ### Hierarchy
0019
0020 ```
0021 Run (run_id: 102363)
0022 ├── Dataset: swf.102363.run
0023 │   ├── STF Files (Super Time Frames)
0024 │   │   ├── swf.102363.000001.stf
0025 │   │   ├── swf.102363.000002.stf
0026 │   │   └── ... ...
0027 │   │
0028 │   └── PanDA Task
0029 │       └── Processes all STF files in dataset
0030 ```
0031
0032 ### Data Product Details
0033
0034 | Product | Created By | Stored In | Purpose |
0035 |---------|-----------|-----------|---------|
0036 | **STF File** | DAQ Simulator | STFFile table | Raw detector data unit |
0037 | **Dataset** |  Data Agent | Rucio | Collection of STF files for a run |
0038 | **PanDA Task** | Prompt Processing Agent | PanDA | Batch/Grid processing jobs |
0039
0040 ## Message Types
0041
0042 ### Broadcast Messages (DAQ Simulator → All Agents)
0043
0044 | Message | Payload | Purpose |
0045 |---------|---------|---------|
0046 | `run_imminent` | `execution_id`, `run_id`, `dataset` | Prepare for new run |
0047 | `start_run` | `run_id`, `state=run`, `substate=physics` | Begin data taking |
0048 | `stf_gen` | `filename`, `sequence`, `run_id` | New STF available |
0049 | `pause_run` | `run_id` | Temporary halt (standby) |
0050 | `resume_run` | `run_id` | Resume from pause |
0051 | `end_run` | `run_id`, `total_stf_files` | Run complete |
0052
0053 ## Configuration
0054
0055 ### prompt_processing_default.toml
0056
0057 ```toml
0058 [testbed]
0059 namespace = "user-namespace"
0060
0061 [workflow]
0062 name = "prompt_processing"
0063 version = "1.0"
0064 description = "Prompt processing of STF files"
0065 includes = ["daq_state_machine.toml"]
0066
0067 [agents.stf-data]
0068 enabled = true
0069 script = "agents/data_agent.py"
0070
0071 [agents.stf-processing]
0072 enabled = true
0073 script = "agents/prompt_processing_agent.py"
0074
0075 [simulation]
0076 # STF generation parameters
0077 stf_count = 10                   # Number of STF files to generate
0078 physics_period_count = 1        # Number of physics periods
0079
0080 # Timing parameters
0081 realtime = true                 # Run in real-time (1 sim second = 1 wall second)
0082 ```
0083
0084 ## Running the Workflow
0085
0086 ### Prerequisites
0087
0088 #### PanDA Setup
0089 - Use `--vo` option with value `"epic"`
0090 - For BNL PanDA submission, you need a **valid OIDC token issued by IAM**
0091
0092 #### Install panda-client
0093 ```bash
0094 # Create a venv
0095 python3 -m venv pclient
0096 source pclient/bin/activate
0097 pip install panda-client
0098 ```
0099
0100 #### Configure panda-client
0101 ```bash
0102 mkdir -p pclient/run
0103 cd pclient/run/
0104 ```
0105
0106 **setup.sh example:**
0107 ```bash
0108 source /etc/panda/panda_setup.sh
0109 export PANDA_URL_SSL=https://pandaserver01.sdcc.bnl.gov:25443/server/panda
0110 export PANDA_URL=https://pandaserver01.sdcc.bnl.gov:25443/server/panda
0111 export PANDACACHE_URL=https://pandaserver01.sdcc.bnl.gov:25443/server/panda
0112 export PANDAMON_URL=https://pandamon01.sdcc.bnl.gov
0113 export PANDA_AUTH=oidc
0114 export PANDA_AUTH_VO=EIC
0115 export PANDA_USE_NATIVE_HTTPLIB=1
0116 export PANDA_BEHIND_REAL_LB=1
0117 ```
0118
0119 ### Execution Methods
0120
0121 #### Option 1: Start Individually in CLI Mode
0122 ```bash
0123 cd swf-testbed
0124 python agents/data_agent.py
0125 python agents/prompt_processing_agent.py
0126 ```
0127 - The agent subscribes to ActiveMQ `/topic/epictopic`
0128 - When a `stf_ready` message is broadcasted, the agent submits the task to PanDA
0129
0130 #### Option 2: Start via Workflow Orchestrator
0131 Both `prompt-processing-agent` and `data-agent` can be started together:
0132 ```bash
0133 testbed run prompt_processing
0134 ```
0135
0136 ### Agent Configuration
0137
0138 #### Add Agents to supervisord (`agents.supervisord.conf`)
0139 ```ini
0140 [program:stf-data-agent]
0141 command=python -u agents/data_agent.py -v
0142 directory=%(ENV_SWF_HOME)s/swf-testbed
0143 environment=SWF_TESTBED_CONFIG="%(ENV_SWF_TESTBED_CONFIG)s"
0144 autostart=false
0145 autorestart=true
0146 stopwaitsecs=10
0147 stopsignal=QUIT
0148 stdout_logfile=%(here)s/logs/%(program_name)s.log
0149 stderr_logfile=%(here)s/logs/%(program_name)s.log
0150
0151 [program:stf-processing-agent]
0152 command=python -u agents/prompt_processing_agent.py -v
0153 directory=%(ENV_SWF_HOME)s/swf-testbed
0154 environment=SWF_TESTBED_CONFIG="%(ENV_SWF_TESTBED_CONFIG)s"
0155 autostart=false
0156 autorestart=true
0157 stopwaitsecs=10
0158 stopsignal=QUIT
0159 stdout_logfile=%(here)s/logs/%(program_name)s.log
0160 stderr_logfile=%(here)s/logs/%(program_name)s.log
0161 ```
0162
0163 ## Monitoring
0164
0165 ### Key Metrics
0166
0167 | Metric | Source | Purpose |
0168 |--------|--------|---------|
0169 | `stf_count` | WorkflowExecution | Total STFs in run |
0170 | `total_stf_files` | end_run message | STF files generated |
0171 | `dataset_status` | Processing Agent | Rucio dataset state |
0172 | `jediTaskID` | PanDA submission | Task identifier |
0173
0174 ### Monitoring Dashboards
0175
0176 - **SWF Monitor**: https://pandaserver02.sdcc.bnl.gov/swf-monitor
0177
0178 ### Monitoring Queries (MCP)
0179
0180 ```python
0181 # Workflow status
0182 get_workflow_monitor(execution_id='prompt_processing-username-xxxx')
0183
0184 # Messages during execution
0185 list_messages(execution_id='prompt_processing-username-xxxx')
0186
0187 # STF files for a run
0188 list_stf_files(run_number=102363)
0189
0190 # Agent logs
0191 list_logs(execution_id='prompt_processing-username-xxxx')
0192
0193 # PanDA jobs for a task
0194 panda_list_jobs(taskid=12345)
0195 ```
0196
0197 ## Example Execution
0198
0199 A typical prompt processing run:
0200
0201 ```
0202 Run 102363 Summary:
0203 ├── Duration: ~10 seconds (simulation time)
0204 ├── STF files: 2
0205 ├── Dataset: swf.102363.run
0206 ├── Messages:
0207 │   ├── run_imminent → agents prepared
0208 │   ├── start_run → physics phase began
0209 │   ├── stf_gen (x2) → STF files generated
0210 │   └── end_run → Dataset closed, PanDA submitted
0211 ├── Agents involved:
0212 │   ├── daq_simulator-agent-username-XXX
0213 │   ├── data-agent-username-XXX
0214 │   └── prompt-processing-agent-username-XXX
0215 └── PanDA:
0216     └── Single task submitted for entire dataset
0217 ```
0218
0219 ## See Also
0220
0221 - [Agent Management](agent-management.md) - Starting and stopping agents
0222 - [Architecture Overview](architecture.md) - System design
0223 - [Operations Guide](operations.md) - Day-to-day operations