swf-monitor/docs/TFSLICE.md

0001 # TF Slices Monitoring
0002
0003 ## Overview
0004
0005 TF Slices (Time Frame Slices) are sub-ranges of a FastMon sample derived from a Super Time Frame (STF) file. Each slice covers a contiguous range of Time Frames and can be processed independently by a worker in approximately 30 seconds.
0006
0007 ## Purpose
0008
0009 The TF Slice system provides:
0010 - **Parallel Processing**: Each slice is assigned to an independent worker, enabling concurrent processing
0011 - **Bounded Work Units**: Fixed-size TF ranges make processing time predictable
0012 - **Load Distribution**: Slices spread work across available PanDA batch workers
0013
0014 ## Data Flow
0015
0016 ```
0017 StfFile (tf_count set at creation)
0018     └── FastMonFile  (fastmon agent samples STF → sets tf_first, tf_last, tf_count)
0019             └── TFSlice[]  (split of [FastMonFile.tf_first, FastMonFile.tf_last])
0020 ```
0021
0022 ### Step 1 — STF File creation
0023
0024 When the DAQ simulator registers an `StfFile`, it sets `tf_count` to the total number of Time Frames contained in the file.
0025
0026 ### Step 2 — FastMon sampling
0027
0028 The fastmon agent receives a notification about the new STF file and selects a contiguous sub-range of Time Frames to sample. It creates a `FastMonFile` record with:
0029
0030 | Field | Description |
0031 |---|---|
0032 | `tf_first` | Index of the first sampled TF. Must satisfy: `tf_first >= 0` and `tf_first < stf_file.tf_count` |
0033 | `tf_last` | Index of the last sampled TF. Must satisfy: `tf_last >= tf_first` and `tf_last < stf_file.tf_count` |
0034 | `tf_count` | Number of TFs in the sample: `tf_last - tf_first + 1` |
0035
0036 ### Step 3 — TF Slice creation
0037
0038 The workflow splits the FastMon sample into a set of `TFSlice` records. Each slice covers a sub-range within the FastMon window:
0039
0040 - `tfslice.tf_first >= fastmon_file.tf_first`
0041 - `tfslice.tf_last  <= fastmon_file.tf_last`
0042 - `tfslice.tf_count = tfslice.tf_last - tfslice.tf_first + 1`
0043
0044 Slices are numbered sequentially via `slice_id` (1-based within the FastMon file). Together, all slices for a given `FastMonFile` cover the full `[tf_first, tf_last]` range without gaps or overlaps.
0045
0046 ## Database Schema
0047
0048 ### TFSlice Model
0049
0050 | Field | Type | Description |
0051 |---|---|---|
0052 | `slice_id` | Integer | Serial number within the FastMon sample (1-based) |
0053 | `tf_first` | Integer | First TF index in this slice |
0054 | `tf_last` | Integer | Last TF index in this slice |
0055 | `tf_count` | Integer | Number of TFs in this slice (`tf_last - tf_first + 1`) |
0056 | `tf_filename` | String | Filename of the parent FastMon TF file |
0057 | `stf_filename` | String | Filename of the originating STF file |
0058 | `run_number` | Integer | Run number |
0059 | `status` | String | Processing status: `queued`, `processing`, `done`, `failed` |
0060 | `retries` | Integer | Number of retry attempts |
0061 | `assigned_worker` | String | Worker ID currently processing this slice |
0062 | `assigned_at` | DateTime | When the worker claimed the slice |
0063 | `completed_at` | DateTime | When processing finished |
0064 | `metadata` | JSON | Extensible metadata |
0065
0066 ### TF Range Constraints Summary
0067
0068 ```
0069 0 <= FastMonFile.tf_first < StfFile.tf_count
0070 FastMonFile.tf_first <= TFSlice.tf_first
0071 TFSlice.tf_last      <= FastMonFile.tf_last
0072                          FastMonFile.tf_last < StfFile.tf_count
0073 ```
0074
0075 ## Web Interface
0076
0077 ### Filtering Options
0078 - **By Status**: Filter slices by processing state
0079 - **By Run Number**: View all slices from a specific run
0080 - **By STF File**: View all slices derived from a specific STF
0081 - **By Worker**: View slices assigned to a specific worker
0082
0083 ### Table Columns
0084 - Slice ID
0085 - TF Filename (parent FastMon file)
0086 - STF Filename
0087 - Run Number
0088 - TF First / TF Last / TF Count
0089 - Status
0090 - Assigned Worker
0091 - Created / Completed Time
0092
0093 ## API Endpoints
0094
0095 ### REST API
0096 - `GET /api/tf-slices/` - List slices with filtering
0097 - `POST /api/tf-slices/` - Create new slice(s)
0098 - `GET /api/tf-slices/{id}/` - Get specific slice details
0099 - `PATCH /api/tf-slices/{id}/` - Update slice status or assignment
0100
0101 ### DataTables AJAX
0102 - `/tf-slices/datatable/` - Server-side processing endpoint for web interface
0103
0104 ## Monitoring Metrics
0105
0106 The system tracks:
0107 - Total slices created per run
0108 - Slice processing success/failure rate
0109 - Average processing time per slice
0110 - Worker utilization across slices
0111 - Retry distribution