Warning, /swf-monitor/docs/SNAPPER.md is written in an unsupported language. File is not indexed.
0001 # Snapper operations in SWF
0002
0003 Snapper is installed as a generic Django application in the SWF runtime and
0004 uses the existing `swfdb` PostgreSQL database. SWF owns the domain adapters,
0005 capture process, configuration, monitoring, and presentation.
0006
0007 The deployment follows the same infrastructure boundary as corun-ai: Apache
0008 serves Django views backed by database rows, while an independently supervised
0009 worker performs asynchronous work. The `epicprod-ops-agent` systemd service is
0010 Snapper's initial worker. It invokes standalone doers and never captures state
0011 inside an Apache request.
0012
0013 ## Capture scheduler
0014
0015 The ops agent evaluates both `testbed` and `epicprod` through:
0016
0017 ```text
0018 scripts/capture-system-snap.py --scope all
0019 ```
0020
0021 The agent polls every 10 seconds by default. The doer reads each scope's
0022 effective policy from `SysConfig` on every invocation, aligns the opportunity,
0023 and lets the PostgreSQL capture cursor make duplicate or quiet evaluations
0024 cheap. The database transaction serializes each scope and locks its registered
0025 components in stable name order. The agent stamps each invocation before
0026 starting the Django doer, so process-start latency cannot move an evaluation
0027 into a later boundary or manufacture a coverage gap.
0028
0029 The first read seeds these operator-visible `SysConfig` keys:
0030
0031 | Key | Commissioning default |
0032 |---|---:|
0033 | `snapper_opportunity_seconds_testbed` | `10` |
0034 | `snapper_baseline_every_testbed` | `10` |
0035 | `snapper_capture_policy_testbed` | `testbed-v1` |
0036 | `snapper_opportunity_seconds_epicprod` | `10` |
0037 | `snapper_baseline_every_epicprod` | `10` |
0038 | `snapper_capture_policy_epicprod` | `epicprod-v1` |
0039 | `snapper_lock_timeout_ms` | `5000` |
0040
0041 Opportunity intervals below 10 seconds are rejected. A lock timeout or another
0042 capture failure is recorded in the scope cursor; the next successful boundary
0043 creates a recovery snap. Periodic quiet and duplicate results remain in the
0044 bounded cursor and service journal. Material captures and failures enter the
0045 epicprod action stream.
0046
0047 ## Manual capture
0048
0049 An authorized operations publisher sends `capture_system_snap` to
0050 `/queue/epicprod.ops`, using the existing `prodops` namespace boundary:
0051
0052 ```json
0053 {
0054 "msg_type": "capture_system_snap",
0055 "namespace": "prodops",
0056 "scope": "testbed",
0057 "created_by": "operator-name"
0058 }
0059 ```
0060
0061 `scope` may be `testbed`, `epicprod`, or `all`. The doer targets the next
0062 aligned boundary when the current boundary has already been evaluated. Results
0063 are published as `snapper_capture_ready` on `/topic/epictopic` and recorded in
0064 the action stream with the requesting identity.
0065
0066 ## Operational status
0067
0068 System Status exposes one check per scope:
0069
0070 - `snapper-testbed-scheduler`
0071 - `snapper-epicprod-scheduler`
0072
0073 The checks read PostgreSQL cursor state only. They report stale heartbeats,
0074 consecutive failures, open coverage gaps, invalid SysConfig values, and the
0075 latest scheduler outcome. The heartbeat threshold is derived from the scope's
0076 opportunity interval, with a 60-second minimum.
0077
0078 ## Web presentation
0079
0080 The monitor mounts Snapper beneath the global System navigation. Its primary
0081 view state is represented by server routes rather than client-side tab state:
0082
0083 ```text
0084 /snapper/<scope>/report/
0085 /snapper/<scope>/report/<snap-uuid>/
0086 /snapper/<scope>/system/
0087 ```
0088
0089 `scope` is `testbed` or `epicprod`. The Report tab presents one selected full
0090 snap and the latest 100 history rows in a sortable table. Component envelopes
0091 use a generic registration-driven presentation; SWF-owned component presenters
0092 may add domain-specific rendering while retaining the complete component JSON.
0093
0094 The System tab presents the effective scope policy, scheduler cursor and
0095 health, and the complete component registration catalog. It is read-only;
0096 operator configuration remains in the existing System configuration editor.
0097 Tab, scope, and selected-snap links are durable URL state, so reload, browser
0098 history, bookmarks, and external proxy links preserve the selected view.
0099
0100 Deployments use the standard `deploy-swf-monitor.sh branch main` path. That
0101 freezes the generic `snapper-ai` package into the release virtual environment,
0102 runs migrations, updates `/opt/swf-monitor/current`, and restarts the
0103 `epicprod-ops-agent` worker.
0104
0105 ## Component maintainers
0106
0107 The full five-minute System status refresh also maintains two Snapper component
0108 families before the independent capture scheduler observes them:
0109
0110 - `health` in both scopes, from the bounded System status registry; and
0111 - `panda` in `epicprod`, from the curated PanDA activity adapter.
0112
0113 The PanDA adapter publishes trailing 24-hour job and task counts, current counts
0114 for every in-flight job state and nonterminal task state, running cores, and
0115 bounded target-site and task-type maps. It excludes users and individual
0116 records. Selected
0117 `refresh-system-status.py --only` runs skip the PanDA query; full manual and
0118 periodic refreshes publish it. A publication error causes the refresh doer to
0119 fail visibly while preserving the last accepted component state.