Warning, /swf-monitor/docs/NOTICE_ROUTING.md is written in an unsupported language. File is not indexed.
0001 # Notice Routing
0002
0003 Notice routing extends the action stream ([ACTION_STREAM.md](ACTION_STREAM.md))
0004 with subscriptions: named incidents matched to autonomously registered
0005 subscribers and delivered per subscriber. The emitting code declares only the
0006 incident; it never names a recipient. Consumers — external systems like personal
0007 feed aggregators, and delivery plugins like the Mattermost publisher —
0008 register their own interest and receive matching incidents through their chosen
0009 delivery mode.
0010
0011 The design separates three concerns that today are partially fused:
0012
0013 - **Incidents** — what happened, recorded once, named.
0014 - **Subscriptions** — who wants what, registered by the consumer.
0015 - **Delivery** — how a matched incident reaches each subscriber.
0016
0017 [](https://raw.githubusercontent.com/BNLNPPS/swf-monitor/infra/baseline-v42/docs/notice_routing.svg)
0018
0019 ## Incidents
0020
0021 The action stream is the incident source. The `action` identifier is the incident
0022 name; the record's structured fields (`subject_type`, `subject_key`,
0023 `subject_label`, `outcome`, `namespace`, free keys) are its attributes.
0024 `subject_key` remains the canonical machine identity; `subject_label` may
0025 provide its human-readable presentation. `ACTION_DEFAULTS` is the incident-name
0026 catalog. Emission sites change nothing: recording an action is publishing an
0027 incident.
0028
0029 Routing operates on the structured action space across logging namespaces
0030 (`app_name`), independent of the `sublevel` and `live` axes — those govern
0031 the built-in human channels (live page, `#epicprod-live`, digests), while
0032 subscriptions are the extension mechanism for systems. A subscription may
0033 filter on importance, but a low-importance mechanical incident is fully
0034 subscribable.
0035
0036 ## Subscriptions
0037
0038 A `NoticeSubscription` row: subscriber name, incident name (exact or trailing
0039 wildcard; the API field is `event`), attribute filters (equality matches over the record's structured
0040 fields; a list-valued filter means membership, so one subscription covers a
0041 value set such as `{"operation": ["pause", "resume"]}`), delivery mode,
0042 enabled flag, creator. Two reserved filter keys reach beyond the incident's
0043 own attributes: `app_name` matches the record's logging namespace, and
0044 `live` matches the incident's effective live state — the runtime live-policy
0045 override where one exists, else the record's `live_default` — so the live
0046 stream is selectable as a subscription. Token-authed CRUD at
0047 `/api/notices/subscriptions/`, so a third-party system registers and
0048 maintains its own subscriptions without swf code changes. Subscription
0049 changes are themselves logged actions.
0050
0051 ## Delivery
0052
0053 Two modes, chosen per subscription:
0054
0055 - **Buffered pull** — the router writes one notice row tagged with the
0056 subscriber name; the subscriber drains its buffer over REST from its own
0057 side. This generalizes the existing Capcom store (`CapcomNotice` gains the
0058 subscriber tag; the existing drain endpoint and retention behavior carry
0059 over), and preserves the external-consumer boundary: swf buffers locally
0060 and holds no credential into any external system.
0061 - **Push plugin** — the router hands the incident to a named in-process plugin:
0062 the subscription's delivery value is the plugin name, resolved in the
0063 registry in `monitor_app/notice_plugins.py`. Push delivery is
0064 at-most-once — a delivery failure is logged and the pass continues, so a
0065 push outage never stalls buffered-pull delivery; the incident remains on the
0066 log record page. The `#epicprod-live` Mattermost publisher is the first
0067 plugin (`mattermost-live`); each plugin's settings live in SysConfig.
0068
0069 Notice composition from the incident is deterministic: title from the operation
0070 when present, otherwise the action, plus `subject_label` when present, otherwise
0071 `subject_key`; detail from `narration`/`reason`/`summary` where present, URL to
0072 the log record, severity from the outcome, dedup key from the record id.
0073 Composition belongs to the router, not to emitters or subscribers. Incidents
0074 may carry `severity` (how bad — e.g. an assessment verdict) and `url`
0075 (where to look — a subject page rather than the log record) as ordinary
0076 attributes; the router honors them and absolutizes a path-form `url` onto
0077 the external face.
0078
0079 The router is a stream tailer — the proven publisher pattern: a single
0080 polling service that matches each new record against enabled subscriptions
0081 and performs deliveries, with per-cycle caps and counted overflow, never
0082 silent drops. Emission stays pure and never blocks or fails on delivery
0083 problems.
0084
0085 ## Workflow completion incidents
0086
0087 Workflow execution status changes arrive by plain REST update and record no
0088 action today. The execution update path gains a terminal-transition incident:
0089 when status reaches `completed` or `failed`, it logs
0090 `workflow_execution_completed` (namespace, execution id, status, elapsed,
0091 and a `notice` attribute carried from the execution's parameters).
0092 `testbed run --notice` (equivalently a `testbed.toml` key) stamps the run.
0093
0094 The first use is the nightly testbed heartbeat: the overnight cron run
0095 carries the stamp, and a subscription (subscriber `capcom`, incident
0096 `workflow_execution_completed`, filter `notice=true`) delivers exactly one
0097 notice per overnight run into the operator's feed — one notice for each
0098 day the testbed ran, with warning severity on failure.
0099
0100 ## Migration
0101
0102 The direct Capcom posters (ops-agent pause/resume terminal notices, the
0103 assessment and delivery-daily notices) emit matching stream incidents carrying
0104 the needed fields; each is a subscription and its bespoke posting code is
0105 retired. The subscriptions replacing them: `panda_task_operation` with
0106 `operation` in pause/resume (single and bulk, terminal outcomes only —
0107 the incidents carry the human count line as `summary` and the task page as
0108 `url`); `assessment_register` filtered to the scheduled kinds (narration,
0109 verdict severity, report url); `assessment_enforce` filtered to error
0110 outcomes (salvage and quarantine, floor-verdict severity); and
0111 `delivery_daily_rebuild` (newest-day arrivals as `summary`, the campaign
0112 view as `url`). The `/api/capcom/notices/ingest/` endpoint remains for
0113 genuinely external posters. The Mattermost publisher's incident selection
0114 (live + importance threshold) is the `epicprod-live` subscription — incident
0115 `*`, filters `{"app_name": "epicprod", "live": true, "sublevel": ["high",
0116 "normal"]}`, delivery `mattermost-live`; its formatting is unchanged. The
0117 importance threshold is the subscription's `sublevel` list, edited over
0118 REST; the `epicprod_live_min_sublevel` SysConfig knob is retired. The
0119 channel name (`epicprod_live_channel`) and poll cadence
0120 (`epicprod_live_poll_seconds`) remain SysConfig knobs.
0121
0122 ## Delivery sequence
0123
0124 1. The router service, the subscription model and REST, buffered-pull
0125 delivery on the generalized store, and the workflow-completion incident —
0126 the nightly heartbeat working end to end.
0127 2. Migration of the direct Capcom posters to subscriptions.
0128 3. The Mattermost publisher as a push plugin.
0129
0130 ## Status
0131
0132 Step 1 is deployed (2026-08-11) and verified end to end: the router runs
0133 in the publisher cycle (`monitor_app/notice_router.py`), subscriptions
0134 serve at `/api/notices/subscriptions/`, and the first subscription —
0135 `capcom ← workflow_execution_completed`, filter `notice=true` — delivers
0136 the nightly testbed heartbeat from the stamped `testbed run --notice`
0137 cron run. Step 2 (2026-08-12) migrated the direct Capcom posters to the
0138 four subscriptions listed under Migration and retired their posting code.
0139 Step 3 (2026-08-12) moved the Mattermost publication into the
0140 `mattermost-live` push plugin, selected by the `epicprod-live`
0141 subscription; the `publish_epicprod_live` command is now only the tailer
0142 loop around the routing pass.