swf-monitor/docs/EVENT_SERVICE_PLOTS.md

0001 # Event Service core plots
0002
0003 Task and job pages use the same Event Service report reader and SVG renderer.
0004 Finished-job metadata takes precedence. When metadata is absent, the reader uses
0005 the harness snapshot filed beside the production job. Task plots include active
0006 jobs and archived jobs, with at most eight timelines shown.
0007
0008 The existing five-minute close-out doer refreshes running jobs' snapshots before
0009 its normal pass. Its `--records-only` option refreshes observations without
0010 changing PanDA job state. The hourly report sweep retains ended jobs' records.
0011
0012 A running timeline ends at the snapshot timestamp. In-flight units and units
0013 awaiting an output close retain their separate states; they are not marked lost.
0014 The page displays the observation age and warns after two five-minute collection intervals.
0015 A terminal job without a normal harness-end marker is an incomplete observation.
0016 The worker-loss code 1256 is required before a shipped snapshot marks a cut,
0017 interrupted units or output lost on the node. Missing final metadata alone does
0018 not establish that outcome. Registered closes remain saved.
0019 The close summary separates started, saved, unfinished and failed closes.
0020 Its duration median includes completed closes only. An unfinished close carries
0021 its elapsed time at the observation, and new payload snapshots identify its
0022 current merge, validation or registration stage.
0023 That last independent observation also supplies the timeline end when PanDA's
0024 archived end time precedes shipped payload activity, as can happen when a job
0025 is reaped after losing its heartbeat.
0026
0027 The report transport and range accounting are documented in swf-epicprod
0028 `docs/JOB_REPORTING.md` and `docs/NODE_EVENT_DISPATCHER.md`.
0029
0030 Task range totals exclude a cancelled historical jobset copy when the same
0031 file/process range has a completed consumer. Consumer outcomes remain visible;
0032 a cancelled range without completed work remains cancelled. Normal ended
0033 harness observations are labelled final records rather than running snapshots.
0034
0035 Resident particle-gun records distinguish initialization, event processing,
0036 and setup/output handling within each unit. Simulation processing begins at
0037 the first event callback, including Geant4's lazy first-run preparation in
0038 initialization. Reconstruction uses the resident service's ready time and
0039 the measured request interval. Both live snapshots and final records retain
0040 these boundaries. Orange marks initialization; slate marks setup/output
0041 handling; outcome colors apply to measured simulation and reconstruction.
0042 Older aggregate intervals are mauve and carry an explicit unavailable
0043 measurement, rather than being counted as measured event processing.
0044
0045 Unit occupancy counts the full assigned interval. Measured saved processing
0046 counts only measured event-processing intervals whose output was saved,
0047 divided by observed allocated core-time. The two quantities are shown
0048 separately. Missing phase measurements never imply zero initialization.
0049
0050 A unit whose reconstruction ran on a shared pool member (`reconstruction.pool_member`,
0051 swf-epicprod `es/reco_pool.py`) has its reconstruction drawn on the member's own
0052 lane (`reco N`), with the member's startup in orange when that unit's request
0053 started it. The member lanes come after the cores. The members run on hardware
0054 threads apart from the job's cores, so their time is summed separately and left
0055 out of the cores' occupancy and saved processing. With asynchronous
0056 reconstruction the slot takes its next unit as the simulation ends, so a pooled
0057 unit holds its core until the next unit on that core starts; a core's last unit
0058 holds it until its simulation ends. Before this, the overlap was drawn twice on the
0059 core and counted twice, and turnaround clamped to zero.
0060
0061 Turnaround between units is the time on a core from the end of one unit's last
0062 measured simulation or reconstruction interval to the start of the next unit's
0063 first. A pooled unit's reconstruction is on a member's lane, so for it only
0064 the simulation counts. The page gives its median, 90th percentile and longest value and the
0065 number of transitions. A unit without measured processing breaks the
0066 sequence, so two gaps are never joined into one. It covers per-unit setup
0067 and output handling, the wait for the next unit and the simulation's own
0068 per-request start (swf-epicprod EPICPROD_PAYLOAD.md, Resident Event Service
0069 units).