pages/contributing/physmon.md

0001 @page physmon Physics Monitoring in CI
0002 @brief Monitoring physics performance in the ACTS CI
0003
0004 The ACTS CI runs a suite of physics performance monitoring jobs dubbed
0005 *physmon*. The purpose is to monitor and detect changes in the physics
0006 performance, both intentional and accidental.
0007
0008 The associated job will run a number of workflow combinations. Currently, this
0009 includes the truth tracking and OpenDataDetector *full chain* workflows. The
0010 latter is further split into configurations with full seeding, truth smeared or
0011 truth estimated seeds. These jobs produce performance output files.
0012
0013 These performance output files are then evaluated using the programs dubbed
0014 [](analysis_apps). After this step, the job will then run comparisons of the
0015 diagnostics histograms created by these programs. If the comparisons indicate
0016 that the histogram contents changed meaningfully, the job will fail and report
0017 this on a pull request.
0018
0019 ## How do I investigate a physmon failure?
0020
0021 The physmon CI job attaches its results as an artifact to the CI run (also for successful runs)
0022 From your pull request, you need to click on the *Checks* tab at the top:
0023
0024 ![Checks button on GitHub](physmon/physmon_checks.png)
0025
0026 From there, click on the *Builds* workflow on the left:
0027
0028 ![Builds runs the physmon job](physmon/physmon_run.png)
0029
0030 On the workflow overview, scroll down to find the attached artifacts, and locate the *physmon* artifact. You can click to download it:
0031
0032 ![physmon artifact](physmon/physmon_artifact.png) {width=100%}
0033
0034 > [!note]
0035 > GitHub Actions artifacts are deleted after a fixed amount of time. If the CI
0036 > completed a while ago, it is possible this report is no longer available.
0037
0038 After the download, you need to unzip the archive, whose contents will look similar to this:
0039
0040 ```
0041 acts_analysis_residuals_and_pulls.root
0042 ckf_seeded.html
0043 ckf_seeded_plots
0044 ckf_truth_estimated.html
0045 ckf_truth_estimated_plots
0046 ckf_truth_smeared.html
0047 ckf_truth_smeared_plots
0048 ivf_seeded.html
0049 ivf_seeded_plots
0050 ivf_truth_estimated.html
0051 ivf_truth_estimated_plots
0052 ivf_truth_smeared.html
0053 ivf_truth_smeared_plots
0054 performance_ckf_seeded.root
0055 performance_ckf_truth_estimated.root
0056 performance_ckf_truth_smeared.root
0057 performance_truth_tracking.root
0058 performance_vertexing_seeded.root
0059 performance_vertexing_seeded_hist.root
0060 performance_vertexing_truth_estimated.root
0061 performance_vertexing_truth_estimated_hist.root
0062 performance_vertexing_truth_smeared.root
0063 performance_vertexing_truth_smeared_hist.root
0064 run.log
0065 truth_tracking.html
0066 truth_tracking_plots
0067 ```
0068
0069 The `.root` files are the performance output files and corresponding histogram
0070 files. The physmon job log file is also available. Finally, the `_plots` folder
0071 contain plots of all the histogram comparisons, and the `.html` files contain
0072 single-file reports showing the detailed results. An example of an HTML report
0073 looks like this:
0074
0075 ![Physmon report](physmon/physmon_report.png) {width=100%}
0076
0077 If you get a physmon job failure on your pull request, please investigate the
0078 failing report(s), and try to understand if the change causing the
0079 discrepancies is expected.
0080
0081 In case the conclusion is that the changes are indeed expected, the reference
0082 files used to generate the comparisons are located in `CI/physmon/reference/`,
0083 and can be updated to the output files found in the artifact zip archive.