Warning, /acts/docs/examples/howto/run_ckf_auto_tuning.rst is written in an unsupported language. File is not indexed.
0001 ACTS Tutorial on Auto-Tuning in CombinatorialKalmanFilter (CKF)
0002 ===============================================================
0003 The tracking algorithms require a number of pre-initialized parameters that are often hand-tuned to obtain high performance. Usually, the value of these parameters change as the underlying geometrical or magnetic configuration changes. An automatic tuning of these parameters can be very useful for obtaining highly efficient parameter configuration as well as for studying different detector geometries. This tutorial is based on parameter optimization studies using two different optimization frameworks: Optuna and Orion. Eight parameters of Track Seeding algorithm have been tuned using these frameworks and their performance have been studied on CKF.
0004
0005 Prerequisites
0006 -------------
0007 Since Optuna and Orion are independent frameworks, these need to be installed separately. The following setup works on any machine with cvmfs access that requires creating a virtual python environment:
0008
0009 .. code-block:: console
0010
0011 $ source /cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc10-opt/setup.sh
0012 $ python3 -m venv PYTHON_VIRTUAL_ENV
0013 $ source PYTHON_VIRTUAL_ENV/bin/activate
0014 $ export PYTHONPATH=
0015 $ python -m pip install --upgrade pip
0016 $ pip install -r acts/Examples/Python/tests/requirements.txt
0017 $ pip install pytest --upgrade
0018 $ pip install cmake dataclasses sphinxcontrib-applehelp sphinxcontrib-jsmath sphinxcontrib-serializinghtml argparse sphinxcontrib-devhelp sphinxcontrib-htmlhelp sphinxcontrib-qthelp AppDirs filelock joblib pandas plotly psutil pyYAML requests scipy tabulate cloudpickle scikit-learn orion==0.2.2 cloudpickle==1.6.0 optuna matplotlib
0019
0020 Once all the dependencies are installed, Build ACTS with python bindings and Pythia8 support.
0021
0022 .. code-block:: console
0023
0024 $ cmake -DACTS_BUILD_EXAMPLES_PYTHIA8=ON -DACTS_BUILD_PLUGIN_DD4HEP=OFF -DACTS_BUILD_PLUGIN_JSON=ON -DACTS_BUILD_PLUGIN_TGEO=ON -DACTS_BUILD_EXAMPLES_DD4HEP=OFF -DACTS_BUILD_EXAMPLES_GEANT4=ON -DACTS_BUILD_INTEGRATIONTESTS=OFF -DACTS_BUILD_UNITTESTS=OFF -DACTS_BUILD_EXAMPLES_PYTHON_BINDINGS=ON -DACTS_BUILD_ODD=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_STANDARD=17 -S . -B build/
0025 $ make
0026 $ source build/python/setup.sh
0027
0028 Once this setup is ready, at each new login, just do:
0029
0030 .. code-block:: console
0031
0032 $ source /cvmfs/sft.cern.ch/lcg/views/LCG_100/x86_64-centos7-gcc10-opt/setup.sh
0033 $ source PYTHON_VIRTUAL_ENV/bin/activate
0034 $ export PYTHONPATH=
0035 $ source build/python/setup.sh
0036
0037 How auto-tuning works
0038 ---------------------
0039 A list of parameters and their range are provided to optimization framework. The framework predict a value for each parameter and provide them to the tracking algorithm. The tracking algorithm runs with these values and return the output to the optimization framework. The optimization framework analyzes the output and provide a new set of values to tracking algorithm and this cycle continues for a number of trials. At each trial, the optimization framework computes a score based on the tracking algorithm output and tries to minimize/maximize this score by suggesting better parameter configurations. The current frameworks compute the score based on the CKF track efficiency, fake rate, duplicate rate and run-time:
0040
0041 Score = Efficiency - (fakeRate + DuplicateRate/k_dup + run-time/k_time)
0042
0043 where k_dup and k_time are the weights for duplicate rate and run-time. These weights play a significant role in determining the configuration of best performing parameters from the optimization frameworks.
0044
0045 The list of track seeding parameters that are auto-tuned using these frameworks are as follows:
0046 * maxPtScattering: upper p_T limit for scattering angle calculations
0047 * impactMax: maximum value of impact parameter
0048 * deltaRMin: minimum distance in r between two measurements within one seed
0049 * deltaRMax: maximum distance in r between two measurements within one seed
0050 * sigmaScattering: number of sigma used for scattering angle calculations
0051 * radLengthPerSeed: average radiation lengths of material on the length of a seed
0052 * maxSeedsPerSpM: number of space-points in top and bottom layers considered for compatibility with middle space-point
0053 * cotThetaMax: maximum cotTheta between two space-points in a seed to be considered compatible
0054
0055 The python scripts utilizing these optimization using Optuna and Orion frameworks are present in acts/Examples/Scripts/Optimization directory.
0056
0057 Run auto-tuning using Optuna
0058 ----------------------------
0059 The Optuna auto-tuning script can be run directly by invoking:
0060 `` python Optuna_tuning.py``
0061
0062 This creates a new optuna study for a given number of trials defined within the script. The direction is set to maximize which means that the framework will try to maximize the score.
0063
0064 .. code-block:: console
0065
0066 $ study = optuna.create_study(study_name=study_name,
0067 storage="sqlite:///{}.db".format(study_name),
0068 direction='maximize',
0069 load_if_exists=True)
0070
0071 The objective function defines the list of parameters to tune along with their range. It suggests a parameter configuration using ``trial.suggest_float`` or ``trial.suggest_int`` function of optuna and runs CKF for this parameter configuration. The output of CKF is read from the performance file named ``performance_ckf.root`` and a score function is constructed within the objective function.
0072
0073 The objective function and the number of trials are passed to optuna optimize function:
0074 ``study.optimize(objective, n_trials=100)``. The best parameter configuration after all the trials is read from ``study.best_trial.params.items()``.
0075
0076 Run auto-tuning using Orion
0077 ---------------------------
0078 The Orion auto-tuning script can be run directly by invoking:
0079 `` python Orion_tuning.py``
0080
0081 A dictionary called space is created by providing a list of parameters and their range and new experiment is build using
0082
0083 .. code-block:: console
0084
0085 $ experiment = build_experiment(
0086 "orion_new",
0087 space=space,
0088 storage=storage)
0089
0090 The objective function picks up a value for each parameter, run CKF and construct a score function from CKF output as in Optuna case. The only difference is that it tries to minimize the score unlike optuna. The objective function and number of trials are passed to the orion workon function
0091 ``experiment.workon(objective, max_trials=100)``.
0092
0093 The best parameter configuration corresponds to the minimum score function value and can be obtained from the experiment.
