EIC code displayed by LXR master/​jana2/​src/​examples/​InteractiveStreamingExample/​README.md	Source navigation Diff markup Identifier search General search
	Version: master test Revert   Change

Warning, /jana2/src/examples/InteractiveStreamingExample/README.md is written in an unsupported language. File is not indexed.

 
 ## Streaming Detector Plugin
 
 ### Overview
 
 The stream detector (streamDet) plugin facilitates both the online and offline analysis of streaming data. Simulated 
 data is generated by the `streamDetSource.py` script which produces a flat text (data) file.  The format of 
 the simulated data is available in a variety of formats and is discussed in further detail below.
 
 Moreover, the `streamDet` JANA2 plugin facilitates the decoding of SAMPA streaming data by either reading from the 
 file directly or by subscribing to ZMQ messages published in the INDRA messaging format which is discussed 
 in further detail below. The plugin decodes SAMPA streaming data and produces a ROOT file `outFile.root` which 
 contains leaves with the corresponding ADC and TDC data.  In addition, the plugin can publish INDRA 
 messages via. ZMQ so that the data can be monitored interactively and in real time via a Jupyter Notebook whose 
 infrastructure is contained in the `jupyter` sub directory.  Further details of the underlying infrastructure 
 is discussed below.
 
 ### Simulating Streaming Data via streamDetSource.py
 
 Below are the parameters which govern the simulated streaming data :
 
 | **Parameter (short)** | **Parameter (long)** | **Description**                                                | **Suggested Value** | **Required?** |
 | --------------------- | -------------------- | -------------------------------------------------------------- | ------------------- | ------------- |
 | -sr                   | --sampleRate         | Sampling rate in MHz                                           | 5                   | true          |
 | -nc                   | --numChans           | Number of channels to simulate ADC data for                    | 80                  | true          |
 | -ne                   | --numEvents          | Number of events to simulate for each channel                  | 100                 | true          |
 | -spec                 | --spectra            | ADC spectra to simulate (gumbel or sampa)                      | sampa               | true          |
 | -m                    | --mode               | SAMPA DAQ mode to simulate (das or dsp)                        | das                 | false         |
 
 #### Streaming Gumbel ADC & TDC Data
 
 In this mode, data are assumed to be comprised of 1024 ADC and TDC samples per readout window.  Each readout window 
 has been defined as an "event".  The simulated TDC data is a continuous stream of sampled TDC pulses whose frequency 
 depends on the sampling rate.  The ADC data is a sampled analog signal whose shape is governed by a Gumbel distribution 
 whose parameters are randomized from pulse to pulse.  Each sample of the ADC data is directly correlated with the 
 continuous stream of TDC pulses ergo, there exists a one to one mapping between the sampled data.  The streamDet plugin 
 no longer supports the decoding of this particular streaming data file format.  It is merely a pedagogical tool for 
 those who are interested.
 
 Example usage for producing a "stream" of ADC and TDC data according to a Gumbel distribution is as follows:
 
 ```
 python streamDetSource.py -sr 10 -nc 100 -ne 100 -spec gumbel
 ```
 
 #### Streaming SAMPA ADC Data in DAS Mode
 
 In direct adc serialization (DAS) mode the streaming SAMPA ADC data emanate from a 2.5 SAMPA chips sampling 80 
 streams of ADC data at 5 MhZ (200 ns/sample). Each readout window is comprised of 1024 ADC samples 
 (204.8 $`\mu s`$/readout window).  An "event is defined to be a single readout window. The simulated ADC 
 data is a sampled analog signal whose distribution is governed by a fourth order semi-Gaussian which is 
 inherent to the charge shaping amplifier present in the SAMPA chip signal processing architecture.  
 The pulse amplitude of the distribution is randomized for each pulse so that the full dynamic range 
 of the ADC (10-bit, 1024 channels) is populated. 
 
 Moreover, the time in which the signal occurs in the readout window is randomized such that it can occur at 
 any location in time in side the 204.8 $`\mu s`$ window however, it is prohibited to occur within the first 
 25% and last 75% of the readout window.  It is required to note that in DAS mode the SAMPA chips can only 
 sample the data at 5 MHz.  Higher sample rates are available in the digital signal processing (DSP) mode which 
 is discussed in more detail below. 
 
 Example usage for producing a "stream" of SAMPA ADC in DAS mode is as follows:
 
 ```
 python streamDetSource.py -sr 5 -nc 80 -ne 100 -spec sampa -m das
 ```
 
 Upon execution the data file `run-5-mhz-80-chan-100-ev.dat` will have been created in the present working directory.
 Each column of the data file corresponds to a single ADC channel and each row corresponds to the ADC sample value 
 for each of the 80 channels that were readout.
 
 #### Streaming SAMPA ADC Data in DSP Mode
 
 Coming soon...stay tuned!
 
 ### Compiling
 
 A few software prerequisites are required in order to successfully build and execute the stream detector plugin. 
 They are the following :
 
 1. The minimum required version of cmake is 3.13
 1. ROOT must be installed and the appropriate environment configured.  Consider one of the two options listed below:
     - `source /path/to/root/installation/bin/thisroot.(c)sh`
     - Manually configure the `ROOTSYS, PATH,` and `LD_LIBRARY_PATH` environment variables
 1. The ZeroMQ library must be present
     - Use your package manager to install the libraries system wide e.g. `yum install zeromq-devel`
 
 In order to compile JANA2 in conjunction with the stream detector plugin, one must execute the following commands:
 
 ```
 git clone git@gitlab.com:indra-astra/JANA2.git
 cd JANA2
 git checkout -b my-branch-name
 cmake . .
 make -j4
 ``` 
 
 ### Configuring the JANA2 Environment
 
 A few environment variables are required for facilitating the JANA2 framework.  In bash one should execute the 
 following commands (or make an alias, or add to .bashrc) :
 
 ```
 export JANA_HOME=/path/to/JANA2/
 export JANA_PLUGIN_PATH=$JANA_HOME
 export PATH=$PATH:$JANA_HOME 
 ```
 
 ### Plugin Components
 
 The various classes that comprise the stream detector plugin are described in the following subsections
 
 #### ADCSample
 
 The class `ADCSample` defines the components, and their associated data types, of ADC sample data as a JObject.  
 The components of the ADC data as a JObject are defined in the following table:
 
 | **Object Name** | **Data Type** | **Description**                                                              |
 | --------------- | ------------- | ---------------------------------------------------------------------------- | 
 | source_id       | `uint32_t`    | 32-bit identifier governed by the INDRA message format                       |
 | channel_id      | `uint16_t`    | Channel number corresponding to the ADC sample                               |
 | sample_id       | `uint16_t`    | Sample number corresponding to the ADC sample (analogous to a TDC sample)    |
 | adc_value       | `uint16_t`    | Value of the ADC sample                                                      |
 
 #### ADCSampleFactory
 
 The class `ADCSampleFactory` processes a `JEvent` object and constructs a `ADCSample` object.  In this instance 
 the "event" is a `DASEventMessage` object (defined in `INDRAMESSAGE.h`) which is discussed in detail in a later 
 section.  In short, the `DASEventMessage` object is a ZMQ message which JANA2 subscribed too.  A single message 
 contains 1024 ADC sample values (a single readout window) for 80 channels along with the member variables 
 available via the `DASEventMessage` class arsing from the INDRA messaging protocol.  The payload of the message 
 is then decoded so that the `ADCSample` object is constructed.  If desired one can configure the parameters 
 `streamDet:rawhit_ms` and `streamDet:rawhit_spread` in order to simulate a bottle neck in the processing method.  
 The default values are 200 ms (5 Hz) $`\pm`$ 0.25 $`\sigma`$.
 
 #### DecodeDASSource
 
 The class `DecodeDASSource` processes a `JEvent` object and constructs a `ADCSample` object.  In this instance
 the "event" is a vector of `ADCSample` objects which were populated from reading a data file from disk as opposed 
 to subscribing to ZMQ messages conforming to the INDRA messaging protocol as was done in the `ADCSampleFactory` 
 class.  More specifically, this class opens the data file specified by the parameter `streamDet:data_file` and 
 parses it.  It assumes that the file conforms to the SAMPA DAS ADC data format in that each event is comprised 
 of 1024 samples for 80 channels.  Once the end of the file has been reached, the file is closed and JANA will 
 terminate upon processing of all available events.
 
 #### INDRAMessage
 
 The INDRA message protocol is governed by that of the 
 [INDRA_Stream_Test](https://github.com/JeffersonLab/INDRA_Stream_Test) libraries.  Each data record begins with a 
 header followed by the data payload.  The header format is as follows :
 
 | **Type**     | **Name**          | **Comment**                                                                   |
 | ------------ | ----------------- | ----------------------------------------------------------------------------- |
 | **uint32_t** | source_id         | 32-bit identifier for this data source                                        |
 | **uint32_t** | magic             | 32-bit marker with the hexadecimal value 0xC0DA2019                           |
 | **uint32_t** | total_length      | The length of the entire record, including the header, in units of bytes      |
 | **uint32_t** | payload_length    | The length of the data that follows the header if the payload is uncompressed |
 | **uint32_t** | compressed_length | The length of the data that follows the header if the payload is compressed   |
 | **uint32_t** | format_version    | An integer value that identifies the header format                            |
 | **uint32_t** | flags             | 32-bit flag which provides a variety of information e.g. end of file reached  |
 | **uint64_t** | record_counter    | A count of the number of records sent since the connection opened             |
 | **uint64_t** | timestamp_sec     | 64-bit number of seconds in the 128-bit timestamp                             |
 | **uint64_t** | timestamp_nsec    | 64-bit number of nanoseconds in the 128-bit timestamp                         |
 
 The payload for each ZMQ message has been configured to contain 1024 samples of SAMPA DAS ADC data for 80 
 channels.  This corresponds to a single readout window for a 2.5 SAMPA chips whose data were transmitted via a 
 single GBTX0 chip on an ALICE FEC.
 
 The class `DASEventMessage` provides the various methods so that one can interact with the INDRA messages
 received via ZMQ. The `INDRAMessage` struct defined here is identical to the `stream_buffer` struct defined in 
 `INDRA_Stream_Test/stream_tools.h`. The constructors define the invariants of the message structure e.g. number 
 of samples in the message, buffer size, and channel count to name a few.  The destructor of course destroys the
 message once it has been processed.  
 
 There are three distinct blocks of code which define the behavior of the class.  The first is used by 
 `JStreamingEventSource` in order to figure out how to emit the message as an `JEvent`.  The second is a 
 series of setter definitions that are NOT required by `JStreamingEventSource` however, they are useful 
 for writing producers which can then be utilized for online monitoring.  The third section is utilized 
 by user-defined `JFactories` and `JEventProcessors` to access whatever data was sent. This provides a 
 "view" into the message buffer which the user can define however they like.
 
 #### JFactoryGenerator
 
 The class `JFactoryGenerator` simply generates the various factories specific to the plugin.  Nothing special here.
 
 #### MonitoringProcessor
 
 The `MonitoringProcessor` is a `JAppliction` that creates a ZMQ sink that publishes a `DASEventMessage` object to 
 the socket specified by the parameter `streamDet:pub_socket`.  For each ZMQ message that the stream detector plugin 
 subscribes too, the `MonitoringProcessor` publishes it so that it can be utilized in the online monitoring of data.
 
 #### RootProcessor
 
 The `RootProcessor` is a `JApplication` that defines the structure and components of the output ROOT file.  Any 
 branches, leaves, and/or histograms are defined and filled within this class.
 
 #### streamDet
 
 The `streamDet` is the `JApplication` that acts as the steering script for the stream detector plugin.  It is here that 
 the plugin is initialized within the JANA2 framework.  Both the plugin specific and command line parameters are 
 handled here.  Furthermore, the mode that the plugin is to be run in is handled here based on the user input, i.e. 
 weather the plugin be subscribing to ZMQ messages or simply reading the data from a flat text file. Each `JApplication` 
 that the user wishes to include in the analysis is also done here e.g. the `RootProccessor` application or the 
 `MonitoringProcessor` application.
 
 #### ZMQTransport
 
 The `ZMQTransport` class handles the subscribing and publishing of ZMQ messages within the JANA2 framework. 
 
 ### Plugin Parameters
 
 Below the various parameters of the `streamDet` plugin are displayed as well as their default values.
 
 | **Parameter**                 | **Description**                                                                          | **Default Value**                |
 | ----------------------------- | ---------------------------------------------------------------------------------------- | -------------------------------- |
 | streamDet:use_zmq             | Subscribe to ZMQ messages for data stream?                                               | true                             |
 | streamDet:data_file           | Name of data file to read if not subscribing to zmq messages                             | run-5-mhz-80-chan-100-ev.dat     |
 | streamDet:use_dummy_publisher | Reads from streamDet:data_file file and publishes ZMQ messages for JANA to subscribe to  | false                            |
 | streamDet:nchannels           | Number of channels in data stream                                                        | 80                               |
 | streamDet:nsamples            | Number of samples in data stream                                                         | 1024                             |
 | streamDet:msg_print_freq      | Frequency at which information should be printed to screen during analysis               | 10                               |
 | streamDet:sub_socket          | ZMQ subscribing socket                                                                   | tcp://127.0.0.1:5556             |
 | streamDet:pub_socket          | ZMQ publishing socket                                                                    | tcp://127.0.0.1:5557             |
 | streamDet:rawhit_ms           | Simulate delay in processing time (ms)                                                   | 200                              |
 | streamDet:rawhit_spread       | Spread in simulated delay in processing time ($`\sigma`$)                                | 0.25                             |
 
 ### Executing the Stream Detector Plugin
 
 The stream detector can operate in two functional modes.  One mode utilizes a dummy publisher to read a flat text from 
 disk and then creates a ROOT file and publishes INDRA messages via. ZMQ.  The following command will read the data 
 file `run-5-mhz-80-chan-100-ev.dat`, produce a ROOT file `outFile.root`, and publish the data as a INDRA ZMQ  
 message on the socket `tcp://127.0.0.1:5557`:
 
 ```
 jana -Pplugins=streamDet -PstreamDet:use_dummy_publisher=1 -PstreamDet:pub_socket=tcp://127.0.0.1:5557
 ```
 
 In the alternate mode JANA2 subscribes to INDRA messages via. ZMQ and creates a ROOT file `outFile.root` as well as 
 publishes those messages via ZMQ for online monitoring purposes.  The following command will receive INDRA ZMQ 
 messages on socket `tcp://127.0.0.1:5556` and publish them on socket `tcp://127.0.0.1:5557`.
 
 ```
 jana -Pplugins=streamDet -PstreamDet:use_dummy_publisher=0 -PstreamDet:sub_socket=tcp://127.0.0.1:5556 -PstreamDet:pub_socket=tcp://127.0.0.1:5557
 ```
 
 ### Publishing ZMQ INDRA Messages via INDRA_Stream_Test
 
 To obtain the `INDRA_Stream_Test` libraries one needs to first clone the 
 [INDRA_Stream_Test](https://github.com/JeffersonLab/INDRA_Stream_Test) repository on GitHub.  A few software 
 prerequisites are required in order to successfully build and execute the `INDRA_Stream_Test` libraries. They 
 are the following :
 
 1. The minimum required version of cmake is 3.13
 1. The ZeroMQ library must be present
     - Use your package manager to install the libraries system wide e.g. `yum install zeromq-devel` 
     and `yum install czmq-devel` 
 
 In order to build the libraries one should execute the following commands :
 
 ```
 git clone https://github.com/JeffersonLab/INDRA_Stream_Test
 cd INDRA_Stream_Test
 git checkout -b my-branch-name
 cmake . .
 make -j4
 ```
 
 In order to utilize the `INDRA_Stream_Test` libraries to stream SAMPA data to JANA2 one must first establish the stream 
 router that will receive data from the stream source over TCP and publish it over ZMQ to the JANA2 subscriber according 
 to the INDRA messaging protocol outlined above.  This is done with the following command :
 
 ```
 ./stream_router -z
 ```
 
 The router will receive a TCP stream input on port `5555` and publish the stream according to the INDRA messaging 
 protocol via ZMQ (as dictated by the `-z` parameter) on URL `tcp://127.0.0.1:5556` by default.  The ports can be 
 configured via command line parameters as documented in the `INDRA_Stream_Test` README.
 
 Once the router is successfully configured, one is required to initiate the stream source.  In the instance that one 
 would like to publish the simulated SAMPA DAS ADC data file `run-5-mhz-80-chan-100-ev.dat` discussed above, one would 
 need to execute the following command :
 
 ```
 ./stream_test_source -j -f run-5-mhz-80-chan-100-ev.dat
 ```
 
 Here the `-j` parameter configures the stream source library to operate in a mode hyper-specific to the JANA2 
 stream detector plugin.  The `-f` parameter reads the source file `run-5-mhz-80-chan-100-ev.dat` and streams it over 
 TCP on port `5555` which is in turn processed via the stream router discussed above.
 
 ### Processing the INDRA Messaging Stream via JANA2
 
 Once the stream router has been configured, one will want to execute JANA2 in a mode that is able to receive INDRA 
 messages via ZMQ from the stream source via the stream router.  To do this one should execute the following command :
 
 ```
 jana -Pplugins=streamDet
 ```
 
 This will fire up the stream detector JANA2 plugin in the default mode.  In the default mode, JANA2 will subscribe to ZMQ 
 messages on port `5556`, publish the INDRA messages via ZMQ on port `5557` for online monitoring, and write the contents 
 of the data stream to the ROOT file `outFile.root`. 
 
 ### Online Monitoring via Jupyter Notebook
 
 In order to monitor the streaming SAMPA ADC data one will need to first create a jupyter notebook session via 
 entering the follow command :
 
 ```
 jupyter notebook --ip=127.0.0.2 --port=5601
 ```
 
 A tab should have opened within your browser of choice.  Once there, navigate to the 
 `$JANA_HOME/src/plugins/streamDet/jupyter` directory and open the `streamDet_Monitoring.ipynb` notebook.  In the 
 first cell one needs to construct the widgets via `Shift+Enter`.  If successful, tabs containing the UI for port, 
 threshold, and plot selection should be visible.  By default the monitoring system will receive ZMQ messages published 
 via the JANA2 stream detector plugin on port `5557` with a threshold set to 100 ADC channels.  There are two plots 
 available for the user to view.  That is an ADC occupancy plot (loaded by default) and waveform plots which allow one 
 to see the individual waveforms for ADC pulses that exceed the programmed threshold.
 
 To begin subscribing to INDRA messages published by JANA2 via ZMQ, one simply needs to execute the second cell in the 
 same manner that was done for cell 1 (after making their plot selection of course).  With the router running, JANA2 
 subscribing, and the monitoring system subscribing, one can execute the stream source as discussed above.  If 
 successful, the plot type you selected withing the jupyter notebook should be created and self updating.
 
 Happy streaming!  

This page was automatically generated by the 2.3.7 LXR engine. The LXR team