Warning, /eic.github.io/_resources/storage.md is written in an unsupported language. File is not indexed.
0001 ---
0002 title: Distributed Storage
0003 description: Storage
0004 name: storage
0005 layout: default
0006 ---
0007
0008 {% include layouts/title.md %}
0009
0010 #### S3
0011 The main storage for EIC-related files is on S3 interface, for EIC use.
0012
0013 This storage is accessed via the "MinIO" client - named *"mc"* - available for download from the MinIO site on its
0014 {% include navigation/findlink.md name='minio_download' tag='download page' %}, or installed on the farms and inside `eic-shell`.
0015 Please obtain the read-only access key from the ePIC mattermost channel. Then, in a bash-like environment add
0016 ```
0017 export S3_ACCESS_KEY=[ask in mattermost]
0018 export S3_SECRET_KEY=[ask in mattermost]
0019 ```
0020 to your initialization file (e.g. `.bashrc`), or
0021 ```
0022 setenv S3_ACCESS_KEY [ask in mattermost]
0023 setenv S3_SECRET_KEY [ask in mattermost]
0024 ```
0025 to your equivalent init file (e.g. `.cshrc`).
0026 You then run
0027 ```
0028 mc config host add S3 https://eics3.sdcc.bnl.gov:9000 $S3_ACCESS_KEY $S3_SECRET_KEY
0029 ```
0030 and from then on you can use commands like
0031 ```
0032 mc ls S3/eictest/EPIC/Tutorials
0033 ```
0034
0035
0036 ##### NextCloud/BNLbox: pre-generaged event datasets
0037
0038 The EIC group at BNL has a large amount (13 TB and growing) of
0039 pre-generated events from Pythia, Milou, Djangoh, and others available
0040 for download via NextCloud/BNLbox.
0041 To access this data, no lab association is necessary. Simply contact
0042 Kolja Kauder, kkauder_at_bnl.gov, and request a guest account.
0043 You will receive an invitation where you can choose your password. From then on, you can log into
0044 https://bnlbox.sdcc.bnl.gov
0045 and find everything in the ```EIC_Data``` directory via the web interface.
0046
0047
0048 ##### Download from the command line
0049
0050 Instructions for using curl, cadaver, and a script that can download entire directories can be found
0051 {% include navigation/findlink.md name='bnlbox_cli' tag='here' %}.
0052 A few additional points that may simplify life:
0053 * A copy of the ```copy_bnl_box.rb``` script is found inside ```EIC_Data```.
0054 * Note to RCF users, you can access this data directly at
0055 ```bash
0056 /gpfs02/eic/DATA
0057 ```
0058 * IMPORTANT: As the documentation notes, guest accounts work simply with the email + password combination.
0059 However, do choose a reasonably strong password, but do NOT reuse a password you use for any other service!
0060 If you use the command line interface, this password may well be sent unencrypted and/or be in a clear-text
0061 file on your computer.
0062 * Assuming your guest account was created for myemail@myprovider.com, the path to the directory will always have a specific form and you can use an environment variable as a shortcut:
0063 ```bash
0064 export eicdataurl='https://bnlbox.sdcc.bnl.gov/remote.php/dav/files/myemail@myprovider.com/EIC_Data'
0065 ```
0066 or
0067 ```bash
0068 setenv eicdataurl 'https://bnlbox.sdcc.bnl.gov/remote.php/dav/files/myemail@myprovider.com/EIC_Data'
0069 ```
0070 This way, you can (with a suitably configured .netrc) use something like:
0071 ```bash
0072 curl -O -n ${eicdataurl}/copy_bnl_box.rb
0073 ```
0074
0075
0076 ##### Adding to the collection
0077 There is more than enough space available to fulfill all storage needs
0078 of the Yellow Report effort as reported in the recent poll to working
0079 groups. To make use of it, please contact the software working group
0080 so we can determine together the most efficient way to upload your
0081 data and make it accessible to the whole community!
0082
0083 <hr/>
0084
