An ESG Walkthrough -ESG Federation website -- DCC File system for ESG Muhammad Atif

ESG-NCI Gateway Latest news Search We highly recommend subscribing to our tweets We highly recommend subscribing to our tweets

Search and Access Data You can search without login; but not download Quick Links → Create Account Follow the on screen instructions You will receive an to confirm your registration; at the same time the administrators are also notified. Admin has to validate you before you can download the data. After confirmation from admin; login to the NCI Node Account → Apply for Group Membership  Mk-3.6  CMIP-5 research (not in our control – request goes to PCMDI) Requests to NCI are usually quick. Same ID can be used on all the gateways. Your OpenID:

Searching for data Recommend that you browse the website and get familiarized. If asked about authentication, you may use a temporary openid Password: abc123 Please note that this openid will be removed after the workshop, it is highly recommended that you create your openid Recommend that you browse the website and get familiarized. If asked about authentication, you may use a temporary openid Password: abc123 Please note that this openid will be removed after the workshop, it is highly recommended that you create your openid

Download data from the Gateway Three Download Methods Using the web browser. A set of wget-scripts. Via GridFTP (Data mover lite)

Download Method-1 (Web browser based) Intuitive but slow Follow on-screen instructions and nothing can go wrong. Works like normal downloads from the browser of your choice i.e. click to download. IE, FF, Chrome and Safari are supported. Works well if you are after couple of files

Download -2 (Wget Scripts) Ability to download multiple files Select the files (variables) you are interested in. Presents a wget-download.sh script, that you need to save and run. Command line based – No GUI Two methods of Authentication  Authorization token (Depreciated)  PCMDI gateway only.  No login required, However the token expires in 24 hours and the script is of no use after that.  My Proxy Login (Official)  Needs a separate step for authentication.  Need to run Java applet or MyProxyClient software  If authentication expires, just run the MyProxyClient software Note: If you are interested in doing lots of downloads, we can provide a custom script to speedup the process on DCC ……… Example to follow later

Process for MyProxyLogon Download on the DCC ssh Download MyProxyLogon-ESG.jar file wget ESG.jar Run MyProxyLogon.jar file; module load java Java –jar MyProxyLogon-ESG.jar It writes the certificates to your $HOME/.esg folder Run the wget-download.sh command

Download-3 (DML) Parallel Downloads DML  Preferences Concurrency Faster than wget Uses GridFTP Caveat: Not available on all ESG nodes NCI one of the few that has the facility

ESG Data on the DCC IPCC AR5/CMIP5 CSIRO-QCCCE Mk-3.6 CAWCR ACCESS Replicated data from the other ESG nodes. Other data CMIP3 Observational data Processed data

DCC File system organization All ESG data in /projects/ESG : Authoritative Unofficial-ESG-replica CAWCR_CVC_processed /projects/ESG/Authoritative Serves data using the policies of the ESG Federation This is the directory that our ESG software serves data from All data is the current official copy User example: login to the DCC and have a look.

Unofficial Replica /projects/ESG/Unofficial_Replica IPCC  The IPCC directory is where you can reference data that we have downloaded from other nodes (though not an official replica). The subdirectories could be partial datasets or complete ones. IPCC_tmp_flat Direct symlinks to files, flat directory structure tmp  You can download your data here in $USER folder.  We can provide you with scripts to help download data here GlobalObs_and_Reanalysis  data sourced from various places that Lawrie Rikus/Ben Hu have been maintaining.  Also served through a THREDDS service - for remote access

Unofficial Replica /projects/ESG/Unofficial-ESG-replica/IPCC User downloads using wget scripts/DML. Partial data; Not all of the data is downloaded. Does not necessarily contain the most up to date version  Data may be changed by the remote node since last download.  ESG (and official replica directory) always has the latest version. Organised as Data Reference Syntax (DRS)

Data Reference Syntax (How files are DCC) This is how the tree looks like compared to DRS DRS cmip File System /projects/ESG/unofficial-ESG-replica/IPCC/CMIP5/output1/NCC/NorESM1- M/historical/mon/seaIce/OImon/r1i1p1/v /sic/

Downloading data to DCC File System If you would like a significant amount of data that we don’t have, then … please contact us. Reasons: It may already be downloaded but not linked Downloading data is still tricky Space management That said – we would like to facilitate downloads of priority data. How? …. Lets do it

Demo Download the wget file from esg-gateway ssh –Y java –jar MyProxyLogon-ESG.jar Copy wget-file to dcc (scp, copy n paste) in a new folder./esg-download.py wget-download.sh View the directory, it should have a number of wget-split-*./esg-qsub-download.py –i wget-split- Press “y” Check the files after some time We will be streamlining it further

Help DCC and ESG both are evolving continuously Comments and suggestions are always welcome Help Desks – Anything related to ESG federation website/ other models are not native to NCI – Related to DCC compute cluster

Downloads are managed by Us GridFTP Fast Downloads managed by you as a user

Controlled Vocab: documentation/cmip5_controlled_vocab.txt

ESG-NCI Gateway We highly recommend subscribing to our tweets Search by categories

Search and Access Data You can search without login; but not download Quick Links → Create Account Follow the on screen instructions You will receive an to confirm your registration; at the same time the administrator(s) are also notified. Admin has to validate you before you can download the data. After confirmation ; login to the NCI Node Account → Apply for Group Membership  Mk-3.6  CMIP-5 research (not in our control – request goes to PCMDI) Requests to NCI are usually quick. For others, this may take time (one – two days)

Searching for data

Download data from the Gateway Three Methods Using the web browser. A set of wget-scripts. Via GridFTP (Data mover lite)

Download – 1 (Web based) Intuitive but slow Follow on-screen instructions and nothing can go wrong. Works like normal downloads from the browser of your choice IE, FF, Chrome and Safari are supported. Works well if you are after couple of files

Download -2 (Wget Scripts) Ability to download multiple files Presents you with a wget-download.sh script. Command line based – No GUI Two methods  Authorization token (Depreciated)  PCMDI gateway only.  My Proxy Login (Official)  Needs a separate step for authentication  Need to run Java applet or MyProxyClient software Note: If you are interested in doing lots of downloads, we can provide a custom script to speedup the downloads on DCC.

Process for MyProxyLogon Download on the DCC ssh Download MyProxyLogon-ESG.jar file wget ESG.jar Run MyProxyLogon.jar file; instructions are provided in the wget download script that you have already downloaded via web-browser. module load java Java –jar MyProxyLogon-ESG.jar It writes the certificates to your $HOME/.esg folder

Download-3 (DML) Parallel Downloads DML  Preferences Concurrency Faster than wget Uses GridFTP Caveat: Not available on all ESG nodes NCI one of the few that has the facility

Interacting with ESG data on DCC

ESG Data on the DCC IPCC AR5/CMIP5 CSIRO-QCCCE Mk-3.6 CAWCR ACCESS Replicated data from the other ESG nodes. Other data CMIP3 Observational data Processed data

DCC File system organization All ESG data in /projects/ESG : Authoritative Unofficial-ESG-replica CAWCR_CVC_processed /projects/ESG/Authoritative Serves data using the policies of the ESG Federation This is the directory that our ESG software serves data from All data is the current official copy. User example: login to the DCC and have a look.

Unofficial Replica /projects/ESG/Unofficial_Replica IPCC  The IPCC directory is where you can reference data that we have downloaded from other nodes (though not an official replica). The subdirectories could be partial datasets or complete ones. IPCC_tmp_flat Direct symlinks to files, flat directory structure tmp  You can download your data here in $USER folder.  We can provide you with scripts to help download data here GlobalObs_and_Reanalysis  data sourced from various places that Lawrie Rikus/Ben Hu have been maintaining.  Also served through a THREDDS service - for remote access

Unofficial Replica /projects/ESG/Unofficial-ESG-replica/IPCC User downloads using wget scripts/DML. Partial data; Not all of the data is downloaded. Does not necessarily contain the most up to date version  Data may be changed by the remote node since last download.  ESG (and official replica directory) always has the latest version. Organised as Data Reference Syntax (DRS) cmip Ref the official link to the standard.

Downloading data to our ESG If you would like a significant amount of data that we don’t have, then … please contact us. Reasons: It may already be downloaded but not linked Downloading data is still tricky Space management That said – we would like to facilitate downloads of priority data. How? …. (new slide)

Downloads are managed by Us GridFTP Fast Downloads managed by you as a user

Controlled Vocab: documentation/cmip5_controlled_vocab.txt