1. ECMWF 1 Com Intro training course – 2004 ECaccess ECaccess A portal to access ECMWF Also known as EasyAccess Dominique Lucas

2. ECMWF 2 Com Intro training course – 2004 ECaccess Content ECaccess concepts telnet, ssh and ftp access X11 access ECaccess file management ECaccess job management Unattended file transfers ECaccess vs. ecbatch Tutorial

3. ECMWF 3 Com Intro training course – 2004 ECaccess ECaccess - Concepts ECaccess provides a portal to access ECMWF archiving and computing facilities Strict authentication via SecurID card and X509 certificates Data integrity/confidentiality guaranteed by SSL

4. ECMWF 4 Com Intro training course – 2004 ECaccess ECaccess - Concepts ECaccess provides: File and job management in batch or interactive mode through an extended FTP server File and job management through a Web browser A secure telnet access to ECMWF A secure X Windows access to ECMWF All this via internet or RMDCN.

5. ECMWF 5 Com Intro training course – 2004 ECaccess ECaccess - Architecture I N R T M E D R C N E T ECaccess MS Gateway Ecgate ECcert Command ECaccess Server (Java app) ECaccess Gateway F I R E W A L L ECproxy Server (C app) ECcmd Server (Java app) ECios Server (C app) JSSE-SSL OpenSSL-SSL ECMWF Gate- way FTP Server X ECaccess Tools FTP Client Telnet Client Web Browser MS Work- station ECaccess CA (OpenSSL) Multi tiers ECaccess Client, ECaccess Gateway, ECaccess Server, ECaccess ecgate

6. ECMWF 6 Com Intro training course – 2004 ECaccess ECaccess client Standard Telnet, FTP, ssh and HTTP/S client Standard X Windows and FTP Server ECaccess tools (eccert …) ECaccess - Architecture

7. ECMWF 7 Com Intro training course – 2004 ECaccess ECaccess gateway Entry point for all ECaccess users Verifies ECaccess users authentication (certificates or passcodes) Implements Telnet, FTP and HTTP/S protocols Secure tunnels through firewalls to ECaccess Server Either local gateway installed at your site or ecaccess.ecmwf.int for internet and msaccess.ecmwf.int for RMDCN. ECaccess - Architecture

8. ECMWF 8 Com Intro training course – 2004 ECaccess ECaccess Server Located at ECMWF: ecaccess.ecmwf.int or msaccess.ecmwf.int Entry point for all ECaccess gateways Authenticate ECaccess gateways Provides job and file management functions Provides functions to monitor file transfers Provides system access functions Keeps track of users activity ECaccess – Architecture

9. ECMWF 9 Com Intro training course – 2004 ECaccess ECaccess ecgate Runs the ECaccess Certificate Authority (CA) Provides job management through NQS and LoadLeveler Provides access to home, scratch and ECFS Keeps job input, output and error files Manages spool for unattended file transfers ECaccess - Architecture

10. ECMWF 10 Com Intro training course – 2004 ECaccess Use local ECaccess gateways if installed Internet ECaccess gateways available at: ctbto.de, dmi.dk, knmi.nl, sma.ch, irmet.ie, smhi.se, meteo.fr, dwd.de, cerfacs.fr, mercator.ocean.fr, eumetsat.de, univie.ac.at, met.hu, cscs.ch, inm.es RMDCN ECaccess Gateways available at: metoffice.com, inm.es, meteor.gov.tr If local ECaccess gateway not installed, then use ecaccess.ecmwf.int or msaccess.ecmwf.int ECaccess – local gateways

11. ECMWF 11 Com Intro training course – 2004 ECaccess ECaccess – telnet/ssh {mshost}$ telnet ecaccess.ecmwf.int … login: xyz Passcode: XXXXXX Which system: ecgate … ecgate{~xyz}:1 {mshost}$ ssh xyz@ecaccess.ecmwf.intxyz@ecaccess.ecmwf.int … login: xyz Passcode: XXXXXX Which system: ecgate … ecgate1{~xyz}:1 Where xyz is your ECMWF User ID On local gateways, port 9023 (telnet) or 9022 (ssh) may be needed.

12. ECMWF 12 Com Intro training course – 2004 ECaccess ECaccess - ftp {mshost}$ ftp ecaccess.ecmwf.int … login: xyz Passcode: XXXXXX … ftp> ls 227 Entering Passive Mode. (193,61,196,110,135,230) 150 Opening ASCII mode data connection drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECJOBS drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECSCRATCH drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECTMP drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECFS drwxr-x--- 1 xyz ecaccess 2048 Nov 05 11:00 ECHOME 226 Transfer complete ftp> sftp not yet available. On local gateways, port 9021 may be needed.

13. ECMWF 13 Com Intro training course – 2004 ECaccess ECaccess - ftp Local environment ftp entry for batch access Local file can be dragged and dropped into ecaccess area.

14. ECMWF 14 Com Intro training course – 2004 ECaccess ECaccess – X11 via telnet {mshost}$ echo $DISPLAY hostname:0.0 {mshost}$ xhost +ecaccess.ecmwf.int ecaccess being added to access control list {mshost}$ telnet ecaccess.ecmwf.int … (Passcode validation required) Which host …: ecgate Which proxy … : X … ecgate{~xyz}:1 echo $DISPLAY (Something like … )ecaccess.ecmwf.int: ecgate{~xyz}:2 xcdp& A control window showing the DISPLAY to be used will appear. This window or an X11 application should remain present to keep the X11 proxy.

15. ECMWF 15 Com Intro training course – 2004 ECaccess ECaccess – ecxterm via ssh A control window will also appear. Keep it while X11 access required. {mshost}$ echo $DISPLAY hostname:0.0 {mshost}$ ssh –X xyz@ecaccess.ecmwf.int (N.B. Port number IS mandatory) … (Passcode validation required) Which host …: ecgate ecgate{~xyz}:1 echo $DISPLAY (Something like)ecaccess.ecmwf.int: ecgate{~xyz}:2 xcdp&

16. ECMWF 16 Com Intro training course – 2004 ECaccess ECaccess – ecxterm

17. ECMWF 17 Com Intro training course – 2004 ECaccess ECaccess – web – http://ecaccess.ecmwf.int/

18. ECMWF 18 Com Intro training course – 2004 ECaccess ECaccess tools - ECtools Must be installed at your end on each platform for which access to ECMWF is required. If you (still) had/have access to old ecbatch commands be careful about the $PATH as some commands are ambiguous between the two packages e.g. eccert, ecget, ecput. Help for each command is available with -help option: {mshost}$ eccert –help ECtools are also available at ecmwf.

19. ECMWF 19 Com Intro training course – 2004 ECaccess ECaccess - eccert Command {mshost}$ eccert –verbose echost: ecaccess.ecmwf.int ecport: 443 eccert: /home/xyz/.eccert.crt Certificate request ECMWF user identifier: xyz Passcode from your SecurID card: Certificate saved (855 bytes) {mshost}$ Certificate valid for 7 days. Certificate only needed for file, job and transfer management, NOT required for unattended transfers (see later - ectrans). Certificate not needed for ECMWF local use of ECtools.

20. ECMWF 20 Com Intro training course – 2004 ECaccess ECaccess environment

21. ECMWF 21 Com Intro training course – 2004 ECaccess ECaccess File Management No support for meta-characters.

22. ECMWF 22 Com Intro training course – 2004 ECaccess ECaccess file management {mshost}$ ecls script.sh ecaccess-tools.tar.gz ecaccess.doc ecaccess {mshost}$ ecdir ecaccess 10838 drwxr-xr-- 4 xyz systems 96 Mar 14 09:30. 3194 drwxr-xr-x 47 xyz systems 4096 Mar 14 09:30.. 12721 drwxr-x--- 4 xyz systems 96 Mar 13 18:55 client 124513 drwxr-x--- 11 xyz systems 2048 Mar 5 11:38 gateway {mshost}$ ecget ecaccess-tools.tar.gz {mshost}$ ecget ecaccess-tools.tar.gz tools.tar.gz {mshost}$ ls *.tar.gz ecaccess-tools.tar.gz tools.tar.gz {mshost}$

23. ECMWF 23 Com Intro training course – 2004 ECaccess ECaccess file management {mshost}$ ecdelete ecaccess-tools.tar.gz DELE command successful {mshost}$ ecput ecaccess-tools.tar.gz {mshost}$ ecmkdir ectest MKD command successful {mshost}$ ecrmdir ectest RMD command successful {mshost}$ ecrmdir ecaccess Directory not empty {mshost}$

24. ECMWF 24 Com Intro training course – 2004 ECaccess ECaccess file management {mshost}$ export ECDOMAIN=ecfs {mshost}$ ecdir 10838 drwxr-xr-- 4 xyz systems 96 Mar 14 09:30. 3194 drwxr-xr-x 47 xyz systems 4096 Mar 14 09:30.. 12721 drwxr-x--- 4 xyz systems 96 Mar 13 18:55 backup 124513 drwxr-x--- 11 xyz systems 2048 Mar 5 11:38 doc {mshost}$ export ECDOMAIN=ecfs[zzz] … {mshost}$ export ECDOMAIN=echome … {mshost}$

25. ECMWF 25 Com Intro training course – 2004 ECaccess ECaccess Job and transfer management

26. ECMWF 26 Com Intro training course – 2004 ECaccess ECaccess Job Management {mshost}$ ecqls ecgate1 NQS submission on ecgate1(INIT=23...) hpca LoadLevelersubmission on hpca(INIT=21...)... {mshost}$ {mshost}$ ecqls hpca diag Diagnostic jobs only temp Temporary small jobs < 5 Nodes inter_class Default interactive class debug Default interactive class os Operational serial/single task work ns Serial/single task work bench Benchmark class np Parallel work op Operational parallel work {mshost}$

27. ECMWF 27 Com Intro training course – 2004 ECaccess ECaccess Job Management {mshost}$ ecjreq -help Syntax: JREQ ECaccess-queue remote-script [args...] -at - start date (yyyy-MM-dd HH:mm) -nd - no directives within the input script -tg - specify the target gateway name -tr - specifiy the access method (msuser[@destination]) -to - transfer output file when the request ends -te - transfer error file when the request ends -ti - transfer input file when the request ends -tk - keep in spool (default: deleted if transfer successful) -mu - send mail for the request to the stated address -mb - send mail when the execution/transfer begins -me - send mail when the execution/transfer ends -mf - send mail when the execution/transfer fails

28. ECMWF 28 Com Intro training course – 2004 ECaccess ECaccess Job Management {mshost}$ ecjreq ecgate script.sh –me 34850 {mshost}$ ecjreq ecgate test.sh Error opening file {mshost}$ {mshost}$ ecjput ecgate1 test.sh –me 34851 {mshost}$

29. ECMWF 29 Com Intro training course – 2004 ECaccess ECaccess Job Management {mshost}$ ecjls 4421 ecgate WAIT Nov 24 21:33 3884 hpca@hpca.ecmwf.int DONE Nov 18 11:42 3146 ecgate1 DONE Nov 15 21:33 {mshost}$ ecjls 3884 Jobid: 3884 Location: hpca@hpca.ecmwf.int Date/Time: Nov 18 11:42 Status: DONE stdout size: 221 stderr size: 219 stdin size: 241Jobid: 4421 {mshost}$

30. ECMWF 30 Com Intro training course – 2004 ECaccess ECaccess Job Management {mshost}$ ecjget o34852 {mshost}$ ls JOB* JOB-o34852 {mshost}$ ecjget o34852 job.out {mshost}$ ls –ail job.out 71196 -rw-r----- 1 xyz group 686 Mar 13 19:10 job.out {mshost}$ {mshost}$ ecjdel 34852 JDEL command successful {mshost}$ ecjdel kills job running and removes all files.

31. ECMWF 31 Com Intro training course – 2004 ECaccess ECtrans Command ecgate, or hpca: ECAccess Member State Gateway: Unattended file transfers, issued from ECMWF It is preferable if the ECaccess gateway is installed at your end. RMDCN

32. ECMWF 32 Com Intro training course – 2004 ECaccess ECtrans - msuser maintenance

33. ECMWF 33 Com Intro training course – 2004 ECaccess ECtrans - msuser maintenance

34. ECMWF 34 Com Intro training course – 2004 ECaccess ECtrans - msuser maintenance

35. ECMWF 35 Com Intro training course – 2004 ECaccess ECtrans - msuser maintenance

36. ECMWF 36 Com Intro training course – 2004 ECaccess ECtrans - msuser maintenance MS associations are specific to each ECaccess gateway.

37. ECMWF 37 Com Intro training course – 2004 ECaccess ECtrans Command – at ECMWF {ec-host}$ ectrans –help usage: ectrans [-get] -gateway name \ –remote msuser@destination \ -source name [args...] -gateway str - target gateway name -source str - source file name -echost str - eccmd host name (default: localhost) -ecport num - eccmd port number (default: 644) -remote str - target user (default: same as ecuser) -target str - target file name (default: same a source) -verbose - verbose mode on -help - this message -get - transfer file from your site to ECMWF -status - check the status on file transfer Default values can also be set by the ECHOST, ECPORT, ECUSER or GATEWAY shell variables. {ec-host}$

38. ECMWF 38 Com Intro training course – 2004 ECaccess ECtrans Command {ecgate1}$ ectrans –gateway ecaccess.ecmwf.int \ -remote from_usl@genericFtp –source./data.grib \ –target data.grib \ -verbose echost: ecaccess.ecmwf.int ecport: 644 ecuser: usl source:./data.grib target: data.grib keep : false option: reject File to upload (585 bytes) 10129250216245 {ecgate1}$ Users can maintain MS users associations on local gateway, via the web interface. ectrans is available in both directions.

39. ECMWF 39 Com Intro training course – 2004 ECaccess ECaccess transfer Management {mshost/echost}$ ectreq –help Syntax: TREQ source [args...] -remote str - target user (default: current) -gateway str - target gateway name (default: current) -target str - target file name (default: same as source) -keep - keep the request in the spool -reject - if the target file already exists (default) -append - if the target file already exists -resume - if the target file already exists -erase - if the target file already exists {mshost}$ {mshost/echost}$ ectreq job.out 10161191420729 {mshost}$

40. ECMWF 40 Com Intro training course – 2004 ECaccess ECaccess transfer Management {mshost/echost}$ ectret 10161191420729 TRET command successful {mshost}$ $SCRATCH used for temporary file storage.

41. ECMWF 41 Com Intro training course – 2004 ECaccess ECaccess transfer Management {mshost/echost}$ ectls 10161191420729 STOP xyz@ecaccess.ecmwf.int Mar 14 15:19xyz@ecaccess.ecmwf.int 10164593445344 DONE xyz@ecaccess.ecmwf.int Mar 14 15:22xyz@ecaccess.ecmwf.int 10161193432443 DONE xyz@ecaccess.ecmwf.int Mar 14 15:19xyz@ecaccess.ecmwf.int {mshost}$ ectls 10161191420729 Copyid: 10161191420729 MS user: xyz Hostname: ecaccess.ecmwf.int Access: ECaccess gateway Status: STOP Error message: No target location for xyz Date/Time: Mar 14 15:19 Source:./job.out Target:./job.out {mshost/echost}$

42. ECMWF 42 Com Intro training course – 2004 ECaccess ECaccess vs. ECbatch FunctionECBATCH softwareECaccess software General InformationNot implemented.MS-host:> echelp … MS-host:> ecinfo MS-host:> ectinfo … Certificate InformationMS-host:> eccert –dMS-host:> eccert –expire MS-host:> eccert –ecuser Certificate generationMS-host:> eccert [-v]MS-host:> eccert Batch mode Job Submission from MS host to ECMWF (script local) Only supported for Ecgate1, never for HPCF. MS-host:> ecqsub … Supported for Ecgate1 and for HPCF MS-host:> ecjput … Batch mode Job Submission from MS host to ECMWF (script at ECMWF) Not implemented.MS-host:> ecjreq …

43. ECMWF 43 Com Intro training course – 2004 ECaccess ECaccess vs. ECbatch FunctionECBATCH softwareECaccess software Return Jobs Stdout/Stderr To MS-host; requires Eccopyd on MS host To be retrieved from ECMWF via ecget or ectrans (or via Web) Push File on ECMWF host to MS host Requires Eccopyd on MS host + MS host:home/.echosts. Ecgate1:> eccopy … Requires MS gateway + URL for user (to be set up by administrator) Ecgate1:> ectrans … File Size LimitationsEccopy limits the file size to 28 MB.File system limits the file size Push File on MS host to ECMWF MS-host:> ecput … Pull File on MS host from ECMWF MS-host:> ecget … (No support for ECFS). MS-host:> ecget …

44. ECMWF 44 Com Intro training course – 2004 ECaccess ECaccess vs. ECbatch FunctionECBATCH softwareECaccess software Pull File on MS host from ECMWF MS-host:> ecget … Pull File on MS host from ECMWF – restart after failure (No support for ECFS).MS-host:> ecreget … ECMWF File Access and Management on MS host Not implemented. ecls, ecdir, ecdelete, ecmkdir, ecrmdir. Web File Access and Management ecls.Files can be listed, viewed, stored, retrieved or deleted.

45. ECMWF 45 Com Intro training course – 2004 ECaccess ECaccess vs. ECbatch FunctionECBATCH softwareECaccess software Web Job Submission and Management Not implemented. Job Submission, termination; status check; view, store, edit, retrieve or delete output. Telnet to Ecgate1Not implemented. Provide SecurID code only once. X11 to Ecgate1Not implemented. Provide SecurID code only once. ServiceDiscontinued on 31 Dec. 2003 Available since early 2002

46. ECMWF 46 Com Intro training course – 2004 ECaccess ECaccess pointers http://www.ecmwf.int/services/ecaccess Users manual Administrators manual Registration center (for local ECaccess gateway) Quick start – check first if local gateway available http://ecaccess.ecmwf.int/ telnet ecaccess.ecmwf.int ssh [-X] xyz@ecaccess.ecmwf.int ftp ecaccess.ecmwf.int ectrans