Presentation is loading. Please wait.

Presentation is loading. Please wait.

PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Cindy Zheng For PRAGMA Grid/Cloud Team And the PRAGMA communities Pacific Rim Application and Grid Middleware.

Similar presentations


Presentation on theme: "PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Cindy Zheng For PRAGMA Grid/Cloud Team And the PRAGMA communities Pacific Rim Application and Grid Middleware."— Presentation transcript:

1 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Cindy Zheng For PRAGMA Grid/Cloud Team And the PRAGMA communities Pacific Rim Application and Grid Middleware Assembly http://www.pragma-grid.net http://goc.pragma-grid.net PRIME 2011 Tutorial PRAGMA Grid/Cloud Access

2 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Grid and Cloud What are they? –Collection of resources for sharing Differences –Grid Resources glued by agreed and verifiable trust –Certification –GLOBUS Provide system/software environment for some applications and users –Cloud Users setup applications on systems Cloud provide mechanisms to allow users run their systems as virtual machine (VM) in shared resources

3 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Grid Trust Organizations and policies –International Grid Trust Federation (IGTF) APGridPMA EUGridPMA TAGPMA –Accreditation Software –Certificate Authorities (CA) Set up CA services, policies Issue/manage certificates (user, server, services, …) –GLOBUS Identify users and resources in a cohesive fashion Map user certificate to a local account Interface local job managers

4 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 PRAGMA CAs PRAGMA-UCSD CA (https://goc.pragma-grid.net/ca)https://goc.pragma-grid.net/ca –Accredited by APGrid PMA –Included in IGTF distribution –Only issue a certificate when needed PRAGMA Experimental CA (http://rocks-56.sdsc.edu/exp-ca)http://rocks-56.sdsc.edu/exp-ca –Only accepted by PRAGMA grid site PRIME 2011 students will use PRAGMA experimental CA unless your project involved with other grids

5 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Grids/Clouds and PRAGMA Grid/Cloud Grids/Clouds –Most are centrally funded PRAGMA grid/Cloud is unique –Grass-root Voluntary contribution Open (PRAGMA member or not, pacific rim or not) Long-term collaborative working experiment –Heterogeneous Funding No uniform infrastructure management Open to all sciences and applications All varieties of site policies, system and network environments –Friendly, helpful, close relationships –More challenging and rewarding Good for development, collaborations, integrations and testing

6 PRAGMA Grid/Cloud 26 institutions in 16 countries/regions, 22 compute sites, 9VM sites (+ 6 site in preparation) UZH Switzerland NECTEC KU Thailand UoHyd India MIMOS USM Malaysia HKU HongKong ASGC NCHC Taiwan HCMUT HUT IOIT-Hanoi IOIT-HCM Vietnam AIST OsakaU UTsukuba Japan IHPC/NGO NTU Singapore MU Australia KISTI KMU Korea JLU China SDSC USA CICESE UNAM Mexico UChile Chile CeNAT-ITCR Costa Rica BESTGrid New Zealand CNIC China LZU China UZH Switzerland LZU China UValle Columbia ASTI Philippines IndianaU USA

7 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 PRAGMA Grid Software Layers http://goc.pragma-grid.net/pragma-doc/userguide/join.html http://goc.pragma-grid.net/pragma-doc/userguide/join.html Local job scheduler (require one) Globus (required) Application Middleware Applications Infrastructure Middleware SGEPBSLSFSQMS… Ninf-GNimrod/GMpich-GX…GfarmSCMSWebMOGASCSF Phylogenetic…FMOCSTFTSavannahMM5AMBERSiesta …

8 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 PRAGMA Grid/Cloud Compute Resources http://goc.pragma-grid.net/pragma-doc/computegrid.html

9 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 PRAGMA Grid/Cloud Members and Team http://goc.pragma-grid.net/wiki/index.php/Site_status_and_tasks http://goc.pragma-grid.net/wiki/index.php/Site_status_and_tasks Team members –>240 and growing –one management contact / site –1~3 technical support contact / site –1~4 application drivers / application –1~5 members / Middleware development team Experts –Application (various scientific domains) –Middleware (various grid middleware) –Security (grid security) –… Communications –Email –Mailing list – pragma-grid-team@googlegroups.com –Coordinator – zhengc@sdsc.edu

10 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 PRAGMA Cloud Access Cloud computing is new –Access procedure has not been formalized Contact Cindy for detail info and arrangements The rest of the slides shows how to access PRAGMA grid

11 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Obtain PRAGMA Grid Access Find guide document at http://goc.pragma- grid.net/wiki/images/5/51/PRIME-accounts.doc http://goc.pragma- grid.net/wiki/images/5/51/PRIME-accounts.doc Be aware –When cut/paste a command line Microsoft word may change –A hyphen to a line character –Double-hyphen to a long line character Learn UNIX concepts and commands  You need for your project  Know how  Understanding (Know why)  Account application as an example  Learn more online, google http://mally.stanford.edu/~sr/computing/basic-unix.html  Try them, hands-on  But be careful with “rm” (remove) command!

12 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Same Basic UNIX Commands List files $ ls $ ls -l (show file permission, ownership, etc.) $ ls -a (show all files, including hidden files) Change directory $ cd (go to my home directory) $ cd ~/.ssh (go to.ssh subdirectory under my home directory) Copy files locally $ cp Copy files between local and remote systems use SSH key $ scp : Remember the command, but forgot how to use it $ --help $ man

13 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

14 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Need an UNIX System on Laptop (Step 1.1) UNIX on laptop –MAC OS 10  HD  Applications  Utilities  Terminal –Windows  Install Cygwin  Click –LINUX  Login

15 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

16 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Create SSH Key (Step 1.2) Secure remote login Generate key files $ ssh-keygen -t rsa  Use default path for ssh key files  Good passphrase (hard to guess, no dictionary words)  Remember the passphrase View SSH key files (private vs. public)  From Cygwin $ ls ~/.ssh -r--------@ 1 cindyzheng staff 951 Mar 19 01:10 id_rsa -rw-r--r--@ 1 cindyzheng staff 234 Mar 19 01:10 id_rsa.pub -rw-r--r--@ 1 cindyzheng staff 8414 Mar 19 01:10 known_hosts  From Windows c:\Program files\Cygwin\Home\ \.ssh Backup SSH key files USB drive, take with you Can be used thereafter Only on secure systems Possibly compromised key Immediately inform all public key holders (system administrators)

17 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

18 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Account Application (Step 2) Email to Cindy –Your full name –Institution name: UCSD –Address of the institution: UCSD –Country of the institution: USA –Your phone number –Your fax number –Your email address –The purpose of using PRAGMA grid: PRIME –Your/your institution's contribution to PRAGMA (can be work, resources): work –Prefered login names (1st, 2nd, 3rd, but may or may not be granted) –Prefered UNIX shell (may or may not be granted) –Hostname(s) and IP address(es) of the system(s) which you launch your applications (This is required for PRAGMA grid site systems to open their firewalls in order to allow you access.) –Your ssh PUBLIC key (NEVER and NEVER give anyone your private key file!! Nor your ssh password!!)

19 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

20 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Cindy Create user info pack Create an UNIX user account on rocks-200.sdsc.edu Email user –Account name on rocks-200.sdsc.edu –Certificate request license id

21 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

22 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Test SSH key (Step 3) Login to rocks-200.sdsc.edu $ ssh @rocks-200.sdsc.edu  Enter passphrase Last login: Tue Apr 5 11:37:56 2011 from dyn137-110-115- 232.ucsd.edu Rocks 5.3 (Rolled Tacos) Profile built 04:25 15-Apr-2010 Kickstarted 22:21 14-Apr-2010 In case ssh fails $ ssh –v @rocks-200.sdsc.edu Email output to Cindy  First time login on rocks system, ssh-keygen ran automatically  Accept default ssh file path  Empty password Only used internally not recommended elsewhere

23 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Make a User Certificate Request (Step 3) $ cd $ /opt/pragma-ca/bin/grid-certreq -sv ra.pragma-grid.net:pragma-exp_ra -new -g "SMIME user" -sou ---------(Sample output)------------- creating a certificate signing request ------------------------------------------- generate private key (size 1024 bit)...................................................oo....................oo ------- input user subject information -------- email can be omitted by putting a char of '.’ input ou : PRIME  type exactly as this input user name : Cindy Zheng  input your full name input user email : cindy@sdsc.edu  input your email address, must! ------- please confirm your inputs ------------ GROUP : SMIME user SUBJECT : CN=Cindy Zheng, Email=cindy@sdsc.edu do you continue operation? (yes/no/retry)[y]:  press return trying to connect RA server : ra.pragma-grid.net (11412)... ok. request for issuing a new certificate... ok. your request is accepted. (AcceptID=0000003) <- remember it CA operator will send an email to tell a result. save a CA certificate file : /home/cindy/.globus/cacert.pem <- note cert request files location save a private key file : /home/cindy/.globus/userkey.pem <- note cert request files location Input PASS Phrase:  type a good password Verifying - Input PASS Phrase:  retype the password (remember it!) Email Cindy the acceptID

24 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

25 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Cindy Issue user certificate –Off-line CA server Inform user via email

26 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

27 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Retrieve User Certificate (Step 4) Login to rocks-200.sdsc.edu $ ssh @rocks-200.sdsc.edu  When prompt, enter your SSH password Retrieve user certificate $ /opt/pragma-ca/bin/grid-certreq -sv ra.pragma- grid.net:pragma-exp_ra -em -recv trying to connect RA server : ra.pragma-grid.net (11412) request for exporting a certificate... ok save a CA certificate file : /home/cindy/.globus/cacert.pem save a certificate file : /home/cindy/.globus/usercert.pem View and understand certificate files $ ls -l.globus -rw-r--r-- 1 cindyzheng staff 2878 Mar 23 18:26 cacert.pem -rw-r--r-- 1 cindyzheng staff 4998 Mar 23 18:26 usercert.pem -r-------- 1 cindyzheng staff 1743 Mar 23 18:26 userkey.pem

28 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Backup Certificate Files (Step 4) Copy.globus directory from rocks-200.sdsc.edu to your laptop  Start Cygwin/terminal on your laptop (Should land in your home directory)  $ scp -r @rocks- 200.sdsc.edu:/export/home/ /.globus. Copy c:\Program files\Cygwin\Home\ \.globus to your USB backup disk Take with you anywhere Can be used usually 1 year Only on secure systems Possibly compromised certificate Immediately inform certificate issuer (Cindy)

29 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

30 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Test Globus Authentication (Step 5) On rocks-200.sdsc.edu: Create a proxy $ grid-proxy-init Your identity: /O=grid/O=pragma/OU=PRIME/CN=Cindy Zheng Enter GRID pass phrase for this identity: <- input passphrase Creating proxy.......................................................... Done Your proxy is valid until: Thu Apr 7 02:18:11 2011 Test globus authentication $ globusrun -a -r rocks-200.sdsc.edu GRAM Authentication test successful Email Cindy if successful, otherwise email output to Cindy To create longer proxy, specify hours $ grid-proxy-init –hours 100

31 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Test Job Submission (Step 5) $ globus-job-run rocks-200.sdsc.edu/jobmanager-sge /bin/date /opt/gridengine/default/spool/rocks-65/active_jobs/330008.1/pe_hostfile compute-0-12 Tue Apr 21 13:06:18 PDT 2009 rm: cannot remove `/tmp/330008.1.all.q/rsh': No such file or directory  ignore this for now Note: depending on how busy the system is, you may need to wait a bit. To find out the status of SGE queue, open/login to another shell and run $ qstat –f queuename qtype used/tot. load_avg arch states ---------------------------------------------------------------------------- … ---------------------------------------------------------------------------- all.q@compute-0-6.local BIP 1/2 0.06 lx26-x86 328085 0.56000 data jas dr 04/15/2009 23:29:01 1 ---------------------------------------------------------------------------- … ##################################################################### - PENDING JOBS - PENDING JOBS - PENDING JOBS - PENDING JOBS - ##################################################################### 330010 0.00000 data zhengc qw 04/21/2009 13:18:39 1 Email Cindy if successful, otherwise email output to Cindy

32 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

33 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Cindy Add your email to pragma-grid-team list Email pragma-grid- team@googlegroups.com ask site adminitrators to create account on their systemspragma-grid- team@googlegroups.com Site administrator will email your username to you and me after your account is created

34 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

35 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Test Access To Your Host Site (Step 6) Host system name/ip (where you will be launching your applications from) Do the same tests as you have done with rocks-200.sdsc.edu –Test SSH (Slide 20) –Test Globus authentication (Slide 29) –Test job submission (Slide 30) More testing tips o http://goc.pragma- grid.net/wiki/index.php/User_Testing http://goc.pragma- grid.net/wiki/index.php/User_Testing o Google

36 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Test Access to Each Site (Step 6) Test SSH –Find the target system name at http://goc.pragma-grid.net/wiki/index.php/Site_status_and_tasks –Test SSH from your host system to the target host (should work) –Test SSH from your laptop to the remote host (May work) –Issues about firewall and site policies Test Globus authentication –From the system at your host site –To the target system Test job submission –From the system at your host site –To the target system Problem with any test –Save output –Check tips at http://goc.pragma-grid.net/wiki/index.php/User_Testing Google –Email site contact and cc zhengc@sdsc.edu

37 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Process At A Glance Laptop platform windowswindows MAC Step 1.1 Install Cygwin Step 1.2 Create a SSH key Step 2 Email application Create account on rocks-200 PRIME StudentCindy start Step 3 Test SSH, Request certificate Issue certificate Step 4 Retrieve certificate Step 5 Test Globus authentication and job submission confirmation Step 6 Test SSH, Globus and job submission to each site sites reply Request to all sites

38 PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Use PRAGMA Grid Information online goc.pragma-grid.net  member sites (hostname, contacts, user guide, …)  computational grid (resource table, jobmanager, cpu architecture, …)  Applications (as examples)  SCMSWeb (realtime system status, software catalog, etc.) wiki.pragma-grid.net  For Users Ask questions (email, skype)  Email, skype  Site contact – always cc Cindy  Looking for experts – ask Cindy  If Cindy is not around, pragma-grid-team@googlegroups.com


Download ppt "PRIME Tutorial, 4/22/2009 Prime Tutorial 4/22/2011 Cindy Zheng For PRAGMA Grid/Cloud Team And the PRAGMA communities Pacific Rim Application and Grid Middleware."

Similar presentations


Ads by Google