CCPR Workshop Lexis Cluster Introduction October 19, 2007 David Ash

Outline What is a cluster? ….two components What is a cluster? ….two components

Outline Continued… How can CCPR’s cluster help you? How can CCPR’s cluster help you? Lexis usage Lexis usage Connecting from Windows Connecting from Windows Unix commands Unix commands Submitting jobs Submitting jobs Managing jobs Managing jobs

Lexis Management System Jobs are submitted via cluster management system Jobs are submitted via cluster management system An advanced batch system An advanced batch system Efficiently allocates resources to jobs Efficiently allocates resources to jobs Remembers node specs and availability Remembers node specs and availability Schedules jobs according to requested resources and priority Schedules jobs according to requested resources and priority Users can easily view jobs Users can easily view jobs

How does this help your research? Remote access to software and data from anywhere! Remote access to software and data from anywhere! Incredible machines – faster processors and more memory Incredible machines – faster processors and more memory Flexible system to share data and programs with colleagues Flexible system to share data and programs with colleagues Opens up your home/office PC for other things, instead of grinding away on a Stata job Opens up your home/office PC for other things, instead of grinding away on a Stata job Advanced users: parallelization, fast compilers Advanced users: parallelization, fast compilers

Connecting to Lexis Secure Shell Software Secure Shell Software SSH (command line access) SSH (command line access) SFTP (file transferring access) SFTP (file transferring access) Obtain from: Obtain from: Alternative Client Software Alternative Client Software PuTTY (SSH) PuTTY (SSH) winSCP (SFTP) winSCP (SFTP)

Connecting to Lexis: Secure Shell Directions for SSH Secure Shell: Directions for SSH Secure Shell: Click Quick Connect Click Quick Connect Enter: Enter: Host Name: lexis.ccpr.ucla.edu Host Name: lexis.ccpr.ucla.edu User Name: your lexis account name User Name: your lexis account name Port Number: 22 Port Number: 22 Click connect and enter password Click connect and enter password Can “Add Profile” for repeat usage Can “Add Profile” for repeat usage Same for SSH Secure File Transfer Same for SSH Secure File Transfer

Mapping Lexis as a Drive using Samba Creating Lexis as a drive in Windows Explorer Creating Lexis as a drive in Windows Explorer Must use the VPN to do this action off campus! Must use the VPN to do this action off campus! Instructions I’m going to show are at: Instructions I’m going to show are at: Step-by-step instructions: Step-by-step instructions:

Mapping Lexis Open Windows Explorer Open Windows Explorer Select Tools -->Map Network Drive Select Tools -->Map Network Drive At "Folder" text box enter: At "Folder" text box enter:\\lexis.ccpr.ucla.edu\your-user-name Click "Reconnect at login" if you want the drive to be connected each time the computer starts Click "Reconnect at login" if you want the drive to be connected each time the computer starts Click the "different user name" link to enter your user name and password for the connection. Click the "different user name" link to enter your user name and password for the connection. Click "OK" then click the "Finish" button on the Map Network Drive window. Click "OK" then click the "Finish" button on the Map Network Drive window.

Linux Commands passwd Change your Lexis SSH/SFTP password smbpasswd Change your Samba password pwd Print your current directory ls List files (can take arguments) ls –lah L = long, A = all, H = human readable cd AND cd.. Change directory AND Go up one directory mkdir foldername Create a directory cp file1 file2 Copy file1 to file2 mv file1 file2 Rename/Move file1 to file2 more file OR cat file Display file to screen du –h Disk Usage (-h = human readable) tar –cvf AND tar -xvf Compress a file/directory / Expand.tar gzip AND gunzip Compress a file (.gz extension) / Expand.gz ln –s actual link Creates a symbolic link/shortcut

Getting Help in Linux man command list help for command man –k command keyword search for command whatis command brief description of command which name search for name apropos keyword list commands with keyword in their description Bryan and myself.

Linux File Editors Pico – easiest editor. Open and type. Commands listed at bottom of editor. Pico – easiest editor. Open and type. Commands listed at bottom of editor. Vi – more advanced. You can move around quickly, do copy/paste, search, etc, but keystrokes required for everything. Vi – more advanced. You can move around quickly, do copy/paste, search, etc, but keystrokes required for everything. Emacs – most advanced Emacs – most advanced OR edit files over Samba from Windows. OR edit files over Samba from Windows.

Submitting Stata and SAS jobs Submit jobs to the Cluster using ONE of the following commands: qjob, qstata, qsas, qmatlab Submit jobs to the Cluster using ONE of the following commands: qjob, qstata, qsas, qmatlab Will be prompted for: Will be prompted for: Job type (for qjob only!) Job type (for qjob only!) Filename Filename Amount of Memory (number in megs, ex. 1050) Amount of Memory (number in megs, ex. 1050) If memory > 1 gig --> Over 2 hours (y/n) If memory > 1 gig --> Over 2 hours (y/n) Confirmation to submit (y/n) Confirmation to submit (y/n) Quick Method: Quick Method: qstata sample.do 500 y qstata sample.do 500 y

Qstata example output Two Logs are created Two Logs are created filename.log - This is the log file from Stata. The output of your entire Stata job is captured in this file. filename.log - This is the log file from Stata. The output of your entire Stata job is captured in this file. filename.do.joblog - This file will capture the output of your cluster command file and any errors that may occur (Stata output is NOT captured by this log.) filename.do.joblog - This file will capture the output of your cluster command file and any errors that may occur (Stata output is NOT captured by this log.) filename.do.cmd - This is the command file that is submitted to the cluster. filename.do.cmd - This is the command file that is submitted to the cluster. Quick TIP: qsub filename.do.cmd Quick TIP: qsub filename.do.cmd

Cluster job monitoring qstat or qstatus to view overall cluster usage qstat or qstatus to view overall cluster usage qstat –j jobnumber for information particular to your job qstat –j jobnumber for information particular to your job qdel jobnumber to kill your job qdel jobnumber to kill your job

Interactive Sessions Interactive Usage Interactive Usage Meant for testing before submitting to cluster Meant for testing before submitting to cluster Limited spots Limited spots Logoff promptly when finished Logoff promptly when finished Non-GUI Interactive Session Non-GUI Interactive Session Type “qinteractive”, can also specify memory after command. Default 256mb. Type “qinteractive”, can also specify memory after command. Default 256mb. GUI Interactive Session GUI Interactive Session Login with XliveCD Login with XliveCD Type “qlive” – Run sas/matlab/xstata Type “qlive” – Run sas/matlab/xstata

Questions / Comments?