File Transfer Olivia Irving and Cameron Foss Blue Waters Student Internship Program 2017

Log into SHODOR Windows Users: Mac or Linux Users: Using putty, connect to <username>@login.shodor.org <ENTER> Mac or Linux Users: From the terminal, $ ssh <username>@login.shodor.org <ENTER>

Motivation Secure and reliable data transfer is essential to a trustworthy network File transfer is an inherent part of working on a computer. Data is not stagnant and is often pushed around. So it is important to be able to work with and manipulate this data in a way that makes it facile and organized .

Motivation Secure and reliable data transfer is essential to a trustworthy network File transfer is an inherent part of working on a computer. Data is not stagnant and is often pushed around. So it is important to be able to work with and manipulate this data in a way that makes it facile and organized .

Types of Nodes Gateway (bw and bwbay) An intermediate node between your system and the login nodes A ‘head’ node for login nodes Login (login.shodor.org, h2ologin.ncsa.Illinois.edu) Nodes where we can compile code, submit batch jobs, or enter an interactive job For h2ologin there are 12 (or less) login nodes of which there may be many users connected to at any given time. MOM An intermediate node between login nodes and compute nodes A ‘head’ node for compute nodes. In interactive sessions you are connected to a MOM node. Compute where jobs are executed https://bluewaters.ncsa.illinois.edu/education-training-allocation-data-transfer https://bluewaters.ncsa.illinois.edu/user-guide

Types of Nodes on Blue Waters MOM Node Gateway End User (You) Login Node Compute Node

File Transfer Methods Command line (for small files): Remote Desktop running Linux (i.e. University Computer) File Transfer Methods Command line (for small files): scp – secure copy protocol sftp – secure file transfer protocol rsync – remote synchronization wget – downloads from URL links User Interfaces (for large files): Globus (webpage utility, accessible on any OS) WinSCP (Windows only) End User (You) Cluster, database, server, etc.

scp – Secure Copy Copies files between hosts on a network. Requires authentication via passwords or passphrases Filenames can be specified with a user and host to transfer between two remote hosts Local filenames can be made explicitly without user and hostnames

scp – Secure Copy BW specific details: Cannot scp directly to BW because the gateway server is not connected to the filesystem, and cannot ssh (scp?) directly to a login (compute?) node Training accounts cannot use scp or sftp to "push" data to Blue Waters from a remote host Can scp from BW once on a login node. Pure copy and pasting $ scp local_file user@remote_host:/remote_file_dest/.

rsync – remote file copying tool fast and versatile Performs transfers: To/from a remote host To/from a remote rysnc daemon Can also move files locally Copying and synchronization Push: $ rsync [options…] file_source user@remote- host:/remote_file_dest/. Pull: $ rsync [options…] user@remote-host:/remote_file_source/ local_dest Daemon is a process that runs in the background and performs a specified operation at predefined times or in response to certain events.

sftp - Secure File Transfer Protocol An interactive file transfer program, similar to ftp Performs operations over an encrypted ssh transport Uses features of ssh, such as public key authentication and compression Connects and logs into a remote host and then enters an interactive mode.

sftp - Secure File Transfer Protocol You can also use the ‘ls’ command to list files on the remote system, a feature not present when using scp or rsync.

wget - the non-interactive network downloader free utility for downloading files from the web supports HTTP, HTTPS, and FTP HTTP(S) - Hyper Text Transfer Protocol (Secure) FTP - File Transfer Protocol non-interacting ⇒ can run in the background

For transferring Large Files Globus A web interface that is compatible across Operating Systems Also has a command line interface (CLI) The preferred file transfer method for Blue Waters does not support drag and drop WinSCP Windows SCP utility Supports drag and drop Possible to run on MAC or Linux via Wine (a compatibility layer for running windows applications on MAC or Linux) Other Cross-Platform Utilities: FileZilla, iMazing (formerly DiskAid), CyberDuck, and many more. Good old fashion leg work!! Physically relocating a large set of data may be faster than moving it over the network.

Globus Tutorial https://www.globus.org/tags/tutorial Many, many tutorials https://www.globus.org/events/oin-workshop/globus-tutorial As training accounts do not have RSA tokens, the accounts are not able to use the Blue Waters Globus Online public endpoints.

Globus Online First go to the appropriate tab on the Blue Waters website then go to globus online.

General Outline To use Globus Online, you must have an account with them. That account is separate from your accounts on the Blue Waters machines and from your account on the Blue Waters user portal, but you can "link" your GO account to your portal account so you can to go Globus Online using your Blue Waters token code and PIN.

Globus Online If you're just creating your Globus Online account, that creation goes as follows: 1.Log into to the Blue Waters portal as usual, with your username and PIN+token code. 2.Go to Globus Online by pulling down the data tab and clicking "Globus Online". 3. You'll authenticate again. This will probably ask for your username and "password". Type in your PIN+token code as before.

Globus Online 4. Now connect to your Globus Online Account, and you'll be given the option of creating a Globus Online account. Follow that procedure to create the new account. During this process, select the option to "link" your new Globus Online account to your Blue Waters portal account. 5. Once the Globus Online account is set up, when you click on the Globus Online from within the Blue Waters portal, you can just authenticate with the same information as you log in to the portal with.

Adding an Endpoint Click to get Globus Connect Personal so you can make your own computer an endpoint.

Getting Globus Connect Personal Follow the appropriate directions for your operating system. & generate your globus connect personal setup key The key will be asked for once you install the program

Globus File Transfer now enter in an endpoint ncsa#BlueWaters this is what contains your project data and the other endpoint should be from those “Administered by you” This will lead to a local place for your file such as your documents.

WinSCP

WinSCP Saved connections

WinSCP s Select the File Protocol: SFTP SCP FTP WebDAV WebDAV – an extension of Hypertext Transfer Protocol (HTTP) that allows clients to perform web content authoring content. The WebDAV protocol provides a framework for users to create, change and move documents on a server, typically a web server or web share. The most important features of the WebDAV protocol include the maintenance of properties about an author or modification date, namespace management, collections, and overwrite protection. Maintenance of properties includes such things as the creation, removal, and querying of file information. Namespace management deals with the ability to copy and move web pages within a server’s namespace. Collections deal with the creation, removal, and listing of various resources. Lastly, overwrite protection handles aspects related to locking of files.

WinSCP s Specify the hostname: h2ologin remote desktops WebDAV – an extension of Hypertext Transfer Protocol (HTTP) that allows clients to perform web content authoring content. The WebDAV protocol provides a framework for users to create, change and move documents on a server, typically a web server or web share. The most important features of the WebDAV protocol include the maintenance of properties about an author or modification date, namespace management, collections, and overwrite protection. Maintenance of properties includes such things as the creation, removal, and querying of file information. Namespace management deals with the ability to copy and move web pages within a server’s namespace. Collections deal with the creation, removal, and listing of various resources. Lastly, overwrite protection handles aspects related to locking of files.

WinSCP s Your username and password for the specified hostname above. WebDAV – an extension of Hypertext Transfer Protocol (HTTP) that allows clients to perform web content authoring content. The WebDAV protocol provides a framework for users to create, change and move documents on a server, typically a web server or web share. The most important features of the WebDAV protocol include the maintenance of properties about an author or modification date, namespace management, collections, and overwrite protection. Maintenance of properties includes such things as the creation, removal, and querying of file information. Namespace management deals with the ability to copy and move web pages within a server’s namespace. Collections deal with the creation, removal, and listing of various resources. Lastly, overwrite protection handles aspects related to locking of files.

WinSCP

WinSCP Remote File System Local File System