1. File sharing solutions - Introducing to Thunder ftp service
Welcome
Self_intro
only mandatory---- training session for BioHPC users, but we hope you’ll come along to many of our other sessions.
Training schedules----
The slides---- for this session will be posted on the portal, in the training section. There’ll be notes with the slides, covering roughly what I tell you here today.
Sign in and register----- will be activated today or tomorrow or else it will be delayed
[web] portal.biohpc.swmed.edu [ ]
Updated for

2. Inform users what/how to use for sharing on BioHPC Intro:
Overview:
Goal:
Inform users what/how to use for sharing on BioHPC
Intro:
Demands and Issues
BioHPC strategies/solutions on file Sharing
Details and comparison of file sharing solutions
What’s new?
Future plans
Quick case study
Tips/tricks

3. Demands and problems on File sharing
Sharing needs at UTSW
within BioHPC
within UTSW
External collaborator
Uploading to specific sites which accepts specific URLs
BioHPC problems/goal:
Allowed quick and safe file sharing
Allow consistent collaboration
Provide multiple methods/protocols to access and sharing files suited to everyone’s needs
Allow transferring of large amount of data
Allow password protected sharing
Allow creation of file URLs to directly pass files
BioHPC
UTSW
Sharing within BioHPC
Lab to lab collaboration
User to User collaborator
Sharing within UTSW
Sharing to non-BioHPC users
Sharing to anyone
Other collaborators from other parts of the world
Accessing data from off campus
Other:
Uploading data to specific sites which accepts specific URLs
BioHPC goal:
Allowed quick file sharing within BioHPC, safe data transfer
Provide Consistant
Provide multiple methods to access and sharing files to suit everyone’s needs
Protect data safety while transferring files
Allow transferring large amount of data
Anyone

4. Demands and problems on File sharing
Current file sharing solutions:
Cluster shared folders
Lamella
Cloud
cluster
lamella
cloud

5. BioHPC Cluster Shared Folder
Departments who have joined BioHPC can share and collaborate directly on BioHPC cluster.
Different level of sharing within biohpc cluster
/project/shared  needs special approval
/project/department/shared
/project/department/lab/shared
Inter-departmental sharing requires approval from department chair of the department hosting shared folder
Can have special permission on the interdepartmental sharing folders
John
Danuser_lab
/project/bioinformatics/ Danuser_lab/shared
Departments who have joined BioHPC can share and collaborate directly on BioHPC cluster.
Shared folder is created on the approval by both department Chair of the collaborating groups
An LDAP group is created
Specific users are allowed to join the group
Special permission – flexibility
Advantages:
Very fast sharing of huge datasets
No data transferring required, can be directly accessed on BioHPC
Permanently shared folder
Easy access, can be accessed with many methods, such as ftp
Disadvantages:
Limited to BioHPC departments and groups ONLY, cannot share with other people
/project/bioinformatics/shared/john_josh
Ana
Danuser_lab
John
Danuser_lab
Josh
Xiao_lab

6. BioHPC Cluster Shared Folder
Bioinformatics
Biophysics
/project
BICF
GCRB
John
Mary
/project/shared/BICF_GCRB
Mike
Ana
Jane
Mike and Jane from BICF is collaborating with Mary from GCRB, BICF belongs to department of bioinformatics and GCRB belongs to Biophysics department

7. How to set up interdepartmental shared folder – Explain and DEMO
wants to collaborate with 1 1
Mike and Mary discuss and came up with the name of the shared folder, members of the group and size of the shared file 2
Send biohpc_help a ticket for the plan of the shared folder, including the name, members and size of the shared folder, also specify hosting department 3
BioHPC will sort out the approval and create the shared directory 2
Gaudenz 3 3 3
BioHPC_help
D1_D2 shared

8. BioHPC Cluster Shared Folder
Pros:
Direct access – simplest and fastest
No size limitations on files
Permanent – long term collaboration
Easy access – easy to setup, easy to access
Cons:
Limited to BioHPC departments and groups ONLY, cannot share with other people
Setup could take up to one to two days
Cluster
Departments who have joined BioHPC can share and collaborate directly on BioHPC cluster.
Shared folder is created on the approval by both department Chair of the collaborating groups
An LDAP group is created
Specific users are allowed to join the group
Special permission – flexibility
Advantages:
Very fast sharing of huge datasets
No data transferring required, can be directly accessed on BioHPC
Permanently shared folder
Easy access, can be accessed with many methods, such as ftp
Disadvantages:
Limited to BioHPC departments and groups ONLY, cannot share with other people

9. Lamella – Internal: Sharing between a BioHPC user and anyone on UTSW:
Lamella and Cloud
Lamella – Internal: Sharing between a BioHPC user and anyone on UTSW:
URL: lamella.biohpc.swmed.edu
Has Dropbox-like web interface
Only accessible when on the UTSW network
Can be used to share specific folders in the cluster directory, allowing easy customization
Cloud – external: Similar to lamella, simple web interface to access shared files and folder:
URL: cloud.biohpc.swmed.edu
Same backend software as lamella
Is accessible from the internet (not on the UTSW network)
Cannot access cluster folders
For all BioHPC users sharing with UTSW

10. Lamella and Cloud – Major Differences
Cluster Storage
FTP server
File mounting server
Lamella
Internal Web
UTSW collaborator
Cloud/file exchange
External Web
external collaborator

11. How to share using lamella and cloud - – Explain and DEMO
Sharing using cloud and lamella is straight forward:
Click this icon on
Files/folder you want
to share
Log in
More detailed Guide:
Select appropriate settings for this share

12. Hands on BioHPC – 2. Manage Files with Lamella / Cloud Storage Gateway
File Sharing
Only you can open, never expires
Any user can open, valid before expiration date
Lamella cloud storage : sharing with user inside UTSW
File Exchange : sharing with user outside UTSW

13. Hands on BioHPC – 2. Setting up Lamella to access project and work space
BioHPC Endosome/Lysosome
project
work
home
For home leave blank
For private project space:
department/lab/user
For lab shared project space:
department/lab/shared
username
password
Log-in credentials, save in session uses the BioHPC login credentials and only saved in the user session, giving increased security. The drawbacks are that sharing is disabled, as lamella has no access to the cluster storage credentials.
Username and password mechanism requires a manually-defined username and password. Remember to click the gear icon and enable sharing.

14. Lamella and Cloud - Pros and Cons
Quick to setup, easy to use
Can set time limit and password to the shared link
Can limit read/write access on the shared folder
Generate secure hash links for folder sharing
Customizable
Lamella – can mount cluster directory
Cloud – can be accessed by anyone
Cons:
Lamella – Limited to UTSW network ONLY
Cloud – Cannot mount cluster directories
Must use the web interface for sharing, so data must go through the web server
Has Limited size on local storage, lamella : 100G; cloud: 50G

15. Achieved goals and persistent issues
Quick file sharing within BioHPC
safe data transfer
Consistent collaboration
Provide multiple methods/protocols to access and sharing for biohpc user
Large data transfer internally to Biohpc user
Can share with external collaborator
issues:
Sharing large data with non-biohpc users in campus and to external collaborators
Limited methods for persons other than BioHPC user to access the shared folder
Unable to generate file address to upload to UCSC like web sites

16. New FTP service on BioHPC, Create temporary password for plain ftp
Thunder FTP:
New FTP service on BioHPC,
Create temporary password for plain ftp
User managed account and guest login
Login directly into BioHPC project folder:
Creates ftp URL to files which can be used to upload/download
Upload files to server which requires FTP url, such as UCSC genome browser
Aiming to bring another FTPS server for external network.
Server Address: Thunder.biohpc.swmed.edu
FTP Directory: /project/thunder_ftp/$USERNAME
Advantages:
Can directly send file link to collaborator
Uses ftp protocol, more reliable
No limit on file size
Disadvantages:
Plain ftp is not as secure as lamella or cloud
Not permanent sharing
Still in Beta version

17. Cluster Storage Thunder Web Sites accepting ftp://
Thunder FTP data flow:
Cluster Storage
/project/thunder_ftp/$username/{specific directory}
Thunder Web
Sites accepting ftp://

18. Use FTP client to access your files: Log in:
How to use Thunder
Use FTP client to access your files:
Log in:
Activate:

19. How to share using Thunder – Explain and DEMO
Step 1
Login to thunder.biohpc.swmed.edu
Click “renew password”
Step 2
Click “Create Guest”
Fill the form, this will create or use a directory
Step 3
User will receive ftp credentials in
User can ftp into the shared folder
Step 4
User can directly disable account or password expired in three days after the creation

20. Chart comparison – which is the best options?
Method
Cluster
Lamella
Cloud
Thunder
Local storage size
Quota
Quota/100G
50GB
Speed
Fastest
Slower
slowest
Fast
Conn. Reliability
Most Reliable
Moderate reliable
Least reliable
Very Reliable
Web based no
yes
Access Ctrl
File/Folder permission
Password
Duration
Permanent
Flexible
Short
Share with
BioHPC
UTSW
Anyone
Easy to Use
Moderate
Easy

21. Speed and Flexibility – the most important aspects?
Data Size Limitation
speed
cluster
Quota
IB -
100G
10G -
50G
Lamella
Thunder
1G -
Cloud
internet - I
Internal I
Campus I
everyone
Flexibility

22. Case Study – How to choose your choice or sharing
Which is the best BioHPC sharing solution for the below cases?
Case 1: John want to quickly share 300GB image sets to a collaborator in another UTSW department which did not also joined BioHPC.
Case 2: Mary is going to work on a project with users in Lab B from another department for at least 6 months, data size around 20GB is exchanged constantly between these labs
Case 3: Ben has a external collaborator who he occasionally sends a few files of size >1G to.
Case 4: Lee has a non-Biohpc collaborator on campus who’s going to send him 10GB of data.

23. Sharing really large data: Internally – thunder Externally – ???
What now? – Future plan
Data mining age:
More storage usage
High precision data
Sharing really large data:
Internally – thunder
Externally – ???
Future plan:
Moving thunder to external network to allow sharing large amount of data to outside.
Working on allowing accessing cloud from cluster
???

24. Sharing made simple – DEMO tricks
Sharing via web is simple, but can it be even easier?
Make a link to the shared folders on your workstation desktop
ln -s /path/to/folder /path/to/link
Create a “persistent connection” between your local system and the shared directory, directly access your data from local
When using cluster sharing, If a large data set (50G) in the project directory, using mv and chmod to share to file instead of cp, the data does not need to be replicated.

25. Questions?