1. Case Study: Georgia Tech
University Private Cloud for Researchers
Didier Contis Director Technology Services College of Engineering Georgia Institute of Technology
Joe Arnold CEO SwiftStack Inc.

2. Session Speakers Didier Contis Joe Arnold
Didier Contis is the Director Technology Services / College of Engineering, Georgia Institute of Technology Greater Atlanta Area. The largest of Georgia Tech’s six colleges, CoE offers more than 50 graduate and undergraduate degree programs through its main Atlanta campus and satellites around the world. Its 13,000 students use an estimated 150 unique apps—the same ones businesses rely on to design airplane wings, model circuit-board layouts, and much more.
Joe Arnold is co-founder and CEO of SwiftStack, a leading provider of object storage software. SwiftStack's customers include some of the largest web and enterprise IT organizations. Joe managed the first public OpenStack launch of Swift after its release as an open source project. He has been active in the OpenStack community since Joe is the author of Object Storage with Swift published by O'Reilly Media.
Georgia Tech Case Study - OpenStack Summit 2014

3. Object Storage and OpenStack Swift
Joe Arnold CEO SwiftStack Inc.
Georgia Tech Case Study - OpenStack Summit 2014

4. Swift Object Storage – Key Attributes
“OpenStack Swift in particular has gained a lot of traction both in the enterprise and in the service provider space”
October 2013
Open-source object storage system
Powers the largest storage clouds
Geographically distributed
Multi-tenant
Massively concurrent
Extremely durable
Runs on standard Linux
Inexpensive commodity x86 hardware
“Swift is a proven solution, suitable for production needs, and should be included in competitive evaluations of object-based storage solutions.”
February 2014
October 2013
Georgia Tech Case Study - OpenStack Summit 2014

5. Swift Data Redundancy
Swift places 3+ replicas of all data as unique as possible
Single Node Cluster
Disks are “as-unique-as-possible”
Large Cluster
Storage Racks are “as-unique-as-possible”
Small Cluster
Storage Nodes are “as-unique-as-possible”
Muti-Region
Distributed data centers are “as-unique-as-possible”
Georgia Tech Case Study - OpenStack Summit 2014

6. Swift Object Storage – Filesystem Conceptual View
CIFS
SwiftStack Filesystem Gateway
NFS
Swift Node
Swift Node
Swift Node
Georgia Tech Case Study - OpenStack Summit 2014

7. Georgia Tech Case Study
Didier Contis Director Technology Services College of Engineering

8. Who we are and what we do
Georgia Tech: 21,471 undergraduate and graduate students (Fall 2013)
Six colleges: Architecture, Business, Computing, Engineering, Liberal Arts and Sciences
6th Top Engineering Graduate Programs
5th Top Engineering Undergraduate Programs
13,000 students in the College of Engineering, largest in the U.S.
Georgia Tech Case Study - OpenStack Summit 2014

9. We have been deploying pre-cloud systems since 2007… Meet our federated condominium systems
Georgia Tech Case Study - OpenStack Summit 2014

10. Our VDI / App publishing farm…
Virtual Lab Project and its supporting shared infrastructure (Matrix)
Georgia Tech Case Study - OpenStack Summit 2014

11. Our HPC farm…
PACE: Partnership for an Advanced Computing Environment HPC federation / condominium system. 28,000 cpu cores and 2PB storage
Georgia Tech Case Study - OpenStack Summit 2014

12. What we learned: 1) Our HPC and VDI users love compute power 2) They love their research data even more They are not alone….
Georgia Tech Case Study - OpenStack Summit 2014

13. All our researchers / students love their research data
They love to
Acquire
Create
Exchange
Receive
Data….
Georgia Tech Case Study - OpenStack Summit 2014

14. Here is a research project generating a lot of data
Remote Sensing and GIS-enabled Asset Management System (RS-GAMS)
Assessment of pavement, bridge, and roadway assets using various sensors
Estimated Storage needs: 2,400 lane miles interstate highways currently on files with plan to analyze 2,000 miles in next few months…
Raw data: 2.2GB per lane mile
Processed data: 1.2GB per lane mile
16 Million jpeg files so far !!!!
Georgia Tech Case Study - OpenStack Summit 2014

15. Here is a research project receiving a lot of data
Effective Capacity Analysis and Traffic Data Collection for the I-85 HOV to HOT Conversion
The effectiveness of the implementation of the HOT lane is being evaluated in a before and after study.
Direct fiber network feed from Georgia Department of Transportation to Georgia Tech
Over 400TB of videos currently stored on random fileservers, USB drives…
Lots more video to collect….
Georgia Tech Case Study - OpenStack Summit 2014

16. Oh, by the way have you heard about: Research Data Curation
The White House Office of Science Technology Policy:
“has directed Federal agencies with more than $100M in R&D expenditures to develop plans to make the published results of federally funded research freely available to the public within one year of publication and requiring researchers to better account for and manage the digital data resulting from federally funded scientific research.”
Georgia Tech Case Study - OpenStack Summit 2014

17. So where do we store all this data?
Georgia Tech Case Study - OpenStack Summit 2014

18. Research Data Storage Challenges
 Our Challenges:
Obviously we have a lot of research data. How much ???
(2 PB just for HPC)
Cheap enterprise level storage is still expensive
Backup is a problem (cost, time)
 Meet our BIGGEST challenges:
Bring-your-own-drive – USB, thumb
Consumer cloud – Dropbox, etc.
Storage on demand
Georgia Tech Case Study - OpenStack Summit 2014

19. Research Data Storage Challenges
 “Sometimes” important research data might be stored might on not so reliable solution:
Due to cost of existing “enterprise storage” and research programs funding
Backup? Could you repeat the question please?
!!! WARNING UNCONFIRMED REPORT !!!
Ultimate cheap NFS File Server circa 2006
Refurbished Desktop tower
Two 5 ports USB cards
13+ USB drives each shared individually via NFS
Georgia Tech Case Study - OpenStack Summit 2014

20. Our magic answer to all our problems ???
VAPOR the hybrid cloud
Georgia Tech Case Study - OpenStack Summit 2014

21. Meet VAPOR
Goal: Build a Georgia Tech Distributed and Federated Academic Cloud
 Proposed design principles:
Led by Academic Units in partnership with Central IT
(Currently College of Engineering, College of Science, College of Computing, Library, HPC PACE Group, Office of Information Technology)
Support Instruction and Research at Georgia Tech
Distributed across campus and beyond (Hybrid)
Federate multiple departmental projects
Design / Architecture by Committee
Academic Governance Oversight
Need to be able to experiment and iterate quickly !!!!
Georgia Tech Case Study - OpenStack Summit 2014

22. Proposed Use Cases for VAPOR Cloud
Ephemeral computing: A machine runs for short term use. Possibly for development/testing purposes.
"Pet" computer: student needs a system which is "permanent", stateful and accessible both off and on campus. Basic usage is like VDI.
IaaS: Running campus services (both production and beta) on VMs. E.g. I don't want to manage a hardware layer but I need to set up a purpose built website to host data and services for an international research group and webhosting doesn't meet my needs.
PaaS: Running a platform. E.g. I don't want to manage hardware or the OS layer, but please give me a database I can use for this application.
Georgia Tech Case Study - OpenStack Summit 2014

23. <DRAFT> Vapor Architecture Vision </DRAFT>
Self-Service
(to be defined / under investigation)
Management
(Microsoft Azure Pack / Redhat CloudForm…..)
Management
On-premise Component
Off-premise Component
HYPERVISOR or CONTAINER PODs
(Hyper-V / KVM / XenServer + NVIDIA vgpu…)
Amazon AWS
Microsoft
Azure
RackSpace
COMPUTE STORAGE
(Gluster / Ceph / Scale-IO / ...)
DATA STORAGE
(Swiftstack / DDN WOS / Gluster…)
NETWORK
(VXVLAN / NVGRE / VPN…)
NETWORK
(VXVLAN / NVGRE / VPN…)
Georgia Tech Case Study - OpenStack Summit 2014

24. Today we are focusing on…. Data Storage
Georgia Tech Case Study - OpenStack Summit 2014

25. Vision of the Data Storage layer
Will hold a large portion of GT “Research Data”
Probably multiple data storage layers (multiple vendors / technology)
Some of our current requirements:
Distributed and Resilient (support multiple catastrophic failures)
Limit vendor dependency / lock-in (priority to open source)
Leverage de-facto standards (S3 / Swift)
Support multiple entry points (API, Cloud NAS, pluggable services)
Flexible design to limit the need to migrate data to new systems down the road
Integration with Georgia Tech identity management system (LDAP & AD)
Georgia Tech Case Study - OpenStack Summit 2014

26. Services supported by the Data Storage Layer / Swift
Research Data Storage
Research Data Curation
Research Data Repositories
DATA STORAGE
(SwiftStack – Storage as a service)
“Dropbox” type service
Filesystem Gateway
(CIFS / NFS / GPFS/ ….)
Georgia Tech Case Study - OpenStack Summit 2014

27. Why Swift / SwiftStack for the Data Storage Layer?
 Like:
Swift is open-source (limit vendor lock-in in our mind)
Turn key approach / manageability provided by SwiftStack
Growing ecosystem around Swift
Low hardware requirement / homogeneous hardware not required
System seem robust -> replication rather than RAID technology
Price is right !!! (so far….)
 Don’t Like:
It is object storage / not native filesystem
Still young project / product
Georgia Tech Case Study - OpenStack Summit 2014

28. Research Project Candidates to use Swift
Projects in Aerospace, Transportations and BioEngineering currently targeted
Examples of research projects looking / experimenting with Swift:
Effective Capacity Analysis and Traffic Data Collection for the I-85 HOV to HOT Conversion
Remote Sensing and GIS-enabled Asset Management System (RS-GAMS)
Georgia Tech Case Study - OpenStack Summit 2014

29. Our current strategy to engage research groups
Goal: Incentivize researchers to store data directly into Swift as objects when it makes sense
This means demonstrating advantages from:
Indexing
Metadata
Scalability
Performance
Future benefits (analytics)
It also means making an up-front investment:
In training and technical assistance
Providing free storage
Files
Apps
Scripts
SwiftStack CIFS/NFS
Gateway
Swift HTTP APIs
Swift
Node
Swift
Node
Swift
Node
Georgia Tech Case Study - OpenStack Summit 2014

30. Filesystem Gateway Using Object Storage natively is difficult
Lots of workflow based on using files. Our students are using Windows / Linux applications packages which are not object friendly.
Latency / speed is also an issue
Strategies being deployed or investigated:
SwiftStack gateway with lots of cache
Would like a GPFS Gateway (High Performance Computing)
Storage abstraction technology (Software defined storage utopia…. EMC ViPR Data Services ?)
Directory
Georgia Tech Case Study - OpenStack Summit 2014

31. Swiftstack FileSystem Gateway

32. Filesystem Gateway – No Data Lock In
Objects
Files
No lock in due to encoding with SwiftStack gateway
Data in/out same via Swift API or via CIFS/NFS filesystem
Traditional gateways (like S3/Glacier, Avere, Panzura) are a 'medieval marriage....forever‘
These gateways severely lock data in – all data going in via gateway MUST come out through same gateway
It's a "Hotel California“ for data – you can check-out any time you like, but you can never leave
Other gateways with lock have other benefits/features
E.g. deduplication, compression, etc.
Or offer POSIX required for some applications
SwiftStack Filesystem
Gateway
Swift HTTP APIs
Swift
Node
Swift
Node
Swift
Node
Georgia Tech Case Study - OpenStack Summit 2014

33. Filesystem Gateway – Simplified Account Management
Traditional gateways often use one 'cloud' account for all access
E.g. organization’s single S3 account shared by all users
Simple to use but makes security and account management meaningless
SwiftStack filesystem gateway uses separate account for each user
Same individual account used for both filesystem and direct object access
ACL's, access, usage all map back to that user's account regardless of path
So authentication of access by either object or file is the same
Authentication, permissions, and utilization (for chargeback) are meaningful
Greatly simplifies management/access for operations teams deploying Swift
Common authentication mechanism means far less to integrate
Georgia Tech Case Study - OpenStack Summit 2014

34. File System Gateway Benefit – Data Access Migration
Multiple access paths to data simpler for both users and developers
Example:
Data acquisition devices capture research data
Device software uses Swift HTTP APIs to write directly to Swift storage
User later browses filesystem gateway with analysis tool that only knows how to work with files
Data capture and use much simpler for users with multiple access paths
Makes Swift storage much more flexible to use and grow
Applications and devices can be migrated over to direct object storage over time
Legacy data capture/analysis can continue to use only files
Users can migrate access paths as data handling tools and practices evolve
Georgia Tech Case Study - OpenStack Summit 2014

35. Back to our programming
Georgia Tech Case Study - OpenStack Summit 2014

36. What about our Research Data Curation problem?
Initiative lead by Georgia Tech Library
Migrate from Dspace to a Research Data Curation repository built around Fedora (repository infrastructure) and Hydra (front end repositories)
Fedora 4.0 will connect to Swift
via JBoss ModeShape and Infinispan storage subsystem
Infinispan connection to Swift initially to use the Swift3 (S3) emulation layer
Georgia Tech Case Study - OpenStack Summit 2014

37. Swift zones distribution across campus
Federated and Distributed Academic cloud:
Each zone is located in a server room which is owned and operated by a different GT department….. No one own the cloud VAPOR ???
We expect more zones to come on-line in the next 12 months
Geographically distant region is on the roadmap (using hosting agreement with other Universities and Internet 2)
Zone ISYE 1
Zone PACE 1
Zone ECS 1
Georgia Tech Case Study - OpenStack Summit 2014

38. What hardware are we using?
Supermicro chassis, primarily 24 bays chassis
Hardware configuration is heterogeneous (different drives capacity, no same number of storage nodes per zone)
Drives are mix of Enterprise / Consumer grade. Mainly 1TB or 2TB
Most storage nodes have 10GB network connectivity (SolarFlare or Mellanox Connect- X). Currently SFP+ 10GB… 10GB Base-T is next
LSI SAS Adapter i (do not forget to re-flash if needed to change card from Integrated Raid to Integrated Target)
SSDs for Account/Container Ring (60GB to 120GB)
Memory to TB ratio?? 1GB of memory per TB can be expensive…
Georgia Tech Case Study - OpenStack Summit 2014

39. Distributed management of our Swift Infrastructure
Sysadmin from multiple departments share administrative responsibilities.
Would like more delegation granularities down the road: delegation on a per zone / region basis to enable for node management of cluster operator role.
Students are a great resource to replace dead drives…. If they know which one to replace…..
Georgia Tech Case Study - OpenStack Summit 2014

40. SwiftStack Auth and LDAP
Initially was considering using AD Integration
LDAP was a definite requirement. Availability delayed some of our testing / usage.
Georgia Tech LDAP size is fairly large:
ou=accounts -> 300K entries
ou=people -> 2M entries
So far so good. Initial integration was easy (5 minutes) but we waited until code was stable…..
Anyone with a GT valid account can access the Swift cluster !!!!
(but we have not advertised its existence…. Please don’t tell our students)
Georgia Tech Case Study - OpenStack Summit 2014

41. Our Financial Model Approach
Limit recurring cost at all costs !!!!
Fund recurring cost at the central level (licensing for example)
Focus on Bring Your Own:
Zone (BYOZ)
Server (BYOS)
Drive (BYOD)
We envision to use HDDs as a form of currency with research groups.
4TB of data to store = 3 x 4TB HDD payment for 5 years.
Hopefully storing 4TB will be negligible in 5 years when drives start to die. √
Georgia Tech Case Study - OpenStack Summit 2014

42. IT’S ALIVE…. Okay… So What’s Next…
Implement Quotas… probably container based Quotas
Re-architect the proxy layer this summer. Dedicated proxy nodes and possibly Load Balancer (Netscaler / F5)
Might deed to identify High performance NAS Gateway for specific workload ('medieval marriage....forever’)
Investigate support for a GPFS based Gateway
Keep educating people on long term benefits of using Swift API to access data
Unified access to data via SwiftStack FileSystem Gateway (convergence)
Georgia Tech Case Study - OpenStack Summit 2014

43. Managing Swift with SwiftStack
Joe Arnold CEO SwiftStack Inc.
Georgia Tech Case Study - OpenStack Summit 2014

44. SwiftStack Object Storage Software
Simple, Web-based MANAGEMENT
DEPLOY
Deploy in Minutes not Days
INTEGRATE
Seamlessly
SCALE
Without Disruption
OpenStack Swift (support included)
Standard Hardware & Linux Distribution
SwiftStack is object storage software that is deployed with OpenStack Swift at its core. We help simplify the management of deploying, integrating and Scaling OpenStack Swift. For instance…
We can help DEPLOY… in minutes
Hardware Selection/BOM
Performance Identification
Networking Design/Configuration
Built-in Load Balancing or Yours
Topology
Proxy Capacity
Zone/Region
Swift Component Install
Ring Building and Distribution
We focus on ensuring your INTEGRATION with your existing infrastructure goes seamlessly including…
Authentication
with AD, LDAP, OS
Monitoring
Networking
Load Balancing
Applications
Client API’s (Java, Python, Ruby, etc..)
Tools
Backup, Archiving
Users
FS Gateway
Client Apps
Web Interface
Policies
Migration S3
File System
Ob, Store ??
And we help you SCALE without disruption, including adding
Adding Hardware
Integrating with your Network
Capacity Planning, Additions and configuration
We have a web-based MANAGEMENT console that is geared toward simple and agile management and includes
Self-Healing Tuning
Alerts
Managing Users
Break/Fix
Rolling Upgrades
Integration Testing
Georgia Tech Case Study - OpenStack Summit 2014

45. Questions & Answers