1. 1 Building a Dynamic Datacenter with Oracle over NFS and NetApp Blackboard Managed Hosting (sm) Peter Waterman Senior Manager of Technology and Innovation pete.waterman@blackboard.com

2. 2 Blackboard Managed Hosting Blackboard Inc. is a world leader in e-Education software - our online learning application, the Blackboard Learning System, is the most widely-adopted course management system among U.S. postsecondary institutions. Blackboard Managed Hosting is a business unit within Blackboard which provides fully managed hosting services for all Blackboard software suites to a global client base of over 700 institutions. Our service includes management of every layer from the server metal to software administration and internet presence. Our clients demand cost effective reliability and performance – we require efficiency and agility to provide it.

3. 3 Six datacenter facilities Six datacenter facilities on three continents 1.2+ petabytes 1.2+ petabytes of NetApp enterprise storage 2200+ Dell servers 2200+ Dell servers (including 1400+ blades) 30+ million users 30+ million users across all installations 1040+ Oracle databases 1040+ Oracle databases 1350+ application servers 1350+ application servers providing web content 350+ million hits 350+ million hits per day across all installations 15+ terabytes 15+ terabytes of data transferred daily At a Glance…

4. 4 Our Typical Installation (mostly single tenant):

5. 5 Rebuilding the Box In late 2006, Blackboard Managed Hosting recognized that our yearly growth projections were so extreme that we would need to perform a core redesign of our methodology towards the following goals: Reduce complexity of system management Massively speed up backup and recovery Provide near-instant deployment and capacity expansion Improve system performance Reduce licensing and infrastructure cost using virtualization Avoid locking into a single current hypervisor

6. 6 Blackboard Advanced Hosting Platform Linux nfsroot: moves the OS to NetApp filers and allows migration between physical and virtual (P2V/V2P/V2V) Virtualization (RHEL5 Xen): allows effective performance upgrades for smaller clients and reduces future infrastructure and licensing costs NetApp Flexible Volumes: provides near-instant provisioning, recovery, cloning, and backups of NFS data Scalent V/OE: enterprise software for managing network booting, physical hardware, and virtual machines Oracle over NFS: moves commodity databases to NetApp filers, increasing performance and simplifying backup/recovery

7. 7 Oracle over NFS: Tier 1 Old standard deployed by Blackboard for 2+ years Follows standard Oracle/NetApp best practices whitepapers High cost offering for more critical clients Typically includes Oracle RAC Hardest piece to manage is disk IO availability and balancing load on filer heads

8. 8 ONFS Tier 2: You are doing WHAT?! For most of our clients, a pure ONFS implementation is not cost effective… Thus we built a solution that makes DBAs cry: Entire /home/oracle is on a single qtree No archive logs (archivelogmode is off) All snapshots are HOT – inconsistent Stop throwing things and Ill tell you why this actually makes sense!

9. 9 Oracle over NFS without Archive Logs…. Easy: One mount. One location. Snapmirror a master oracle qtree and you have a new database in moments. Cheap: Considerably less storage used without archive logs (snapshot < archive logs) Most Manageable: Clone a running database and start up the clone in seconds with a couple commands. Recover from any snapshot in seconds, whether rollback or clone, without having to match up archive logs.

10. 10 Not for Everyone though… Minor loss of data is a given in recovery/rollback/cloning when DB is hot Any PITR not available – can only recover to snapshots Minor performance hit during startup of inconsistent state database after recovery/rollback/cloning Some chance of planetary alignment causing complete loss of data (havent seen it yet, but weve tried pretty hard)

11. 11 Educate. Innovate. Everywhere. Okay, now you can throw things… Follow up questions: pete.waterman@blackboard.com Blackboard Managed Hosting