1. Disaster Recovery Planning: Sharing Experiences from Blackboard ASP Services
Harry Choi – Director, ASP Services
Jonas Hirshfield – Director, ASP Infrastructure Operations
© Blackboard, Inc. All rights reserved.

2. About Forward-Looking Statements
Any statements in this presentation about future expectations, plans and prospects for Blackboard and other statements containing the words “believes,” “anticipates,” “plans,” “expects,” “will,” and similar expressions, constitute forward-looking statements within the meaning of The Private Securities Litigation Reform Act of Actual results may differ materially from those indicated by such forward-looking statements as a result of product development changes and other important factors discussed in our filings with the SEC. We may make statements regarding our product development and service offering initiatives, including the content of future product upgrades, updates or functionality in development.  While such statements represent our current intentions, they may be modified, delayed or abandoned without prior notice and there is no assurance that such offering, upgrades, updates or functionality will become available unless and until they have been made generally available to our customers.

3. Agenda: Blackboard ASP Services Introduction
MTTR Planning vs. Disaster Recovery
Planning for Disaster Recovery:
Defining the Definition
Setting up the Requirements
Planning & Testing

4. Blackboard ASP Services Introduction
Over 450+ Customers from All Market Sectors
4 Datacenters – 2 in Northern Virginia; 1 in Vancouver; 1 in Amsterdam
Over 4 Million Users in our Systems – Over 3 Million Active Users
1500+ Servers, 250 TB Storage Capacity & Growing
7 Terabytes of Data Transferred Daily
100 Million + HTTP Requests Served Per Day at Peak Times
Blackboard Academic SuiteTM, Blackboard Commerce SuiteTM, Hosting
99.7% or Better Uptime Guarantee

5. Blackboard ASP Clients
450+ Clients from All Market Sectors

6. Fully Managed Service: Backed by Team of Experts

7. What is “Disaster Recovery”?
“Ability to recover from the loss of a complete site, whether due to a natural disaster or malicious intent.”
“A plan of action to recover from an unlikely event of a severe or catastrophic business disruption.”
It’s NOT a planning for Mean-Time-To-Recovery (MTTR) from daily operational risks.

8. MTTR Choices: Recovery-Oriented Scalable Infrastructure
On-Demand, Redundant Scaling Technologies
Use of attached clustered storage allows quick use of client growth
2N Redundancy at Core Infrastructure Level
Burstable, Redundant Internet Connectivity
Load Balancing
Dual Core CPU, Caching, Hyper-threading
Clustering Technology
Recovery Oriented Choices:
Autonomic Capabilities – Datacenter / Network / Systems
Warm Standby (rack, stacked and powered up)
SnapMirror Technologies
Oracle DataGuard & RMAN Technologies
Cold Standby
“Platinum” Service Contracts from Service Providers and Vendors
N+1 Redundancy capabilities at client level
It’s All About Tolerance for MTTR vs. Costs

9. Bb ASP Network Highly available network design
No single point of failure at Infrastructure layer
Quarterly Infrastructure Testing
Maintenance with minimal downtime
Enterprise class hardware devices
Clustered Network Appliance Storage
Foundry Routers and Switches
Juniper Firewall/VPN
Tipping Point IPS
NetScaler Load Balancers
Infloblox hardened DNS Appliances
Optional Security Services
VPNs for secure data transfer
SSL for encrypted access

10. Levels of Data Backups
1st level backup: using snapshot technology, file systems & database backed up on Network Attached Storage systems
2nd level backup: using NetApp NearStore devices, 1st level backups are stored online, off-site for 30 days
3rd level backup: weekly backups are stored in tape, off-site for 30 days

11. Monitoring Tools for Proactive Management
Key Performance Metrics polled every 5 minutes
cpu = 5-minute load average
disk = % free on local mount points
nfs = % free on NFS mount points
msgs = scanning syslog for key words
memory = real, virtual and swap
procs = scanning ‘ps’ output
Obak = Oracle backup status
ofiles = Oracle data/index file sizes
files = other file/directory sizes
svcs = Windows Services checking
Application Check for Response Time

12. Disaster Recovery Requirements
Set the institution’s definition of “disaster”
Driven by Business Impact
Priority of Mission Critical Applications
Define Requirements
Set Threshold for Recovery

13. Questions to Consider:
What is the threshold on recovery time (RTO) and recovered data (RPO)?
What is the objective during disaster recovery period:
Minimum Basics function – i.e. online materials availability and course continuation?
Full Production Availability, including LDAP, Customizations/Building Blocks availability?
What is the plan for post-DR?
How does Blackboard ASP Service address the Management & Monitoring factor? By operating 24 x 7 with dedicated, experienced personnel.

14. ASP Business Continuity Service Offering:
Recovery Time Objective (RTO)
RTO is the time-measured objective to have the Blackboard Business Continuity Service operation up and running from the point in time that Blackboard is made aware of the client’s primary Blackboard applications system failure.
Recovery Point Objective (RPO)
RPO is the objective to minimize the loss of the client’s database and file storage content by constantly backing up the client’s information no less than the time guaranteed under each service level.
Customizations & Configuration
Dependent on Client’s requirement and RTO & RPO objectives
How does Blackboard ASP Service address the Management & Monitoring factor? By operating 24 x 7 with dedicated, experienced personnel.

15. Business Continuity Service Levels
Recovery Time Objective (RTO)
Recovery Point Objective (RPO)
Customization & Configuration
Locally hosted client:
Weekly backup service
24 hours
Last backup less than
8 days
Fully backed up
(if possible)
Monthly backup service
72 hours
Last backup less than 32 days
Rebuild
Incident-based service
120 hours
Latest backup received
N/A
Blackboard ASP hosted client:
Level 3 Service
6 hours
3 hours
Level 2 Service
Last backup less than 12 hours
Level 1 Service
Last backup less than 24 hours

16. Business Continuity Service Offering:
Testing & Collaboration During Setup
Customized Service
Building Blocks, LDAP, & other considerations
Simulation Tests Once a Year
Database Server(s) is dedicated & App Servers are Reserved
In DR situation, Service up for 30 days
Client Audit of Datacenter & Operation Welcome
How does Blackboard ASP Service address the Management & Monitoring factor? By operating 24 x 7 with dedicated, experienced personnel.

17. Disaster Recovery Option
Optical Fiber Ring
Private Zone
Public Zone
Private Zone
Public Zone
Blackboard ASP VA1 Facility
Blackboard ASP VA2 Facility

18. Disaster Recovery – Client Options
Client Application Content
Use of Network Appliance’s Snapmirror technology to mirror client application data in real time across facilities.
Dedicated standby hardware
Client Oracle Content
Traditional Oracle Exports. Long RTO & RPO
Oracle RMAN Technology Long to Medium RTO & RPO
Warm dedicated Standby Server
Oracle DataGuard – Best RTO & RPO
Hot dedicated standby server

19. The Human Factor: Rigorous & Regular Training Redundancy in Skill Sets
Plan for loss of critical staff in a DR event.
Change Management Control Tools – Central Authentication System, Automated Scripts, Documentation, etc.
Readiness Tests – e.g. Fall Preparation Readiness Testing
Perform routine testing to ensure technology is working as expected
Documentation
Disaster recovery procedures should be well documented.
Plan for the unexpected.
Loss of critical staff
No Physical Access to the facility
Loss of traditional internet access to the facility
Install POTS line with serial connections to infrastructure

20. Questions? Comments?