1. Redesigning the Enterprise to Achieve 5-Nines
Link Alander, Associate Vice Chancellor
Shah Ardalan, Vice Chancellor/CIO
September 2010

2. 1 - Key Objective
37 – Major Projects
18 months
99.999

3. 152 - OTS Staff Members 1 - Key Objective 37 – Major Projects
18 months
152 - OTS Staff Members

4. 6 – Strategic Partners 152 - OTS Staff Members
1 - Key Objective
37 - Projects
18 months
6 – Strategic Partners
152 - OTS Staff Members

5. Project High Level Overview
agenda
Project High Level Overview
Overview Lone Star College System
Our Challenges
Our Approach - Simplify IT
It Isn’t Just Technology - IT Maturity
Dynamic Master Plan
LAN/WAN, AD, SAN, VM …….
Design Scenario - Modeling
The Outcomes – Into the Cloud
Lessons Learned – Takeaways
Q&A
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

6. Overview LSCS - Enterprise
13 Locations across 1,400 square miles
5 Colleges
2 University Centers
5 Instructional Centers
Systems Office and Training Center
21 new buildings this year
5 new geographic locations
Our Customers (90,000 strong)
69,363 Credit Students
21,000 Non Credit Students
Average 25% growth per semester
Over 4,800 employees
Supported by 152 OTS staff members
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

7. This was not our only project!
Our Challenges
Dec 2007
Unreliable service delivery to faculty, staff and students
Zero confidence in IT!
Feb 2008
New Vision - Mission
6 IT departments into 1
The Office of Technology Services
Building the team
Setting standards
May 2008
One Major Objective (5-nines) that impacts all systems and services
37 Major projects
Critical dependencies
Accelerated timelines
This was not our only project!
20 Strategic Initiatives
New ERP
IT Security
Client Mgt System
… much more
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

8. The plan – Simplify IT
March 2008
The term “Simplify IT” can be found in every business journal, book, and a couple million times on the web “What does it mean?”
Evaluate Everything – Reduce Complexity
Set Enterprise Standards
Select “Best in Class” Solutions
Develop Detailed Process Methodologies
Increase Staff Knowledge
Reduce Interdependencies
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

9. It isn’t just technology!
You cannot achieve 5-nines with technology alone!
- It is about People, Process and Technology
By raising your
IT maturity,
you will become a Strategic Business Partner
that will
add value
to the organization
IT Maturity Model and Assessment
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

10. Dynamic master Plan Key Objectives Assessments Network Redesign
Active Directory Infrastructure
Maximize Virtualization
Security and Monitoring
Assessments
Enterprise Standards
Storage Area Network
Data Centers
Target Objective - (99.999%) Tier I
Eliminate Single Points of Failures
Expand Replication Technologies
Leverage Automation
Deep Monitoring Strategy
18 months to build new environment and migrate users
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

11. Network Redesign – Starting Point
March 2008
Starting Point - Network
300MB ONS Ring
48 Single Points of Failure
4,599 miles of Dark Fiber
1 – 500 MB Internet connection
1 – 10 MB Internet connection DR only
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

12. Network Redesign - Head End
December 2009
New Network Head-End
2 physical locations 37 miles apart
Logically separated networks (not extended)
Nexus 7000 & 5000
Dual Internet Connections
Redundant firewalls
Redundant DMZ
ACE Load Balancer
10 GB SAN Fabric X 4
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

13. Network Redesign - WAN Final State - Network July 2010
Full Mesh 10GB links
28 VSS Core Switches
49,240 Network ports
1,374 Network devices
4,750,342 Sq ft of Wireless network coverage
Comprehensive network monitoring - SolarWinds
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

14. Active Directory - Exchange
June 2008
Phase I - Build New AD/Exchange Environment
Problems
14 domains with limited trusts (one-way)
27 DNS zones
16 Wins servers
Constant replication errors
No health monitoring
No automated account management
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

15. Active Directory - Exchange
March 2009
Phase II – Migrate Users
New AD Environment
2 Domains (Admin/Student)
Exchange 2007
Challenges
10,000 Exchange accounts
27 DNS zones to 5
16 Wins servers to 3
Automation (Self Healing)
Implemented Change Auditor
Implemented Health Monitoring
Automated Account Provisioning
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

16. Active Directory - Exchange
November 2009
Phase III –
Migrate Servers
Decommission Old Domain
Challenges
Over 400 physical servers
14 locations
70% of hardware end of life
Deployed Enterprise Virtualization Solution
Decommissioned 260 Servers
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

17. SAN – Back-up & Recovery
Starting Point - SAN
3 Disparate SAN’s in Main Data Center
5 Separate Solutions Deployed on Campuses
End-of-Life Arrays
No Standard or Reliable Back-up and Recovery Solution
High Data Security Risks
Starting Point - SAN
2GB Broacade
20 TB replicated
Limited A/B Paths
Majority single paths
No “Single Pain of Glass” to manage the SAN
No true integration with backup solution
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

18. Consolidated all SAN’s High Availability Design
SAN – Backup/Recovery
September 2008
March 2009
September 2009
Consolidated all SAN’s
Traditional Tier I,II & III
High Availability Design
Disk to Disk Enterprise Backup
Critical for Short RTO, RPO Windows
Single Enterprise Tape/Off-Site Solution
Integrated EMC Storage Management with VM Storage Virtualization
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

19. One Main System Data Center
Main Data Center
July 2008
One Main System Data Center
Tier II design
Inefficient – high risk!
Campus Data Centers
Lacked
Back-up power/cooling
Fire Suppression
Evaluated Hosted Locations
Problems extending co-located hot systems
Typical All Hot, All Cold or Warm (not Active-Active)
Acquired New Facility - LSC-University Park
Formerly Compaq World Headquarters
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

20. Decision to Build New Data Center – March 2010
Main Data Center
September 2010
Decision to Build New Data Center – March 2010
Selected Design and Construction Firm – June 2010
Commissioned New Tier IV Data Center – September 2010
Focus - Green – High Density Design
Dual Commercial Power Feeds
Dual Generator Plants
240KW load capacity – per A/B Path
99.9 % Efficient and Scalable UPS
Closed Loop Cooling (water cooled racks)
In-Rack Fire Suppression System
Packed into 970 square feet!
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

21. Starting Point - Virtualization
Less than 5% of servers were virtualized
Limited Test-Dev Utilization
A few Production Servers
Limited Deployments
System Office
One Campus
Typical Single Server License Purchased
LSCS-VM Farm Capacity Today
2 Virtual Centers
14 Virtual Data Centers
468 VM’s
327 SRM Cold VM’s
136 Physical CPU’s
544 CPU Cores
1,500 GHz
1.4 TB Memory
790 VM CPU Count
4 HP C7000 Blade Systems
6 HP C3000 Blade Systems
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

22. VMware ELA (all u can eat)
Virtualization
December 2008
September 2009
VM First Policy
VM Farm Design
Campus VM Solution
VMware ELA (all u can eat)
Lifecycle Manager
Site Recovery Manager
Today 82% Virtualized
Including Tier I applications
Target 95% by May 2011
Storage Virtualization
VM-SRM moves servers
VM-SRM moves servers
VM-SRM moves servers
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

23. Enterprise DEEP Monitoring
January 2009
February 2009
May 2009
Starting Point – Nothing!
Implemented Proactive Monitoring - Server and Service Level
Deep Monitoring
Service Level
Quest Foglight/Spotlight
Oracle Enterprise Manager
Alert Bot
Server Level
vFoglight
Solarwinds
Servers Alive
Nexpose`
E-Alert
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

24. Key Performance Indicators!
January 2009
September 2009
January 2010
Starting Point – Measured Nothing!
Key Performance Indicators
Financial (8), Operational (65), Strategic (4)
Risk Performance Indicators (32)
Reporting
Standard Monthly Reports
Executive Overview
Capacity / Demand Management
Agility!
Drills
Predefined Drills to Test Availability
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

25. Modeling – PeopleSoft (FSM Failover)
100 Virtual Server – 2 Physical Database Servers Per Side
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

26. Modeling – People Soft (FSM Failover)
NORMAL
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

27. Application Level - PeopleSoft
ERROR – FSM WEB
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

28. Application Level - PeopleSoft
ERROR – FSM WEB and APP
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

29. Application Level - PeopleSoft
ERROR – FSM WEB, APP and Database
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

30. Outcome – into the clouds
SAS
Hybrid
Dynamic Services
Hybrid
Dynamic Services
Dynamic Services
Hybrid
Hybrid/Public
Traditional Public
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

31. It’s not a Train! Lessons Learned Dynamic Master Plan Staff exhaustion
Enabled us to move faster
Made scope creep difficult to manage
Staff exhaustion
Motivating everyone – staff and customers
There is a light at the end of the tunnel – and -
Communication and Constant Measurement
Training, Training and more Training
Containing VM Sprawl
Continuous Service Improvement (CSI) – Is critical during the life of the project, not just after
It’s not a Train!
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

32. today Internet Redundancy WAN Redundancy Data Center & Co-Location
Co-located VM Farms
Storage Redundancy
LAN Redundancy
Server Virtualization (SRM)
Storage Virtualization
Intro
Planning
LAN/WAN
AD/Exchg
SAN
Datacenter VM
Monitoring
KPIs
Modeling
Close

33. Thank You - Q & A One Major Initiative - “5-nines” – 37 Major Projects
WAN Phase I – Core
Infrastructure Assessment
Wireless Network Phase I
VoIP Rebuild
Unified Infrastructure Project Phase I
SAN Phase I
VM First Policy – Datacenters
Quest Foglight
Quest Spotlight
KPI’s
Orion SolarWinds
Quest vFoglight
EMC - Avamar
Unified Infrastructure Project Phase II
EMC – Networker
Quest Change Auditor
VoIP Phase II
SAN Phase II
ERP Architecture
ERP Side A
Campus VM solutions
Cisco ACE Load Balancer
Unified Infrastructure Project Phase III
Temp Datacenter
SO Datacenter
Nexpose Deployment
Orion SolarWinds P II
Dual Internet Connections
ERP Side B
SAN Assessment
New Datacenter – CoLo
VM Assessment
LAN/WAN Phase II
SAN Phase III
Service Level Monitoring
New Datacenter Network
Datacenter infrastructure monitoring SO/CoLo