Implementing System Center Operations Manager 2007 at Microsoft Published: February 2007 Updated: February 2008

Microsoft IT wanted to improve and streamline its monitoring services. Decentralization of monitoring services increased support costs and caused a significant duplication of effort. Microsoft IT wanted to improve and streamline its monitoring services. Decentralization of monitoring services increased support costs and caused a significant duplication of effort. Solution Overview Business Challenge Solution Results/Benefits Microsoft IT implemented a centralized monitoring service by deploying System Center Operations Manager 2007 within its corporate network environment. Microsoft IT implemented a centralized monitoring service by deploying System Center Operations Manager 2007 within its corporate network environment. A more robust monitoring service A more robust monitoring service Elimination of monitoring fragmentation and better utilization of resources Elimination of monitoring fragmentation and better utilization of resources Ability to take advantage of previous efforts in creating, tuning, and managing management packs Ability to take advantage of previous efforts in creating, tuning, and managing management packs

Products and Technology ● System Center Operations Manager 2007 ● Microsoft Operations Manager 2005 ● Windows Server 2003 ● SQL Server 2005 SP1 ● Active Directory

Challenges of Using Operations Manager 2005 for Monitoring ● Significant duplication of effort to provide fault monitoring ● Gaps in the service offering ● No comprehensive, holistic view of the health of the network environment ● Dependence on a single expert to manage each group’s Operations Manager 2005 implementation

Benefits of Using Operations Manager 2007 for Monitoring ● Centralized monitoring, better resource utilization ● Rapid development of management packs ● Retention of previous work on management packs ● Use of health models ● More robust monitoring service ● Reduction of agent deployment efforts ● Efficient collection/storage of security event logs ● SDK that provides an interface for automation of administrative tasks

Microsoft IT Monitoring Solution ● Operations Manager 2007 enabled Microsoft IT to completely rebuild its monitoring infrastructure ● Application has significantly improved features introduced in previous versions ● Phases for building and deploying the new monitoring solution: 1. Planning the environment 2. Designing the infrastructure 3. Deploying Operations Manager 2007

Planning the Operations Manager 2007 Environment ● Main objectives for the design of the environment: ● Provide a complete monitoring service offering ● Provide a centralized and holistic monitoring platform ● Reduce resources devoted to monitoring

Designing the Infrastructure: Server Roles ● Operational database server ● Root management server ● Management servers ● Gateway servers ● Audit collection servers ● Data warehouse and reporting server

Designing the Infrastructure: Pre- production Environment

Designing the Infrastructure: Production Environment

Designing the Infrastructure: Microsoft IT Hardware Specs for Servers Server role Number of servers per unit ProcessorMemory Operating system Disks* Operational database server Two per management group— Windows Clustering cluster Two dual core 8 GB Windows Server 2003 with SP1, 64 bit SAN Windows Clustering Quorum: 5 GB SQL data: 130 GB RAID 10 SQL log: 10 GB RAID 5 TempDB: 10 GB RAID 5 Root management server Two per management group— Windows Clustering cluster Two dual core 16 GB Windows Server 2003 with SP1, 64 bit SAN Windows Clustering Root management server state drive: 5 GB Failover database server One per management group One dual core 8 GB Windows Server 2003 with SP1, 64 bit SAN SQL data: 130 GB RAID 10 SQL log: 10 GB RAID 5 TempDB: 10 GB RAID 5 Operations Manager 2007 Audit Collection database One per Operations Manager 2007 Audit Collection deployment Two dual core 8 GB Windows Server 2003 with SP1, 64 bit SAN SQL data: 1 terabyte RAID 10 SQL log: 50 GB RAID 5 TempDB Management server Two per management group One dual core 4 GB Windows Server 2003 with SP1, 64 bit Not applicable Management Server and Operations Manager 2007 Audit Collection One per Operations Manager 2007 Audit Collection deployment One dual core 4 GB Windows Server 2003 with SP1, 64 bit Not applicable Data warehouse One per the entire deployment Four dual core 16 GB Windows Server 2003 with SP1, 64 bit SAN SQL data: 400 GB RAID 10 SQL log: 30 GB RAID 10 TempDB: 30 GB RAID 10 Gateway server Two per security-enhanced environment Two dual core 4 GB Windows Server 2003 with SP1, 64 bit Not applicable

Designing the Infrastructure: Redundancy and Disaster Recovery Role Redundancy/high availability Disaster recovery Management server Approach: Deploy multiple management servers for agent failover. Description: In a management group, if one management server goes offline, the agents and gateway systems reporting to it will fail over to another management server in the management group. In Microsoft IT, agent failover behavior is defined in Active Directory, where agents can automatically discover it. Approach: Deploy management servers in multiple locations. Description: If one site goes offline, the agent will fail over to the management server in another site, assuming that the site’s failover configuration allows this. Microsoft IT is considering having management servers hosted on virtual machines in the failover location, as opposed to procuring more dedicated hardware. Root management server Approach: Cluster the root management server. Description: By using Windows Clustering, Microsoft IT has deployed the root management server role onto a two-node cluster to allow the underlying servers to go offline without having to take the role itself offline for a prolonged amount of time. Approach: Promote a remote management server to root management server. Description: If the root manager server cluster becomes unavailable, Microsoft IT will promote one of its remaining management servers into the root management server role. Management servers are deployed in remote locations to account for a site-wide outage. Gateway server Approach: Deploy multiple gateway servers per location. Description: The approach is the same as management servers, with two notable exceptions. First, multiple gateway servers need to be deployed in each distinct network/Active Directory space where the gateway's agents are located. Second, gateway servers need to have multiple management servers that they can communicate with to ensure that they themselves can fail over if necessary. None: Because the gateway servers are located in the same physical segment as the agents, if the location is offline, the agents will be offline as well. Operational database server Approach: Cluster the SQL Server–based server that the Operational database is hosted on. Description: By using Windows Clustering, Microsoft IT has deployed the SQL Server–based server that the Operational database role is running on onto a two-node cluster to allow the underlying servers to go offline without having to take the role itself offline for a prolonged amount of time. Approach: Implement log shipping to a remote SQL Server–based server. Description: A stand-alone SQL Server–based server has been deployed in a remote location and log shipping was set up to replicate data from the primary Operational database to the failover SQL Server–based server. If a disaster occurs, the management group can be manually pointed to use the failover database. Operations Manager 2007 Audit Collection collector None.None. Operations Manager 2007 Audit Collection database None.None. Data warehouse None. Approach: Use database backups. Description: Scheduled periodic database backups with off-site storage are implemented. If a disaster occurs, the latest backup sequence is restored to a new server.

Deploying Operations Manager Conducting lab validation 2. Deploying the infrastructure 3. Deploying the data warehouse and reporting servers 4. Deploying management and gateway servers 5. Deploying Audit Collection components 6. Running workflows to create necessary objects in Active Directory for agent deployment 7. Deploying agents 8. Deploying management packs 9. Deploying the UIs 10. Deploying additional integration

Conducting Lab Validation ● Goals of lab validation: ● Perform rudimentary tests against the application ● Verify that the server and client components would not adversely consume resources ● Verify that the build was suitable for production deployment by observing resource utilization ● Deploy basic management packs and confirm no adverse effects on resource utilization

Deploying the Infrastructure ● Procure hardware ● Implement necessary ACLs and firewall policies ● Deploy Operational database and root management server cluster with three management groups: ● Quorum resource group ● SQL Server resource group ● Root management server resource group

Deploying the Data Warehouse and Reporting Servers ● To prepare for warehouse installation, Microsoft IT: ● Installed and configured prerequisite software ● Determined the service account to be used for the Write Action and Data Reader accounts ● Configured service account security ● Obtained information for the installation ● Set up ACLs ● Ran the prerequisite checker

Deploying Management and Gateway Servers ● Microsoft IT: ● Installed two management servers per management group and then set up gateway servers ● Requested one certificate for each server and imported them into the local certificate store ● Followed setup steps in deployment guides ● Deployed two gateway servers per network or Active Directory space

Deploying Audit Collection Components ● Operations Manager 2007 Audit Collection provides security-enhanced, real-time export of security events ● Server roles affect how many servers can be monitored on a single collector ● Current scope of this deployment of Audit Collection is 135 domain controllers and 3,500 member servers ● Event filtering method affects storage requirements and data retention

Deploying Audit Collection Components ● Data retention goal is 30 days ● Deployment method for Audit Collection is SMS ● Microsoft IT had to consider its audit policies and plan for scalability ● Audit Collection includes performance monitors and AdtAdmin.exe command-line tool ● DNS service records point agents to collectors

Running Workflows to Create Necessary Objects in Active Directory ● To provision the necessary Active Directory containers, Microsoft IT: ● Gathered a list of servers that needed agents ● Gathered a list of domains where agents would reside ● Mapped domain names to management groups ● Ran MOMAdAdmin.exe to provision containers ● Created agent assignment rules and executed workflows

Deploying Agents ● Earlier infrastructure relied on Operations Manger 2005 for push installations ● Microsoft IT now deploys agents through SMS advertisement ● Package was created to contain agent installation files ● Systems rights let agents enter management groups via Active Directory

Deploying Management Packs ● Microsoft IT took action based on categorization of management packs: ● Identify and convert existing management packs ● Identify and disable existing management packs not to be deployed ● Deploy new Operations Manager 2007 management packs

Deploying the User Interfaces ● Customers of the monitoring service can use the operations console to work with operational data ● notifications can be generated in response to alerts ● Web console contains trimmed-down functionality as an alternative to the operations console ● Connector converts alerts to trouble tickets

Deploying Additional Integration ● Microsoft partners supply connectors that integrate their applications with Operations Manager ● Microsoft IT uses EMC Smarts to monitor network devices ● Microsoft IT wrote a custom connector to integrate Operations Manager with trouble ticketing ● Users of the operations console can use ticketing functionality via customer resolution states

Deploying Additional Integration ● Microsoft IT used the SDK to automate administrative tasks ● Custom management pack uses the SDK to extend the base system class in Operations Manager 2007 ● Custom management pack placed a service offering attribute on every new base system class

Best Practices ● Gather feedback from end users in advance ● Consider new server roles and features ● Consider the number of systems to be monitored and the number of rules that will be run ● Consider the performance capabilities of the servers that will host the Operational database ● Use the configuration advice in Help files and in System Center Capacity Planner

Best Practices ● In a migration from Operations Manager 2005: ● Evaluate which management packs can be carried forward ● Run the two infrastructures in tandem during the transition ● Deploy management servers and gateway servers in pairs ● Use SMS to distribute agents ● Use Active Directory integration ● Deploy management packs in batches

Conclusion ● IT teams and product groups at Microsoft work together to develop viable enterprise technology ● Deploying Operations Manager 2007 enabled Microsoft IT to: ● Improve its service offerings and give feedback ● Provide end-to-end service management in a scalable, centralized monitoring infrastructure ● Streamline how it identifies and resolves issues ● Take advantage of work invested in deploying previous monitoring solutions

For More Information ● Operations Manager 2007 product pages: systemcenter/opsmgr/default.mspx systemcenter/opsmgr/default.mspx systemcenter/opsmgr/default.mspx ● Server roles in Operations Manager 2007: ● Planning and deploying a monitoring infrastructure: systemcenter/sccp/default.mspx systemcenter/sccp/default.mspxhttp:// systemcenter/sccp/default.mspx ● Performance and maintenance implications of log shipping:

For More Information ● Additional content on Microsoft IT deployments and best practices can be found on ● Microsoft IT Showcase Webcasts ebcasts ebcasts ebcasts ● Microsoft TechNet

This document is provided for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT. © 2008 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS SUMMARY. Microsoft, Active Directory, SQL Server, Windows, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. The names of actual companies and products mentioned herein may be the trademarks of their respective owners.