1. Microsoft Exchange Server Best Practices Analyzer Tool Paul Bowden Program Manager Exchange Server Development Microsoft Corporation

2. What is it? The Exchange Sever Best Practices Analyzer 'encodes' the top product support issues into a tool which can be run against a live deployment. –Step by step documentation tells you how to resolve each problem The tool can be run as part of a proactive 'health check' which can expose availability or scalability problems. Additionally, the tool can be run as part of a reactive troubleshooting step for problem diagnosis and identification. –The tool will report issues currently causing problems within the topology, and discrepancies which may cause future outages. The tool can be used to actively document the design and configuration of the Exchange topology. This data can be used to track the history of a deployment, or provide a ‘quick-start' to administrators and product support staff who need to analyze the history and configuration of an unfamiliar deployment.

3. Why we developed it Administrators are finding it difficult to keep up with the documentation that we produce –Urgency –Relevance Customers find it difficult to keep track of whether they are conforming to all the best practices Exchange has many options and finding root cause for a problem can be a long process –~60% of Exchange problems are mis-configurations We have many tools for collecting information, but not many provide auto-analysis

4. Design Principles Concentrate on Performance, Scalability and Availability of Exchange Servers –ExBPA does not check security configuration Make it easy to run –No complex configuration settings –Auto-detect everything –Allow multiple credentials to be entered –No server-side components to install –No impact on Exchange performance, even at peak periods Don’t leave me hanging –Every Error | Warning | NonDefault rule has a specific article which tells you more about the problem and how we detected it Keep it up-to-date –Provide best practice updates every month –Make the tool auto-download the updates Work in all environments –From single server SBS implementations through to the largest enterprise –Make the tool work seamlessly in both open and closed networks

5. Similar Tools MBSA – Microsoft Baseline Security Analyzer SQLBPA – Microsoft SQL Server Best Practices Analyzer The ExBPA engine has now been mandated as part of the WSS 2006 Common Engineering Criteria –BPAs for other Microsoft products are forthcoming

6. Architecture One tool runs against all versions of Exchange –No support for pure Exchange 5.5 topologies You generally install the tool on a Windows XP workstation, and it remotely collects the data –Don’t need to install any components on the server ExBPA is written in managed code (C#) Input/output data model is XML based Analysis engine is based on XPath

7. Where do we look? We look for data in… –Active Directory –DNS –WMI –Registry –Metabase –Performance Monitor –Files on disk –TCP/IP ports First pass of execution - collection –ExBPA collects the data and places it in the same namespace Second pass of execution – analysis –Individual settings are analysed against the defined rules. Cross- checking between data sources is possible as the data is in the same hierarchy

8. How it works Active Directory Exchange Server Exchange Server Exchange Server ExBPA Dispatcher XML Rules collectors Output Data ExBPA Analyzer Import XML Export ExBPA Interface

9. Demonstration…

10. What does ExBPA check today? This following is not an exhaustive list of the checks that the tool performs, but it should give you a general idea!

11. Exchange Roles ExBPA detects and understands the difference between… –Small mailbox servers –Large mailbox servers –Clustered Exchange servers –Front-end servers –Bridgehead servers Rules are conditioned for their roles (e.g. Circular logging needs to be disabled on mailbox servers, but should be enabled on bridgehead servers)

12. Rule Types Error –We found something that is causing, or will cause a problem –Example: No maximum message size set for the organization Warning –We found something that looks suspicious –Example: An ADC connection agreement is scheduled to ‘Never’ NonDefault –We found a setting which has been changed –Example: One of the many store parameters has been tuned/tweaked Time –We found something that was changed during the past 5 days –Example: The cost on an SMTP connector was changed BestPractice –We found that a best practice is not being followed –Example: Dr. Watson crashes are not being uploaded to Microsoft for analysis Info –We found something of interest –Example: Your server has 8 processors installed

13. Active Directory Forest-wide –Forest functionality level –Exchange schema extensions –Default policy changes Per-domain –Domain functionality level –Domains which have been renamed –Check availability of FSMO servers –EDS/EES group renamed/deleted/moved –MESO container renamed/deleted/moved

14. Active Directory Connector ADC Server –Server is overloaded –Server is idle (i.e. no connection agreements) –There’s a newer version of the ADC available –Server is running the latest OS Service Pack Connection Agreements –Orphaned agreements –Schedule set to never –Nominated server is missing –One way agreements –Out-of-date agreements

15. Exchange Organization Check –Global message size limits are enforced –Stray Exchange objects in LostAndFound container –More than 10 administrators defined –ForestPrep version –Mixed/native mode –OMA/EAS options –UCE thresholds –Recipient Update Service definitions –Address List and OAB definitions

16. Admin Groups Check –Validity of legacyExchangeDN –Policy containers intact Routing Groups –Check for valid routing master –Enumerate all connectors –Check for connectors that have recently changed

17. Exchange Server object Check –Validity of server name –FQDN/NetBIOS name resolution –Latest Exchange Service Pack / Roll-up –Time synchronization with the Active Directory

18. Cluster Configuration Checks both Active and Passive nodes Cluster-specific checks –Number of nodes in the cluster –Configuration discrepancies between nodes –Cluster account TEMP/TMP path –Quorum configuration –Heartbeat configuration –DNS/WINS configuration –Enumerates all resources and parameters –Kerberos configuration

19. Directory Access Check –DSAccess cache configuration and non-default parameters. E.g. MaxMemoryUser | MaxMemoryConfig LdapKeepAliveSecs, DisableNetLogonCheck MinUserDC –DSAccess cache efficiency DSAccess topology –Round-trip times between Exchange and each DC/GC in the topology –Hardware/OS configuration of each DC/GC –Calculates the GC to Exchange processor ratio

20. Information Store Check –ESE cache configuration –Current state of virtual memory –Online maintenance window –Checkpoint depth –Circular logging state –Log buffer configuration –Log generation level –File system characteristics (NTFS/Compression/Encryption) –Validity of legacyExchangeDN –Database and logs on the same LUN –Content Indexing state –Non-default parameters in Private|Public-GUID registry –Database size –E-mail address on Public Folder stores –RPC Compression / Buffer Packing settings –Hard-coded TCP/IP ports, and clashes with other Exchange ports

21. Store Process Parameters Examples –Disable MAPI Cllients –Enable Tracing –Initial Memory Percentage –Initial Reserve Size KB –Ignore Zombie Users –Logon Only As –Mailbox Cache Age Limit –Mailbox Cache Idle Limit –Mailbox Cache Size –MaxOpenMessagesPerLogon –Reserve Increment KB –SuppressOOFsToDistributionLists –Trace User LegacyDN –VM Warning|Error Level More –objtAttachment –objtFolder –objtFolderView –objtMessage –ProrateFactor –ProrateStart –ProrateMax –IMAIL settings –ExIFS drive Check for non-default settings and bad values

22. Transport Check –Main configuration parameters in the AD –Cross-check AD and metabase for consistency –Non-default settings –File system characteristics for ‘mailroot’ folders (NTFS/Compression/Encryption) –SMTP stack verb validation (e.g. X-LINK2STATE) –SMTP mail submission test –Enumeration of transport event sinks –Enumeration of MTA settings, calling out any non-defaults –Detection of Archive Sink and configuration –Non-default routing parameters (e.g. SuppressStateChanges)

23. System Attendant Check –Service state –File system characteristics for message tracking folder (NTFS/Compression/Encryption) –RFR service –RFR / NSPI Target Server configuration –Hard-coded TCP/IP ports

24. Anti-virus Support CA eTrust 6/7 file-level AV configuration and exclusions Trend Micro ScanMail –Patch level –Performance tuning configuration (threads/thresholds/debug settings) Product detection and configuration settings for –McAfee GroupShield –Symantec Mail Security for Exchange –Sybari Antigen VS API configuration settings –Warn if number of threads is not appropriate for underlying hardware

25. Other Installed Applications Check –RPC Client|Server binding order configuration –Presence of LeakDiag –For old versions of Simpler-Webb ERM –ISA 2000 Service Pack level –Presence of MOM Agent

26. Hardware Configuration Check –System BIOS is not over a year old –Specific support for HP, Dell and IBM servers –Processor configuration –Physical memory installed

27. Disk Storage System Check –Performance counters are enabled –Enumeration of physical and logical disks –Enumeration of identification of mount points –Enumeration of disk controllers and driver levels –Configuration of Host Bus Adaptors –Version of multi-pathing software (e.g. SecurePath, PowerPath)

28. File Versions Verify 29 key Exchange binaries –Physical presence –Make sure that they’re not too old –Identify binaries which are hotfixes Check –Server MAPI subsystem –Presence of old Roll-ups –Presence of ESE API virus scanners

29. Hotfixes Detect all hotfixes and Service Packs installed for –Windows 2000 –Windows 2003 –Exchange 5.5 –Exchange 2000 –Exchange 2003 Call out any updates that were installed during the past 5 days, and the logon name of the user that performed the installation

30. Network Subsystem Enumerate all network cards Check –NIC connection status –DNS/WINS configuration –IP Gateway settings –Primary DNS is alive –Domain suffix

31. Operating System Check –Page Table Entry (PTE) levels –Paged|NonPaged pool configuration –CrashOnAuditFail configuration –HeapDeCommitFreeBlockThreshold –TEMP/TMP paths –SystemPages configuration –/3GB /USERVA configuration –Physical Address Extensions (PAE) detection –OS Version and SKU (e.g. Standard, Enterprise, etc) –Dr. Watson configuration –Debug settings (including GlobalFlag, PageHeapFlags) –Virtual PC / Virtual Server / VMWare detection

32. Success Stories Identified that circular logging was enabled on a 12,000 user Exchange cluster –Was a potential time-bomb Identified incorrect memory configuration that required the Exchange server to be restarted every two weeks Identified a case where database files were being stored on a compressed volume –Root cause of the performance problems

33. ExBPA Timeline V1.0 – September 21st –1200 point collection / 800 rules V1.1 – December 6th –Usability improvements –1300 point collection / 900 rules V2.0 – Early March –Localized in all Exchange Server languages –Performance sampling and root cause analysis infrastructure –Admin API support (e.g. find out time of last backup) –Optional integration with MOM 2005 –Export to XML / HTM / CSV –New baseline logic V3.0 – Later on in the year –More rules and refinements –MAPI.NET collector

34. Appendix: Screen Shots