Microsoft CSS ADPerf Core Team and Justin Turner Active Directory Performance Troubleshooting Microsoft CSS ADPerf Core Team and Justin Turner INF341

Overview / Agenda Symptoms, Cause and Resolution of AD Performance issues Troubleshooting workflow “Peeling the Onion” Common Scenario review Preventative Measures and References

Symptoms, Cause and Resolution Microsoft Ignite 2015 11/23/2018 11:37 AM Symptoms, Cause and Resolution A high level overview before we go deeper © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Symptoms DC side symptoms Client side symptoms Microsoft Ignite 2015 11/23/2018 11:37 AM Symptoms DC side symptoms Primarily: High LSASS CPU Utilization (could be high memory) Client side symptoms Timeouts, application failures due to slow / no DC response Slow LDAP(S) bind Repeated prompts for credentials © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Cause Excessive or Inefficient Workloads Other Bottleneck conditions Microsoft Ignite 2015 11/23/2018 11:37 AM Cause Excessive or Inefficient Workloads LDAP; SAM; LSA; Change notification (LDAP); Other Bottleneck conditions MaxConcurrentAPI (MCA); Null domain Auth; Null domain lookups © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Resolution Client DC Client DC Reduce Optimize Redistribute Microsoft Ignite 2015 11/23/2018 11:37 AM Reduce Modify app/script Implement a cache Apply updates and rollups Optimize Index Modify config. Client DC Resolution Client DC Redistribute AD Site configuration Increase Capacity More DCs, CPUs etc. © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Troubleshooting Workflow “Peeling the Onion” DC-Side Data Collection AD Diagnostics Data Collector Set (With DC workload review) 1644 Event Logging Client-Side Data Collection

Troubleshooting process Microsoft Ignite 2015 11/23/2018 11:37 AM Reduce, Optimize, Distribute, Increase Capacity Client and DC-side methods If it hurts, stop doing it. Client-Side Data Collection Identify application, process, or script Tasklist, netstat, network trace, process monitor, etc. DC-Side Data Collection Map workload to client-side caller AD Diagnostics, 1644 Events, network trace, etc. © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

DC-Side Data Collection Microsoft Ignite 2015 11/23/2018 11:37 AM DC-Side Data Collection AD Diagnostic Data Collector Set (SPA) 1644 Event Logging – Tracking Inefficient / Expensive Queries Network trace, netstat –anob, tasklist /svc, Netlogon.log © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Demo: AD Diagnostics Data Collector Set Microsoft Ignite 2015 11/23/2018 11:37 AM Demo: AD Diagnostics Data Collector Set © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

DC Workload Directory operations AD Replication Search and Bind Microsoft Ignite 2015 11/23/2018 11:37 AM Directory operations AD Replication Search and Bind Kerberos ticket operations DSCrackNames, Account operations Sid2Name and Name2Sid NTLM operations © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Microsoft Ignite 2015 11/23/2018 11:37 AM 1644 Event Logging Enable logging of expensive and inefficient searches in Event ID 1644 HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics\15 Field Engineering Set to a value of 0x5 to log one event per LDAP search that exceeds the threshold (Increase the size of the Directory Service Event Log) Thresholds If no threshold value is specified (registry value not set) then the following values are applied: Data analysis is difficult when looking at individual events. Event 1644 script available from TechNet Script Gallery Extracts 1644 events into Excel with pivot tables to make analysis easier Registry Path Data Type Default value OS Comment HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Expensive Search Results Threshold DWORD 10,000 ALL Using the default values, a search is considered expensive if it visits more than 10,000 entries HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Inefficient Search Results Threshold 1,000 A search is considered inefficient if the search visits more than 1,000 entries and the returned entries are less than 10 percent of the entries that it visited. HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Search Time Threshold (msecs) 30,000 Server 2012 R2 or later or MSKB 2800945 is installed (Server 2008, Server 2008 R2, Server 2012) Event is logged if search exceeds 30,000 milliseconds (30 seconds) – probably a bit too long for a threshold © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Common Issues LDAP Search vs ATQ Threads SAM calls Microsoft Ignite 2015 11/23/2018 11:37 AM Common Issues LDAP Search vs ATQ Threads SAM calls Null Domain Auth / Null Domain Lookups How long queued requests take to be serviced Requests are being queued Max ATQ Threads is equal to ATQ Threads Total © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Data Analysis Demos: High CPU Utilization LSASS Scenario 1 and Scenario 2

Preventative Measures DC and Client-side Updates / Rollups DC Sizing / Capacity General Configuration/Optimization Guidance

Updates to install: Recommendations for Clients Microsoft Ignite 2015 11/23/2018 11:37 AM Updates to install: Recommendations for DC role computers Update for 1644 Event log details LDAP Query Optimizer Update MS15-096 / KB 3072595 LSASS Memory Usage – Windows Server 2012 R2 only (Due 12/15) Recommendations for Clients Install relevant updates / rollups SBSL rollup for Windows 7 and Windows Server 2012 © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

DC Sizing / Capacity 64-bit OS only Microsoft Ignite 2015 11/23/2018 11:37 AM DC Sizing / Capacity 64-bit OS only Sufficient memory to cache NTDS.DIT (even more with Windows Server 2012 R2) DIT + log files deployed to different drive than OS with sufficient spindles to support I/O volume generated by environment See capacity planning document Sufficient cores to handle load Proximity to load (location of servers/clients- to workload) –ie. in-site DCs Sufficient boxes to handle availability / redundancy © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Configuration / Optimization Identify and optimize inefficient LDAP queries Find and consider disabling null domain lookup behavior Have a well-defined AD Topology –clients map to AD sites Increase MCA API settings

Additional Resources Creating More Efficient Microsoft Active Directory-Enabled Applications LDAP Query Optimizer changes 1644 Event Improvements AD Data Collector Sets ATQ Performance Counters How to Find Expensive Inefficient Queries using 1644 and script LDAP

Summary AD Diagnostic Data Collector set and 1644 event logging Map DC workload to client-side caller Get client-side data to identify culprit Reduce, Optimize, Redistribute or Increase Capacity

A. Connor, Ming Chen, Ken Brumfield, Herbert Mauerer, Wayne McIntyre Contributors A. Connor, Ming Chen, Ken Brumfield, Herbert Mauerer, Wayne McIntyre

Complete your session evaluation on My Ignite for your chance to win one of many daily prizes.

Continue your Ignite learning path Microsoft Ignite 2015 11/23/2018 11:37 AM Continue your Ignite learning path Visit Microsoft Virtual Academy for free online training visit https://www.microsoftvirtualacademy.com Visit Channel 9 to access a wide range of Microsoft training and event recordings https://channel9.msdn.com/ Head to the TechNet Eval Centre to download trials of the latest Microsoft products http://Microsoft.com/en-us/evalcenter/ © 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.