2. Microsoft CSS ADPerf Core Team and Justin Turner
Active Directory Performance Troubleshooting
Microsoft CSS ADPerf Core Team and Justin Turner
INF341

3. Overview / Agenda
Symptoms, Cause and Resolution of AD Performance issues
Troubleshooting workflow
“Peeling the Onion”
Common Scenario review
Preventative Measures and References

4. Symptoms, Cause and Resolution
Microsoft Ignite 2015
11/23/ :37 AM
Symptoms, Cause and Resolution
A high level overview before we go deeper
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. Symptoms DC side symptoms Client side symptoms
Microsoft Ignite 2015
11/23/ :37 AM
Symptoms
DC side symptoms
Primarily: High LSASS CPU Utilization (could be high memory)
Client side symptoms
Timeouts, application failures due to slow / no DC response
Slow LDAP(S) bind
Repeated prompts for credentials
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. Cause Excessive or Inefficient Workloads Other Bottleneck conditions
Microsoft Ignite 2015
11/23/ :37 AM
Cause
Excessive or Inefficient Workloads
LDAP; SAM; LSA; Change notification (LDAP);
Other Bottleneck conditions
MaxConcurrentAPI (MCA); Null domain Auth; Null domain lookups
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Resolution Client DC Client DC Reduce Optimize Redistribute
Microsoft Ignite 2015
11/23/ :37 AM
Reduce
Modify app/script
Implement a cache
Apply updates
and rollups
Optimize
Index
Modify config.
Client DC
Resolution
Client DC
Redistribute
AD Site configuration
Increase Capacity
More DCs, CPUs etc.
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

8. Troubleshooting Workflow
“Peeling the Onion”
DC-Side Data Collection
AD Diagnostics Data Collector Set (With DC workload review)
1644 Event Logging
Client-Side Data Collection

9. Troubleshooting process
Microsoft Ignite 2015
11/23/ :37 AM
Reduce, Optimize, Distribute, Increase Capacity
Client and DC-side methods
If it hurts, stop doing it.
Client-Side Data Collection
Identify application, process, or script
Tasklist, netstat, network trace, process monitor, etc.
DC-Side Data Collection
Map workload to client-side caller
AD Diagnostics, 1644 Events, network trace, etc.
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

10. DC-Side Data Collection
Microsoft Ignite 2015
11/23/ :37 AM
DC-Side Data Collection
AD Diagnostic Data Collector Set (SPA)
1644 Event Logging – Tracking Inefficient / Expensive Queries
Network trace, netstat –anob, tasklist /svc, Netlogon.log
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

11. Demo: AD Diagnostics Data Collector Set
Microsoft Ignite 2015
11/23/ :37 AM
Demo: AD Diagnostics Data Collector Set
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. DC Workload Directory operations AD Replication Search and Bind
Microsoft Ignite 2015
11/23/ :37 AM
Directory operations
AD Replication
Search and Bind
Kerberos ticket operations
DSCrackNames, Account operations
Sid2Name and Name2Sid
NTLM operations
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. Microsoft Ignite 2015
11/23/ :37 AM
1644 Event Logging
Enable logging of expensive and inefficient searches in Event ID 1644
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics\15 Field Engineering
Set to a value of 0x5 to log one event per LDAP search that exceeds the threshold (Increase the size of the Directory Service Event Log)
Thresholds
If no threshold value is specified (registry value not set) then the following values are applied:
Data analysis is difficult when looking at individual events.
Event 1644 script available from TechNet Script Gallery
Extracts 1644 events into Excel with pivot tables to make analysis easier
Registry Path
Data Type
Default value OS
Comment
HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Expensive Search Results Threshold
DWORD
10,000
ALL
Using the default values, a search is considered expensive if it visits more than 10,000 entries
HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Inefficient Search Results Threshold
1,000
A search is considered inefficient if the search visits more than 1,000 entries and the returned entries are less than 10 percent of the entries that it visited.
HKLM\SYSTEM\CurrentControlSet\Services\NTDS\Parameters\Search Time Threshold (msecs)
30,000
Server 2012 R2 or later or MSKB is installed (Server 2008, Server 2008 R2, Server 2012)
Event is logged if search exceeds 30,000 milliseconds (30 seconds) – probably a bit too long for a threshold
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Common Issues LDAP Search vs ATQ Threads SAM calls
Microsoft Ignite 2015
11/23/ :37 AM
Common Issues
LDAP Search vs ATQ Threads
SAM calls
Null Domain Auth / Null Domain Lookups
How long queued requests take to be serviced
Requests are being queued
Max ATQ Threads is equal to ATQ Threads Total
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Data Analysis Demos: High CPU Utilization LSASS
Scenario 1 and Scenario 2

16. Preventative Measures
DC and Client-side Updates / Rollups DC Sizing / Capacity General Configuration/Optimization Guidance

17. Updates to install: Recommendations for Clients
Microsoft Ignite 2015
11/23/ :37 AM
Updates to install:
Recommendations for DC role computers
Update for 1644 Event log details
LDAP Query Optimizer Update MS / KB
LSASS Memory Usage – Windows Server 2012 R2 only (Due 12/15)
Recommendations for Clients
Install relevant updates / rollups
SBSL rollup for Windows 7 and Windows Server 2012
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. DC Sizing / Capacity 64-bit OS only
Microsoft Ignite 2015
11/23/ :37 AM
DC Sizing / Capacity
64-bit OS only
Sufficient memory to cache NTDS.DIT (even more with Windows Server 2012 R2) DIT + log files deployed to different drive than OS with sufficient spindles to support I/O volume generated by environment See capacity planning document
Sufficient cores to handle load
Proximity to load (location of servers/clients- to workload) –ie. in-site DCs
Sufficient boxes to handle availability / redundancy
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. Configuration / Optimization
Identify and optimize inefficient LDAP queries
Find and consider disabling null domain lookup behavior
Have a well-defined AD Topology –clients map to AD sites
Increase MCA API settings

20. Additional Resources
Creating More Efficient Microsoft Active Directory-Enabled Applications
LDAP Query Optimizer changes
1644 Event Improvements
AD Data Collector Sets
ATQ Performance Counters
How to Find Expensive Inefficient Queries using 1644 and script
LDAP

21. Summary AD Diagnostic Data Collector set and 1644 event logging
Map DC workload to client-side caller
Get client-side data to identify culprit
Reduce, Optimize, Redistribute or Increase Capacity

22. A. Connor, Ming Chen, Ken Brumfield, Herbert Mauerer, Wayne McIntyre
Contributors
A. Connor, Ming Chen, Ken Brumfield, Herbert Mauerer, Wayne McIntyre

23. Complete your session evaluation on My Ignite for your chance to win one of many daily prizes.

24. Continue your Ignite learning path
Microsoft Ignite 2015
11/23/ :37 AM
Continue your Ignite learning path
Visit Microsoft Virtual Academy for free online training visit
Visit Channel 9 to access a wide range of Microsoft training and event recordings
Head to the TechNet Eval Centre to download trials of the latest Microsoft products
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.