1. SQL BEST PRACTICES OPTIMIZING AX PERFORMANCE Mark Blevins Landscape Structures ERP Systems & Data Services Manager

2. Who am I?  ERP Systems / Data Services Manager for Landscape Structures, an employee owned commercial playground and water play company with locations in Delano and Edina, MN and Dallas Texas.  Oversees the AX application architecture, custom solution development and overall system performance.  Oversees the SQL technical architecture, database administration best practices, SQL/Reporting security, BI usability, and overall SQL DB performance.

3. Landscape Structures, Delano MN

4. Landscape Structures, Inc.  Our company headquarters and manufacturing facility is located in Delano, MN, approximately 25 miles west of Minneapolis.  We are the playground experts.  With more than 40 years of experience, we create the highest-quality, safest and most innovative playground products available.  Landscape Structures playgrounds last longer, look better, stay safer and thrill more kids, year after year.

5. Central Park of Maple Grove

6. Who am I?  15 years’ experience with AX working with versions 2.1 through 2012R3  13 years’ experience working with SQL server 2000 through SQL 2014.  Have implemented AX in the US, Canada, Mexico, China and Thailand.  Typically worked with companies in the 100m to 200m range.  Have performance tuned SQL 2005 through 2014. Linked In: www.linkedin.com/in/xppdevwww.linkedin.com/in/xppdev Email: markblevins@playlsi.commarkblevins@playlsi.com

7. Disclaimer  All information and settings used have been compiled from multiple sources.  Although these setting work for most installations, keep in mind each environment is different and you may not see the same results.  All setting changes should be promoted through development and test environments prior to placing into production.  Use at your own risk!

8. Agenda  Documentation, paramount to best practices  SQL Edition - Which one is best?  SQL Hardware - foundation to performance  Windows Server Settings  SQL Server Settings  Trace Flags, Temp DB Configuration, Database configuration, Database Maintenance, Alerts  Other contributing factors

9. Best Practice  A procedure that has been shown by research and experience to produce optimal results and that is established or proposed as a standard suitable for widespread adoption.  Great start, but fine tuning for each environment requires an iterative approach. http://www.merriam-webster.com/dictionary/best%20practice ResearchTestDocument

10. Documentation, paramount to best practices  SQL server best practice should include:  SQL version and edition of choice  Hardware specifications  Any specific bios setting requirements  Operating system specifications  Operation system settings specific to SQL server installation

11. Documentation, paramount to best practices  SQL server best practice should include:  SQL installation settings  SQL trace flag settings  SQL DB settings  SQL agent settings  Database maintenance plans  Responsibility matrix

12. SQL Edition- Which one is best for you?  Determine the edition BEFORE you spec out your server  Standard: Core or server + Cal  Business Intelligence: Server + Cal  Enterprise Edition: Core Only  Resources for feature Comparison  https://msdn.microsoft.com/en-us/library/cc645993(v=sql.120).aspx https://msdn.microsoft.com/en-us/library/cc645993(v=sql.120).aspx  https://www.microsoft.com/en-us/server-cloud/products/sql-server-editions/overview.aspx https://www.microsoft.com/en-us/server-cloud/products/sql-server-editions/overview.aspx

13. SQL Edition Recommendation  SQL Enterprise Edition:  Only limited by the OS for cores and memory  Best solution for High Availability (Always on)  Provides online indexing capability  SSRS subscription based reports  Be careful when purchasing hardware!  Per core licensing expense  Reduce number of cores, increase memory

14. SQL Hardware - foundation to performance  Based on Enterprise Edition  Pick a lower number of cores with the highest MHz processing power  Fill up the memory  Use SSD drives  Split drives  C:Operating System  D: Installed software  M: Database file storage  L: Database Log file storage  T: Temp DB’s  Use RAID 10 if possible.

15. Windows Server Settings  Lock Pages in Memory  LOCK PAGES IN MEMORY should be enabled on any SQL server that host other services. Implementing on a server that is a dedicated SQL server will produce minimal results as nothing else is competing for the memory usage.  https://msdn.microsoft.com/en-us/library/ms190730.aspx https://msdn.microsoft.com/en-us/library/ms190730.aspx  Instant File Initialization  In SQL Server, data files can be initialized instantaneously. Instant file initialization reclaims used disk space without filling that space with zeros. Instead, disk content is overwritten as new data is written to the files. Log files cannot be initialized instantaneously.  https://msdn.microsoft.com/en-us/library/ms175935.aspx https://msdn.microsoft.com/en-us/library/ms175935.aspx

16. SQL Server Settings- Max Memory  Under the default setting, which is to use all available memory, SQL Server may leave no more than 128MB of physical memory free, and may not release memory fast enough if needed by the Operating System.  This can cause the Operating System to page heavily to disk which can significantly decrease overall system performance Max Server Memory = ([Total Server Memory] - ([OS/Apps Memory] + ([Threads * Thread Size]))) [OS/Apps Memory] = 2GB (if RAM 16GB) Threads = SELECT MAX_WORKERS_COUNT FROM SYS.DM_OS_SYS_INFO Thread Size = 2MB for x64 systems or 512k for 32-bit systems Ex: Server memory 96gb, 64bit sql with 704 max threads. Max memory should be set to 90GB or (96-(4GB + (704 * 2mb))) https://msdn.microsoft.com/en-us/library/ms178067.aspx

17. SQL Server Settings-Max Degree of Parallelism  MAX DEGREE OF PARALLELISM should be set to 1.  AX does not make use of multi-threaded calls to the database and having more than one can slow performance. http://support.microsoft.com/kb/2806535/en-us

18. SQL Server Settings-Max Degree of Parallelism  Resource Governor can be utilized if setting MAX DOP = 1 is not possible on your environment  Available from SQL 2008 R2 and above.  Resource Governor enables you to manage SQL Server workloads and resources by specifying limits on resource consumption by incoming requests  Uses a traffic classifier to determine which resource group to utilize https://msdn.microsoft.com/en- us/library/bb933866.aspx

19. SQL Server Settings-Max Degree of Parallelism AXSSRS Classifier Function Resource Pool Internal Resource Pool Default Resource Pool AX MAXDOP = 1 Other

20. SQL Server Settings-Max Degree of Parallelism  Classifier function should be short and sweet. CREATE FUNCTION [dbo].[UDFClassifier]() RETURNS SYSNAME WITH SCHEMABINDING AS BEGIN DECLARE @WorkloadGroup AS SYSNAME IF(APP_NAME() = 'Microsoft Dynamics Ax') SET @WorkloadGroup = 'AX_Group' ELSE SET @WorkloadGroup = 'default' RETURN @WorkloadGroup END  Classifier is a Scalar Function on the Master Database. https://msdn.microsoft.com/en- us/library/bb933866.aspx

21. SQL Server Settings-Max Degree of Parallelism

22. SQL Server Settings-Optimize for ad hoc workloads  The optimize for ad hoc workloads option is used to improve the efficiency of the plan cache for workloads that contain many single use ad hoc queries.  When this option is enabled SQL stores a small compiled plan stub in the plan cache when a query is compiled for the first time, instead of the full compiled plan.  The compiled plan stub allows SQL to recognize that this is previous ad hoc query and to store the full plan on the second run.  This helps to relieve memory pressure by not allowing the plan cache to become filled with compiled plans that are not reused.

23. SQL Server Settings-Optimize for ad hoc workloads  A Restart of SQL is advised, but you can run DBCC FREEPROCCACHE to clear all previously saved plans. https://msdn.microsoft.com/en-us/library/cc645587.aspx  Ad hoc setting info: http://www.sqlskills.com/blogs/kimberly/plan-cache-and-optimizing-for- adhoc-workloads/  Script for easily identifying if you need to optimize for ad hoc workloads http://dba.stackexchange.com/questions/35500/why-would-i-not-use-the- sql-server-option-optimize-for-ad-hoc-workloads

24. SQL Server Settings-Optimize for ad hoc workloads

25. select usecounts,cacheobjtype,objtype,size_in_bytes,[text] from sys.dm_exec_cached_plans cross apply sys.dm_exec_sql_text(plan_handle)

26. SQL Server Settings-Trace Flags  Flags that turn on or off a specific set of behaviors for SQL server  List of most of the trace flags can be found at https://msdn.microsoft.com/en- us/library/ms188396.aspx https://msdn.microsoft.com/en- us/library/ms188396.aspx  DBCC Commands for trace flags  DBCC TRACEON(1234,-1)  DBCC TRACEOFF(1234,-1)  DBCC TRACESTATUS() – list of all flags turned on.  Must put in SQL Configuration startup parameters for restarts  Availability can vary by SQL Edition.

27. SQL Server Settings-Trace Flags, TempDB Trace FlagDescription 1117 Forces SQL Server to Auto-Grow all data files at the same time for a database. Reason: Inconsistent growth of multiple data files can cause performance issues. 1118 Removes most single page allocations on the server, reducing contention on the SGAM page. When a new object is created, by default, the first eight pages are allocated from different extents (mixed extents). Afterwards, when more pages are needed, those are allocated from that same extent (uniform extent). The SGAM page is used to track these mixed extents, so can quickly become a bottleneck when numerous mixed page allocations are occurring. This trace flag allocates all eight pages from the same extent when creating new objects, minimizing the need to scan the SGAM page. Note! Beginning with SQL Server 2016 Release Candidate (RC3) this behavior is controlled by the SET MIXED_PAGE_ALLOCATION option of ALTER DATABASE, and trace flag 1118 has no affect. For more information, see ALTER DATABASE SET Options (Transact-SQL). Reason: Reduced contention in the temp database. SGAM = Shared Global Allocation Map. http://www.sqlskills.com/blogs/paul/inside-the-storage-engine-gam-sgam-pfs-and-other-allocation-maps/

28. SQL Server Settings-Trace Flags, Dead locks Trace FlagDescription 1205 Sends detailed information about deadlocks to the error log. Reason: When deadlock occur, this setting will provide better information for tracking the problem. 1222 Returns the resources and types of locks that are participating in a deadlock and also the current command affected, in an XML format that does not comply with any XSD schema. Reason: Provides more details for diagnostics. 1224 Disables lock escalation based on the number of locks. Reason: Ax can create numerous locks when processing transactions. By reducing the escalation, less data will be locked during these updates. 7646 This Trace Flag is only relevant if you are using Full Text Indexing in Dynamics AX. The Trace Flag avoids blocking when using full text indexing. Reason: Reduction of blocking during full text indexing.

29. SQL Server Settings-Trace Flags Statistics/Queries Trace FlagDescription 2371 Changes the behavior of when "Auto Update Statistics" is triggered for an object. With this Trace Flag is enabled the more rows an object has the smaller the percent change in data triggers an Automatic Statistics update. Reason: Ax has a few tables that contain millions of records. This switch will update the statistics faster in order to provide better query plan optimization. 4199 Enables all the all non-default Query Processor Enhancements fixes that were previously made under many trace flags. http://support.microsoft.com/kb/974006 Reason: Microsoft has put in place numerous improvements to the query engine. To take advantage of these features, they must be turned on. 4136 Disables parameter sniffing. For more information, see Microsoft Support article. To accomplish this at the database level, see ALTER DATABASE SCOPED CONFIGURATION (Transact-SQL). To accomplish this at the query level, add the OPTIMIZE FOR UNKNOWNquery hint. *NO LONGER RECOMMENDED FOR AX 2012

30. SQL Server Settings-Trace Flags  Can verify trace flags with DBCC TRACESTATUS()

31. Temp DB Configuration  Split into multiple files  Placed on a separate mirrored pair or raid 10 drive is possible  Should properly size the temp DB to prevent auto growth  Number of files is dependent on the number of cores  Splitting of the files can be accomplished through SQL Management Studio, or through a script. Number of CoresNumber of Temp Files. <=8Equal to the # of cores > 88

32. Temp DB Configuration Ex: Server has 4 cores and the TempDB’s total size requirement is 4 GB. Four files should be created with a file size of 1024mb. ALTER DATABASE tempdb ADD FILE (NAME = tempdev2, FILENAME = 'X:\tempdb2.mdf', SIZE = 1024); ALTER DATABASE tempdb ADD FILE (NAME = tempdev3, FILENAME = 'X:\tempdb3.mdf', SIZE = 1024); ALTER DATABASE tempdb ADD FILE (NAME = tempdev4, FILENAME = 'X:\tempdb4.mdf', SIZE = 1024); GO

33. Temp DB Configuration  TempDB Auto Growth TempDB file Size DB File Growth increment TempDB Log Log File Growth increment 0 to 100mb10mb 1mb 200mb to 1000mb20mb 2mb 1000mb to 2000mb100mb 10mb 2000mb to ?1000mb200mb20mb

34. Database Configuration – Options  The following settings are the current default settings for a new database, but due to potential performance and consistency check issues, they are documented here as a best practice. OptionSetting DB OwnerSA Auto shrinkFalse Auto update statisticsTrue Auto Update Statistics AsynchronouslyFalse Page verifyCHECKSUM Recovery ModelFull

35. Database Configuration – File Sizes  Database size will vary by AX version and customizations  Sizes below are meant as a best practice starting point.  Databases should be regularly analyzed to ensure proper database size and growth settings. Database file Size DB File Growth increment Database Log Log File Growth increment AX Data database500mb32,000mb4,000mb AX Model Database100mb32,000mb4,000mb Ax Baseline10mb100mb

36. Database Configuration – File Sizes  Database log files should be manually created to the proper size.  Files should be created in 8,000mb chunks.  Log files smaller than 8,000mb should be created in one step.  Auto growth can be set to 4,000 EX: If we needed a 20gb log file, you would initially set the log file to 8000mb, then extended it to 16, then to 20 to create the appropriate amount of log files.

37. Database Configuration – File Sizes  Once the log files are created with the proper size, the log file should not be truncated unless there is an issue with the backup, or space needs to be temporarily relieved on the SQL server.  By creating log files manually, we can control the number of virtual log files, which can impact performance. http://www.sqlskills.com/blogs/kimberly/transaction-log-vlfs-too- many-or-too-few/

38. Database Configuration – File Sizes  Prevents performance issues  Quicker Inserts, updates, deletes  Reduced database startup time  Reduced restore time

39. Database Configuration – File Sizes  View log files with DBCC LogInfo (Each row is a VLF)  Should stay below 100 VLF’s if possible.

40. Database Maintenance - Backups Recovery ModelFull Database BackupTransaction log BackupMinimum Retention Full DailyEvery 30 Minutes7 Days WeeklyN/A31 Days MonthlyN/A366 Days YearlyN/A3 Years A backup/Retention schedule a should be created / maintained. Point in time restore is only as good as your last known good back plus the transaction logs since your last backup.

41. Database Maintenance – Integrity Checks  Over time the database may experience corruption in it’s database.  All databases should have integrity checks performed at a minimum of once per week.  Includes a running a full check using the DBCC CHECKDB command.  Due to the intense processing requirements of this command, the integrity check should be performed during a maintenance window.  Scripts from Ola Hallengren can simplify the process. https://ola.hallengren.com/

42. Database Maintenance – Indexing and Statistics  Microsoft recommends not performing any operation on indexes that are fragmented less than 10%.  Indexes which are between 5 and 30 percent fragmentation, a REORGANIZE should be used.  Above 30% fragmentation, the cost of an ONLINE REBUILD is actually less than that of a REORGANIZE.  ONLINE option is available only in SQL Enterprise edition and should be used whenever possible. https://msdn.microsoft.com/en-us/library/ms189858.aspx

43. SQL Alerts  Operators  Set up a generic operator  This will be the default for all SQL failed jobs and alerts.  DB Mail  Set up to email failed job information and SQL alerts to the operators email as a default. SettingValue Account NameSQL Agent Alerts DescriptionDB Mail profile to be used for SQL Alerts E-Mail address(from)SQLInstanceName@playlsi.com Display NameSQL Instance Name Reply E-mailSQLInstanceName@playlsi.com Server name(email server)smtp.playlsi.com SMTP Port25 AuthenticationAnonymous

44. SQL Alerts – Recommended Alerts AlertAction Severity 16 Error: Misc User Error Reviewed by DBA and forwarded to appropriate resource when changes are required. Severity 17 Error: Insufficient Resources Error Severity 18 Error: Non-fatal Internal Error Severity 19 Error: Fatal Error in Resource Reviewed by DBA, ticket created for tracking and resolution. Possible DB restore required. Severity 20 Error: Fatal Error in Current Process Severity 21 Error: Fatal Error in Database Process Severity 22 Error: Fatal Error: Table Integrity Suspect Severity 23 Error: Fatal Error Database Integrity Suspect Severity 24 Error: Fatal Hardware Error Severity 25 Error: Fatal Error Error 823: Write Request Failed Reviewed by DBA and forwarded to appropriate resource when changes are required. Error 824: Logical Consistency-based I/O Error Error 825: Read-Retry Required Error Deadlock

45. Other impacts to SQL performance  Inefficient code  Incorrect joins, indexes and views  AX AOS server settings  SQL Buffers  Forced Literals (Partition & DataAreaId) https://community.dynamics.com/ax/b/dynamicsaxsustainedenginee ring/archive/2014/06/30/an-important-data-access-kernel-hotfix- related-to-dataareaidliteral-and-partitionliteral-configuration-is- available-for-microsoft-dynamics-ax-2012-r2  Other Applications  Runaway reports & queries  Orphaned AX SPIDS https://blogs.msdn.microsoft.com/axinthefield/dynamics-ax-and-little- orphaned-spid-annie/

46. Other impacts to SQL performance  Bad / inefficient SQL plans  Can clear with : DBCC FREEPROCCACHE DBCC DROPCLEANBUFFERS DBCC FREESYSTEMCACHE ('ALL') DBCC FREESESSIONCACHE

47. Ideas to maintain SQL Best Practices  Create Visual Studio Project containing server settings and scripts  SQL Central Management Server with Policies installed  Central Management Server can also double as a performance data warehouse

48. Resources  AX Performance Team Blog  https://blogs.msdn.microsoft.com/axperf/ https://blogs.msdn.microsoft.com/axperf/  AX Engineering in the field  https://blogs.msdn.microsoft.com/axinthefield/ https://blogs.msdn.microsoft.com/axinthefield/  Other helpful sites  https://www.sqlskills.com/ https://www.sqlskills.com/  https://www.brentozar.com/ https://www.brentozar.com/  Finding User AX/SQL SPIDS https://blogs.msdn.microsoft.com/amitkulkarni/2011/08/10/finding- user-sessions-from-spid-in-dynamics-ax-2012/

49. QUESTIONS?