1. Andreas Grabner Your place in DevTOps is not about finding more bugs, but problem solutions

2. That’s why I ended up talking at BTD

6. The stuff we did when we were a Start Up and we All were Dev s, T esters and Ops

7. Release Acceptance Testing Unit Testing Performance Testing Performance Testing Coding Reduce Lead Time Improve Process

8. 700 Deployments / Year 50-60 Deployments / Day 10+ Deployments / Day Every 11.6 seconds

9. Inside the Amazon Numbers! 75% fewer outages since 2006 90% fewer outage minutes ~0.001% of deployments cause a problem Instantaneous automatic rollback Deploying every 11.6s

10. Deploy Faster!!

11. Fail Faster!!?

12. Failures happen!! Nobody likes it when …

13. Unless you work for Google or Microsoft

14. … or this …

16. .. as it leads to this …

17. The “War Room” Facebook – December 2012 The “War Room” Facebook – December 2012

18. … and potentially to this …

20. And this isn’t helping either …

25. Tester Developer

28. This ATTITUDE is the main PROBLEM as it LEADS to …

29. 80% $60B

30. Release Acceptance Testing Unit Testing Performance Testing Performance Testing Coding % of Bugs in Software Relative Cost of a Bugfix 100% 80% 50% 40% 1x 10x 50x 150x 25x Barry Boehm: “EQUITY Keynote Address”, March 19 th, 2007 Insufficient Focus on Quality

32. 20% 80%

33. Release Acceptance Testing Unit Testing Performance Testing Performance Testing Coding % of Bugs in Software Relative Cost of a Bugfix 40% 50% 40% 1x 10x 50x 150x 25x Barry Boehm’s numbers adjusted by Andreas Grabner 80% 20% 10% 5% 100% 80% Focus on “Well Known” problems! Automated!

35. C ollaboration A utomation S haring M easuring 4 Pillars of DevOps + Quality Focus

36. Level-Up Skills Browser Diagnostics Wireshark, Fiddler JBoss, Tomcat, JConsole IIS, ASP.NET, PerfLib Oracle, SQL Garbage Collection Thread Dumps

37. Architecture # of Services # of Databases # of Servers # of Calls between Components

38. Performance Page Load Time Render Time SQL Query Time Service Call Time

39. Scalability Memory Usage per User # Connections Cache Utilization Load Distribution Component Roundtrips

40. C ollaboration A utomation S haring M easuring L evel-Up Skills P erformance S calability A rchitecture 4 Pillars of DevOps + Quality Focus

41. 4 Use Cases: WHY, HOW & METRICS 4 Use Cases: WHY, HOW & METRICS

43. Bloated Websites

44. Mobile Web Site: This SHOULDN’T happen! 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB

45. Not following Web Performance Best Practices 282! Objects on that page 9.68MB Page Size 8.8s Page Load Time Most objects are images delivered from your main domain Very long Connect time (1.8s) to your CDN

46. Developers not using the browser built-in diagnostics tools Testers not doing a sanity checks with the same tools Some tools for you (Quick Demo) Built-in Inspectors via Ctrl-Shift-I in Chrome and Firefox YSlow, PageSpeed, SpeedTracer Dynatrace Level-Up: Automate Testing and Diagnostics Check Lessons Learned – NO Excuse for …

47. # Resources # of Domains Usage of CDNs Page Load and Size

49. Not every Architect makes good decisions

50. Symptoms HTML takes between 60 and 120s to render High GC Time Developer Assumptions Bad GC Tuning Probably bad Database Performance as rendering was simple Result: 2 Years of Finger pointing between Dev and DBA Project: Online Room Reservation System

51. Developers built own monitoring void roomreservationReport(int officeId) { long startTime = System.currentTimeMillis(); Object data = loadDataForOffice(officeId); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, officeId); } Result: Avg. Data Load Time: 45s! DB Tool says: Avg. SQL Query: <1ms!

52. #1: Loading too much data 24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory

53. #2: On individual connections 12444! individual connections Classical N+1 Query Problem Individual SQL really <1ms

54. #3: Putting all data in temp Hashtable Lots of time spent in Hashtable.get Called from their Entity Objects

55. … you know what code is doing Challenge the developers Explore Tools that “might seem” out of your league! Built-In Database Analysis Tools “Logging” options of Frameworks such as Hibernate, … JMX, Perf Counters, … of your Application Servers Performance Tracing Tools: Dynatrace, NewRelic, AppDynamics, … Lessons Learned – Don’t Assume …

56. # SQL Executions # of SAME SQLs Conn. Acquisition Time

58. “Deployment” Gone Bad!

59. Test Environment Production Environment 8x slower 3x more SQL

60. Test Environment Production Environment Hibernate, Classloading, XML – The Key Hotspots I/O for Web Requests doesn’t even show up! Hibernate, Classloading, XML – The Key Hotspots I/O for Web Requests doesn’t even show up! That’s Normal: Having I/O for Web Request as main contributor

61. Top Contributor Class.getInterfaces Called from Hibernates FieldInterceptionHelper These calls all originate form thousands of calls to find item by code

62. Top Methods related to XML Processing Classloading is triggered through CustomMonnkey and the Xalan Parser Classloading is triggered through CustomMonkey and the Xalan Parser

63. Plan enough time for proper testing Anticipate changed user behavior during peak load Only test what really ends up in Production Lessons Learned

64. Time Spent in API # Calls to API

66. Incorrect Sizing of Pools and Queues

67. Online Banking: Slow Balance Check 1.69m (=101s!) To Check Balance! 87% spent in IIS 600! SQL Executions

68. #1 Time really spent in IIS? Tip: Elapsed Time tells us WHEN a Method was executed! Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET Tip: Thread# gives us insight on Thread Queues / Switches

69. #2 What about these SQL Executions? Finding: EVERY SQL statement is executed on ITS OWN Connection! Tip: Look at “GetConnection”

70. #2 SQL Executions! continued … #1: Same SQL is executed 67! times #2: NO PREPARATION because everything executed on new Connection

71. Lessons Learned! ASP.NET Worker Thread Pool Sizing! DB Connection Pools More Efficient SQL DB Connection Pools More Efficient SQL

72. How to Monitor: Web Server Idle vs. Busy Threads

73. How to Monitor: Application Server App Server Threads

74. Idle vs. Busy Threads # SQLs / Request # GetConnection

75. http://blog.dynatrace.com http://www.perfplanet.com/ http://highscalability.com/ http://blog.ruxit.com/ More of these Use Cases?

77. # Images # Redirects Size of Resources # SQL Executions # of SAME SQLs # Items per Page # AJAX per Page Remember: New Metrics When Testing Apps Time Spent in API # Calls into API # Functional Errors 3rd Party calls # of Domains Total Size …

79. Putting it into Test Automation 120120ms 3168ms Build 20testPurchaseOK testSearchOK Build 17testPurchaseOK testSearchOK Build 18testPurchaseFAILED testSearchOK Build 19testPurchaseOK testSearchOK Build #Test CaseStatus # SQL# ExcepCPU 120120ms 3168ms 12560ms 3168ms 750230ms 3168ms Test Framework Results Architectural Data We identified a regresesion Problem solved Exceptions probably reason for failed tests Problem fixed but now we have an architectural regression Now we have the functional and architectural confidence Let’s look behind the scenes

80. #1: Analyzing each Test #2: Metrics for each Test #3: Detecting Regression based on Measure

81. #1: Test Status Overview based on our new Metrics #2: Lets the build fail

82. Release Acceptance Testing Unit Testing Performance Testing Performance Testing Monitor Tests Analyze Results Quality Gate in your Build Tool    Every 11.6 seconds

83. Deploy Faster!! With Better Quality

84. Andreas Grabner Your place in DevTOps is not about finding more bugs, but problem solutions Slides: slideshare.net/grabnerandi Get Tools: bit.ly/dttrial YouTube Tutorials: bit.ly/dttutorials Contact Me: agrabner@dynatrace.com Follow Me: @grabnerandi Read More: blog.dynatrace.com