1. + Continuous Quality: What 21 st -Century IT Means for QA Jeff Sussna, Ingineering.IT October 18, 2013 Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

2. Preface Changing business drives changing IT drives changing QA Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

3. Me Career:  Jobs spanning Development, QA, and Operations Practice:  IT Service Innovation consultant  Clients: SaaS companies and enterprise IT organizations  Help Dev/QA/Ops work more closely with each other  Help them better understand each other and their customers

4. Chapter One 21 st -Century IT Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

5. Post-Industrial Society 1973: Daniel Bell, ‘The Coming of Post-Industrial Society’  Knowledge/creativity-based work  Service-based economy  Pervasive computerization Just now coming to fruition

6. The Shift to Post-Industrialism From products to services From siloed to infused lives From complicated to complex systems From efficient to adaptive companies

7. From Products to Services Creation: from 'things' to experiences  Journey = touchpoints over time Value: from delivery to co-creation Sales: from events to relationships Marketing: from convincing to understanding

8. From Silos to Infusion Is the coffee shop for chatting or working? Is your phone for calls, or photos, or email? Is the library for books or Internet access? Is your car for driving or listening to online music?

9. From Complicated to Complex Is it Honda’s or Pandora’s fault if your car isn’t working? Do you work for yourself or your client or your broker? Do your customers complain to support or on Twitter? Is your invoicing data managed by Freshbooks or Heroku or Amazon?

10. Complicated One thing  Many parts  Hierarchy  Clear/contained relationships Predictable/controllable repercussions

11. Complicated Automobile Data Center Corporate structure

12. Complex Many (semi-)independent things Many interactions between them Fluid relationships

13. Complex Flock of birds Economy Cloud computing Corporate reality

14. Characteristics of Complexity Emergence Cascading failures Sensitivity to history/initial conditions Failure-prone yet resilient

15. Implications of Complexity Can't model/predict/control Control can make things worse System failure without component failure

16. Enterprise Complexity Outsourcing Social business Cloud providers API ecosystems BYOD/CoIT

17. Facing Complexity From fail-safe to safe-to-fail From reductionism to systems thinking From hierarchies to networks From tight coupling to loose coupling

18. From Efficient to Adaptable Kodak lasted 100 years before being disrupted Microsoft lasted 30 years Apple went from most world’s valuable company to ? in 1 year AWS accelerating away from competitors

19. Changing Business Imperatives From efficiency/scale/stability…  …to agility/nimbleness/adaptability From static discontinuous broadcast… ...to dynamic continuous conversation From complicatedness to complexity

20. Adaptive Business Frameworks Cybernetics: Norbert Wiener OODA Loops: John Boyd

21. Cybernetics Ancient Greek: steersman, governor, pilot Adaptive loops:  Gun targeting  Walking Feedback: adjust future conduct by past performance

22. OODA Loops Mental model for airborne battle:  Observe  Orient  Decide  Act Win by making your OODA Loop tighter than your enemy’s

23. Implications for IT IT must undergo the same transformations:  Products -> services  Silos -> Infusion  Complicated -> Complex  Efficient -> Adaptive Digital infusion raises the stakes

24. IT is Business-Critical Old: inventory reports improve efficiency New:  Biz can't function w/out IT  Can't converse w/out IT  Can't optimize OODA Loop w/out IT

25. Business and IT are Inseparable Beyond “alignment” Transformation impacts how IT operates…  … and how IT-enabled business operates

26. 21st-Century IT Resilience over stability MTTR over MTBF Loose coupling over centralized coordination Diversity over monoculture Propagation over mandated adoption Customer outcomes over offerings

27. 21st-Century Development Agile: cybernetics not project management Cloud: don't slow down the OODA Loop Continuous Delivery: let business speed throttle IT speed

28. 21st-Century Operations Design-for-fail: assume and allow underlying failure Game days: anticipate outages Chaos Monkey: introduce faults Blameless postmortems: facilitate learning

29. 21st-Century Organization Service-oriented architecture vision coming to fruition  Loosely coupled micro-services  Graceful failure Open source as a model for change propagation Customer/service-centricity across functions  Need to integrate systems, practices, organizations

30. Tradeoffs Harder to control  “Governance” has to evolve More resilient/adaptable/agile Better suited to post-industrial business reality

31. Chapter Two Continuous Quality Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

32. Old Definition of Quality The software meets the spec Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

33. New Definition of Quality The service helps customers accomplish their jobs-to-be-done The service is resilient and adaptable The service helps the company compete Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

34. Service Co-creates Value Service provider helps customer create value-in-use Airline travel:  Not just about flying from one city to another  Also about having a vacation or accomplishing business Cloud:  Not just about acquiring Linux instances  Also about cost-effectively operating elastic, resilient applications Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

35. Value Co-Creation is a Journey Customers engage via multiple touchpoints over time Customers judge service by the entirety of their experience "The plane was nice but the service was terrible" Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

36. Airline Travel Journey:  Find a flight  Get to the airport  Go through security  Buy a cup of coffee  Board the flight  Collect baggage  Get to the hotel  Recommend (or not) Touchpoints:  Website, taxi service, security, Starbucks, waiting area, airplane 

37. Cloud Journey:  Understand a new service or feature  Provision instances  Import data  Get help/provide feedback  Incorporate updates  Deal with outages  Deprovision Touchpoints:  Website, console, API, documentation, blogs, email, phone, twitter 

38. Software as Service Co-creation has four dimensions:  Functionality  Operability  Deliverability  Coherency Their quality is inseparable in the customer’s mind Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

39. Functionality Does the service help me do what I want to do (e.g., get paid on time)?  Track my time  Generate and submit invoices  Pester clients who haven't paid on time  Track my overall cash flow Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

40. Operability Is the service available when and how I need it?  Is it up when I need to bill my clients?  Does it perform whether I’m the only user or one of a million?  Does it let my data get lost or stolen?  Can I get help when I need it?  Can I get information when it’s down?  Does it fail gracefully so I can partially continue my work? Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

41. Deliverability Does the service respond as my needs change?  70% functionality OK on day 1, not in year 2  Today I want to manage invoices, tomorrow expenses  I want bugs fixed in hours, not days/weeks/months  I want changes delivered in an absorbable manner Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

42. Coherency Does the service engage me throughout my journey?  Discovery/understanding  Adoption/onboarding  Administration  Usage  Help/feedback  New functionality delivery  Outage management  Departure Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

43. Service Changes Development User-centered design Test-driven development Continuous delivery Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

44. Complexity Changes Operations Preventing failure becomes responding to failure Steady-state operations becomes learning-from-failure Monolithic systems become service ecosystems Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

45. Adaptation Changes Planning Design becomes feedback Competition becomes OODA loop optimization Execution becomes continuous validation Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

46. Old QA Role Disintegrates Design/Development/Operations/Marketing blend with QA Automation takes over mechanical activities Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

47. Opportunity for New QA Role Representative/advocate for:  Service not software  Built-in quality  Adaptation Boundary-spanning mirror Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

48. Service Not Software Four dimensions of software as service Their inseparability in the customer's mind Co-creation as the fundamental unit of value The need for internal alignment

49. Built-in Quality Chaperone embedded QA activities Represent testing as 1st-class design and planning citizen Facilitate “moving bugs forward”

50. Adaptation Acceptance of incompleteness avoids catastrophic failure Pervasive, continual feedback improves dynamic quality QA is really a belief in the need for feedback loops  An adaptive org is a QA-friendly org

51. New QA Mindset Lift one’s gaze above mechanics of testing Treat tests as code  Adopt development tools and practices Focus on building quality into the system Focus on helping others think about quality Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

52. New QA Skills Understand and think about service  Outages involve communications, not just fixing servers Understand and think about ops Understand and think about process/automation Understand and think about complex systems Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

53. New QA Practices Deep participation in cross-functional teams Facilitation of service requirements understanding Outside-in testing Service delivery machine validation Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

54. Participation Good testers see systems and their parts Good testers ask probing questions  Did you think about…?  You forgot about… Good testers design good tests Engage that proficiency in design and test plan critiques Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

55. Requirements Understanding Bugs found earlier in the lifecycle are cheaper to fix Most important quality question:  Do design/dev/qa/ops understand each other and the customer? Techniques for modeling service requirements:  Customer Journey Mapping  Service Blueprinting  Behavior-Driven Development Understanding through conversations, not artifacts Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

56. Customer Journey Mapping Identifies:  Touchpoints where customers interact with you  How customers navigate between touchpoints  The surrounding context  Key moments and pain points Supports unified customer understanding Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

57. Service Blueprinting Identifies:  Front-stage activities and systems  Back-stage activities and systems  Connections and dependencies Key to coherent, efficient, satisfying fulfillment Surfaces internal customers and service design requirements Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

58. Behavior-Driven Development Shared specification language  User requirements  Development specs  Test definitions  Documentation/support scripts Minimizes loss of fidelity due to translation Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

59. Outside-in Testing User testing validates usability Unit testing validates code Still need to test whether the thing does what it’s supposed to:  Functional tests  Integration test  Non-functional tests Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

60. Continuous Outside-in Testing Continuous delivery challenges testers’ ability to keep up Solution:  Part software tools and design skills  Part agile process improvement Test pyramid focuses test development and execution Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

61. Delivery Machine Validation Continuous delivery/adaptation is part of the service Resilient operations is part of the service Co-creation necessitates designing for internal users Coherency necessitates system/practice/organization alignment Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

62. Validating the Delivery Machine Do resilient systems and practices function as desired:  MVP’s  Continuous Delivery pipeline  A-B Testing  Game Days  Chaos Monkey  OODA Loop optimization  Sprint retrospectives  Scaled agile  DevOps Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

63. New QA Value Unique holder of service understanding across all dimensions Antidote to confirmation bias  Everyone else is invested in what they designed and built  QA is invested in customer satisfaction  QA understands value of feedback Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

64. Continuous Quality Old: tail wagging the dog  Testing happens at the end  QA acts as bottleneck  Everyone hates QA New: continuous quality  Testing happens all the time  Entire process and organization infused with quality mindset  Entire process and organization focused on resilience/adaptation  QA acts as mirror Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

65. QA as Systems Thinking DevOps is about spanning boundaries Service is about inseparable value Infusion means brand quality == ops quality Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

66. QA as Complex Systems Thinking Complexity challenges reductionist/mechanistic thinking Local optimization impedes global optimization Loose coupling increases delivery machine resilience Learning indicates quality

67. QA As Mirror Need to translate between common and specialist viewpoints Need to translate between vendor and customer viewpoints QA can provide a mirror that helps everyone see:  The whole  Their relationship to it and to each other  Interdependence of systems, practices, organization Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

68. Co-Creation and Feedback The product is not the functionality The product is continuous, adaptive delivery & operation of functionality True co-creation dissolves customer/provider boundaries  Feedback requires permeable boundaries Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.

69. The Ultimate Definition of Quality The ability to continuously deliver and operate functionality that helps customers accomplish their jobs-to-be-done Copyright © 2013 Ingineering.IT, LLC. All Rights Reserved.