1. DevOps And the Agile DBA
Kellyn Pot’Vin-Gorman | Technical Intelligence Manager | September, 2017

2. Kellyn Pot’Vin-Gorman
Technical Intelligence Manager, Delphix
Multi-platform DBA, (Oracle, MSSQL, MySQL, Sybase, Postgres…..)
Oracle ACE Director, (Alumni)
Oak Table Network
APEX Women in Technology Award, CTA 2014
STEM education with Raspberry Pi and Python
President, Rocky Mtn Oracle User Group
Liaison, Denver SQL Server User Group
Author, blogger, (

3. DevOps
DevOps is a short for DEVelopment and IT OPerationS. The term is used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably
DevOps derives from both development and operations, groups that DBAs often have a foot in each of.
There is a high focus on collaboration, geared on methodologies, process and practice.
The goal is to release more frequently, more successfully and with less bugs.

4. How the DBA is Impacted by DevOps
Database Administrators heavily influence decisions.
Adverse to bleeding edge, high risk.
Introduction of the cloud, especially and SaaS, (Software as a Service).
Demand for decreased development cycles introduce risk, DBAs are seen as a roadblock to this.
Still, DBA skills are in high demand towards DevOps initiatives…

5. History of DevOps First called “Agile Infrastructure” in 2008
2009, the first time DevOps was coined in Belgium
Has taken over most of the fellow terms of WinOps and in some cases, Agile.
Introduced at relational database events in 2010, but rarely was given much credit until the last five years in the DBA community.
Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure”
The term DevOps was popularized through a series of "devopsdays" starting in 2009 in Belgium

6. DevOps Relationship to Agile
Agile and DevOps aren’t one in the same, but as it’s well known, DevOps came out of Agile’s success.
Agile= culture, where DevOps focuses more on the organization changes.

7. Don’t Confuse the Two… Agile’s success pushed the need for DevOps
Agile and DevOps have similarities, but Agile focuses more on culture and DevOps focuses on organizational changes to create success.
DevOps uses the success of release implementation as a scoring method.

8. Common Terms, (Alphabetically)
Agile
Build Automation
Configuration Management
Continuous Delivery/Rapid Deploy
Functional Testing/Unit Testing/Test Driven Development
Lean/Scrum
Monitoring/Alerting
Packages/Containers/Virtualization
Release Coordination/Orchestration

9. Build Automation
DBAs are a natural fit, due to their history with scripting and automation experience for backups, monitoring and alerting.
Build automation is the process of automating the creation of a software build and the associated processes including: compiling computer source code into binary code, packaging binary code, and running automated tests.

10. Configuration Management
Concept & Development
Testing & Verification
Configuration management (CM) is a systems engineering process for establishing and maintaining consistency of a product's performance, functional, and physical attributes with its requirements, design, and operational information throughout its life.
A DBA’s desire for low risk and stability assists here as we desire routine that results in expected outcomes.
Operations & Support
Deployment to Production

11. Continuous Delivery
DBAs are much slower to embrace continuous delivery. The risk is high, with it’s “fail first, fail fast” approach.
Continuous delivery (CD) is a software engineering approach in which teams produce software ... incremental updates to applications in production. A straightforward and repeatable deployment process is important for continuous delivery.

12. All Tiers of Specific Environment
Rapid Deploy
Software
Content
Enablement
Service
All Tiers of Specific Environment
Need or Focus Area
This is another area that introduces risk, so DBAs can be very adverse to it, but the focus on a single feature, helps minimize the impact and often isolate any issues.

13. Release Coordination/Orchestration
A demand for release tools and orchestration was at times a direct result to DBAs hesitation to risk production with a release. This helped automate and bypass the DBA to get code and features into production.
Release Orchestration is the use of tools like XLRelease which manage software releases from the development stage to the actual software release itself.

14. Source/Version Control
“A component of software configuration management, version control, also known as revision control or source control, is the management of changes to documents, computer programs, large web sites, and other collections of information.”
I’m going to add to this definition with Data version control.

15. The Goals of Each are the Same
Shorter Delivery Cycles
Higher Success Rate on Releases
Shorten Time to Address Issues
Continued Improvement in Communication
Increased Revenue

16. For the DBA, Why Virtualization
A technical approach in which users and applications do not use physical machines, but simulated systems running on actual, “real” hardware. Virtualization can be used to eliminate resource usage and enable savings for databases, network, file and application management, along with server infrastructure.
In computing, virtualization means to create a virtual version of a device or resource, such as a server, storage device, network or even a database. The framework divides the resource into one or more execution environments. For data, this can result in a golden copy or source that is used for a centralized location and removal of duplicated data. For read and writes, having unique data for that given copy, while duplicates are kept to singular.

17. Many Types of Virtualization
Each Virtual Database takes up around 5-10Gb upon creation, (dependent upon parameters)
PRODUCTION
Database/App Tier
1 TB
Storage Pool for Delphix
DEV
PATCH TEST QA
TEST
Point out the engine and size after we’ve compressed and de-duplicated.
Note that each of the VDBs will take approximately 5-10G vs. 1TB to offer a FULL read/write copy of the production system
It will do so in just a matter of minutes.
That this can also be done for the application tier!
▶ ▶ ▶
Virtualize and Deployed▶ ▶ ▶
Read AND Write
Read From Production
600GB

18. Why Database Virtualization is Different
DBA is “only as good as their last backup”.
Many database tools take considerable time to recover.
DevOps is often about automation- automate an “undo” for development and testing that includes data rewind.
Include a self-service tool that will allow for rewind without DBA intervention.

19. Data Holdup to Development Cycle
Timeline in a Project
Database Refresh
Testing Cycle
Development
Production Release
Data Holdup to Development Cycle
Waterfall- 5 Releases
January February March April
Agile with Virtualization- 12 Releases
January February March April
Over 80% of time is waiting for RDBMS, (relational databases) to be refreshed. Developers and Testers are waiting for data to do their primary functions.
This allows for faster and less costly migrations to the cloud, too.
The Goal of Agile

20. Packages/Containers
Containers offer the ability to isolate application code and/or the whole infrastructure stack into a package able entity to ease deployment, even from the same kernel. This is a powerful tool for DevOps to ease deployment for complex tiers, applications and multiple data stores.
Package software into standardized units for development, shipment and deployment. A container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.

21. “Containerize” for Ease of Delivery
Delphix Virtualization Engine
Same Storage Size as Production
Deployed using Jenkins
Reporting
NFS
iSCSI
Testing
NFS
iSCSI
Development
NFS
iSCSI
Rsync (UNIX/Linux)
Robocopy (Windows)
Create “Container”
We refer to a container as a template in our product.

22. Branching and Bookmarking
The ability to mark each iteration of development with a bookmark
Simplify to lock and deliver while testing a consistent image via a virtual database, (VDB)
If a something goes wrong, the ability to “bookmark”, (and subsequent snapshot) to deliver to development to address.
This is a cornerstone to developers and testers, so as DBAs, we know the pain when a developer comes to us to flashback a database and before that, recover or logically recover, (import or datapump) independent objects. What is The developer/tester could do this for themselves?

23. Using Jetstream Branch 1.1 Release 1.2 Release 1.3 Branch 1.3
This is the interface for Developers and testers- they can bookmark before important tasks or rewind to any point in the process. They can bookmark and branch for full development/testing needs.
Release 1.2
Release 1.3
Branch 1.3

24. DevOps and Source Control
This may appear to be a traffic disaster of changes, but for developers with Agile experience, a “sprint” looks just like this. You have different sprints that are quick runs and merges where developers are working separately on code that must merge successfully at the correct intersection and be deployed.
Versioning with source control is displayed at the top, using Virtual images. You can see each iteration of the sprints.
In the middle section is the branches of that occur during the development process. A virtual can be spun from a virtual, which means that it’s easier for developers to work from the work another developer has produced.
Stopping points and release via a clone is simply minutes vs. hours or days.

25. Maturity
The maturity of the DevOps environment will decide how silo’d or how blended the role you’ll have in DevOps vs. your standard role as a DBA.

26. Collaboration Practices/Tools
Methods- Scrum, KanBan, Extreme Programming, Continuous Delivery
Hybrid approaches combine, such as Scrumban and KanPlan
Collaboration- Jira, SlackStorm, Pivotal Tracker, HipChat, Rally, (now Agile Central)
Deployment- RapidDeploy, ElasticBox, JuJu, Otto
Automation- Jenkins, Chef, Ansible
Build Tools- Maven, Gradle, Make, Packer, MSBuild
Methods provide a format or guide to work from. Hybrid approaches often implement best.
Collaboration methods ensure that communication continues when team members return to their desks
Deployment tools help with documenting and lessons learned
Build tools help with automation and orchestration

27. Agile Methodologies
Scrum- A lightweight PM framework. Benefits include a flexible set of management and project protocols and strong collaboration focus.
Lean/Kanban- Strong focus on team and deliverables. Efficiency and ability to see the “whole”.
Extreme Programming, (XP)- Now the most popular, with continuous delivery key.
Crystal- A lightweight agile framework. A removal of hurdles, including bureaucracy, obstacles and other impediments to the development cycle.
Feature Driven Development, (FDD)- client and architecture centric software development, focused on the feature deliverable.
Scrum focuses on features, bug fixes and backlog debt. Serves very large teams, including those 800+
Lean’s goal is to eliminate all waste, over demand on resources and ability to deliver faster and more effectively each time.
XP is one of the most controversial due to the ability to deliver even to large companies every 1-3 weeks. Very disciplined approach.
Crystal is often known under Crystal Clear, Yellow Orange and others.

28. What is Scrum? An Agile Framework Scrum Framework consists of:
A Product owner creating a wish list and the “sprint” begins
Sprint planning and backlog is created.
Team sets up schedule and beings to have daily scrum standups, (commonly 5 minutes)
Scrum master keeps team focused and collaborating, keeps track of status
Product is released
Sprint ends with feedback and lessons learned
Next sprint begins

29. Example of a Scrum Sprint

30. Lean Kanban
Considered a very “visual” development process, based off of grocery store shelf stocking.
Uses standardized cues and refined processes
Goal to reduce waste and maximize value
Most often uses sticky notes and whiteboard to create a picture of the work to complete, what’s in process and what’s done.
Visualize Work
Limit Work in Process
Focus on Flow
Continuous Improvement

31. Picture is Worth a 1000 Words
Often uses a whiteboard with sticky notes…

32. Extreme Programming
Not THAT Extreme!

33. This Extreme Programming, (XP)
Code may come first in XP, but testing must already exist to know what the successful outcome will be.
Code is written by pairs of programmers, allowing for better collaboration.
Believes in the power of doing, vs. extensive planning. Failure is expected.
Always build foundations that can be built on later.
Rarely specialize- everyone develops, tests, designs, etc.

34. XP at a Glance

35. Crystal
Alistair Cockburn said, “Crystal is a family of human- powered, adaptive, ultra light, ‘stretch-to-fit’ software development methodologies.”
People are the most important aspect of Crystal.
Development is self-sufficient and self-organizing, (very different from Scrum)
Very lightweight
Very little documentation, overhead management or reporting.

36. Feature Driven Development
Like Rapid Deploy, more focused on delivering one feature as the product.

37. DevOps Tools for the DBA

38. The Periodic Table of DevOps Tools

39. Ansible Learn about Playbooks
Powerful collection of steps, (aka plays)
Plays consist of a list of tasks
Handlers allow for remote operations
YAML Syntax
Security configurations based on hosts and users
Ability to run as secondary users, (privilege escalation)
Huge support community and best practices are mature

40. Example of an Ansible Call
- name: Transfer and execute a script.
hosts: server
remote_user: test_user
sudo: yes
tasks:
- name: Transfer the script
copy: src=test.sh dest=/home/test_user mode=0777
- name: Execute the script
command: sh /home/test_user/test.sh

41. Jenkins Jenkins is an open source automation server-
Easiest executed via a jetty servlet
Has an easy to access URL, (commonly host:8080)
Automation is done via individual projects.
Has numerous plugins to support environments
Supports multiple scripting languages, including shell, batch commands and Ant scripts.
Supports robust revision control and compatible with version control systems like CVS and Subversion.
Supports notifications like qmail and sendmail.
Ant is another java based built tool that’s part of Apache open-source project.
Similar to Make and written in XML.

42. Jenkins Groovy Script Example
// prepare variables for subscript execution:
String subscript = "cdmserver-monitor-instance-status.groovy";
println "executing subscript ${subscript}"
// variables to be passed to the subscript
status = (action == "start" ? "started" : "stopped")
def s = ScriptlerConfiguration.getConfiguration().getScriptById(subscript)
File scriptSrc = new File(ScriptlerManagment.getScriptDirectory(), s.getScriptPath());
This Groovy script executes another script, making it valuable in environments that already have a number of mature scripts in place that should be reused in automation.

43. Jenkins Plugin for Delphix

44. Puppet Review, automate, deploy and invest in all your software.
Control and enforce consistency across environments and platforms.
Share, test and enforce changes on-premise and in the cloud.
Similar to Puppet
Chef
Jenkins

45. Docker Virtualization is important, this is OS level
Resource level isolation at the kernel level
Technology such as OverlayFS, allows for containers, which removes overhead of other virtualization technologies.
Docker Alternatives:
Open Container Initiative, (OCI)
Kubernetes, (Google)
CoreOS and rkt
Apache Mesos
Canonical and LXD

46. GitHub
Git is an open-source version control system that was started by Linus Trovalds
If you aren’t using it now, start. Yes, as a DBA…
Great option to highlight your scripting work
GitHub Alternatives
Subversion
CVS
Mercurial

47. Creating a GitHub Repository Project

48. Github Repository Example

49. So, Know Your Tools
Incredible variation of tools and knowing tool classifications doesn’t mean that the tools are alike. Many groups are often comparing one feature of a given tool.

50. From the DBA Perspective
DevOps include the following focus areas for Database Administrators:
Automating repeatable and complex processes
Ensuring the databases and data stores aren’t the bottleneck to the development cycle and critical data is protected.
Locating heterogeneous tools/scripting languages that can support multiple tiers of technology.
Understanding that automation does not lessen your value.
Embracing new skills to add more value to the business and to your career.

51. Summary- DBA Focus Areas
Update skills, (scripting, monitoring, automating) to remove bottlenecks
Enhance monitoring utilities that take infrastructure and development into overall view
Recognize a need to become more skilled in scripting and become knowledgeable of DevOps tools.
Data is EVERYWHERE! Recognize the integral part Database Administrators play in DevOps.
DBA
Skills
DevOps

52. Thank you! kellyn@delphix.com http://dbakevlar.com http://delphix.com
@DBAKevlar
Free eBook Download: