2. Automating operational procedures with Daniel Fernández Rodríguez - danielfr@cern.ch (IT-CM-RPS)danielfr@cern.ch Akos Hencz - ahencz@cern.ch (IT-CM-LCS)ahencz@cern.ch CERN IT-CM Group 21/04/2016 HEPiX Spring 2016 Workshop1

3. Outline What is Rundeck? What does Rundeck provide us? Deployment Use cases Missing things Summary 21/04/2016 HEPiX Spring 2016 Workshop2

4. What is Rundeck?  Automate routine operational procedures  Company -  Language Java  Apache Software License - Code on  Actively developed 21/04/2016 HEPiX Spring 2016 Workshop3

5. Concepts Projects: used to separate management activity. All Rundeck activities occur within the context of a project Jobs: a sequence of steps, job options, and the nodes where the steps are executed Executions: An execution is a representation of the activity of a running or completed job Plugins: extend and expand Rundeck’s functionality http://rundeck.org/plugins/index.html 21/04/2016 HEPiX Spring 2016 Workshop4

6. What does Rundeck provide us? Common place for implemented workflows and jobs Clean and easy-to-use Web interface Allow for scheduled jobs, cron-style Jobs reusability 21/04/2016 HEPiX Spring 2016 Workshop5

7. What does Rundeck provide us? Fine-grain access control  Ex: Sysadmins can only execute, Cloud Team modify + execute Traceability and auditing Delegation internal sensitive tasks to other groups without exposing credentials or procedures  Ex: Sysadmins remove node from infrastructure w/o having rights Complete log of every executed job 21/04/2016 HEPiX Spring 2016 Workshop6

8. All puppetized https://github.com/voxpupuli/puppet-rundeck Currently working on a Puppet module to deploy Rundeck with some CERN specifics Different deployments for different services Credentials Deployment 21/04/2016 HEPiX Spring 2016 Workshop7

9. Deployment mod_shib Apache :443 Rundeck :4440 Tomcat :8009 rundeck-01.cern.ch 21/04/2016 HEPiX Spring 2016 Workshop8

10. Use cases  OpenStack Management  Project Creation  Project Quota Updates  Reporting on the Cloud  Usage and status  Health reports  Notifications  Hyper-V Patching Campaigns  VM owners  Checks and verifications  Consistency checks  GNI Tickets  Configuration  Rename physical host  Register new Puppet entities  Various scheduled synchronizations  Host Management  Enable/disable Compute Host  HW Interventions  Remove physical host  Internal  Auto backups  Reload nodes  …. 21/04/2016 HEPiX Spring 2016 Workshop9

11. HW Intervention on a Compute Node Disable compute node Disable nova-service Switch Alarms OFF Update associated Service-Now ticket Notifications Send e-mail to owners of affected VMs Other tasks Post new message broker Add remote AT job Save intervention details Send calendar invitation 21/04/2016 HEPiX Spring 2016 Workshop10

12. HW Intervention on a Compute Node Kerberos granting ticket for svcrdeck obtained successfully [INFO] Changed OS_AUTH_URL to https://keystone.cern.ch:5001/v2.0/ [INFO] Changed SERVICE_ENDPOINT https://keystone.cern.ch:35358/v2.0/ [INFO] Changed OS_REGION_NAME to cell12 [INFO] Appending cern.ch to the host name... [INFO] Trying to disable nova-compute on P05614911S20073.cern.ch... +-------------------------+--------------+----------+---------------------------------------------------------+ | Host | Binary | Status | Disabled Reason | +-------------------------+--------------+----------+---------------------------------------------------------+ | P05614911S20073.cern.ch | nova-compute | disabled | [INC0982341] Disabled by rsantalu using Rundeck (11823) | +-------------------------+--------------+----------+---------------------------------------------------------+ [INFO] nova-compute sucessfully disabled on P05614911S20073.cern.ch. TOTAL HOSTS SUCCESS ERROR [PERFORM] SUMMARY NOVA DISABLE 1 1 0 [INFO] Using roger to disable alarms on the following hosts: P05614911S20073 [INFO] Trying to disable roger alarms on P05614911S20073... [INFO] Roger alarms sucessfully disabled on P05614911S20073. [ { "app_alarmed": false, "appstate": "production", "expires": "", "hostname": "p05614911s20073.cern.ch", "hw_alarmed": false, "message": "[INC0982341] Disabled by rsantalu using Rundeck (11823)", "nc_alarmed": false, "os_alarmed": false, … 21/04/2016 HEPiX Spring 2016 Workshop11

13. Register new Puppet entities Create new puppet modules & hostgroups JIRA Configuration Change ticket  Ticket status APPROVED  JIRA Web-Hook executes job via Rundeck REST API 21/04/2016 HEPiX Spring 2016 Workshop12

14. Register new Puppet entities Parse JIRA ticket with the given ID  Create project in Gitlab, set ACLs, user groups, LDAP links  Create puppet skeleton in project & QA branch  Register project in Foreman and other systems Close ticket 21/04/2016 HEPiX Spring 2016 Workshop13

15. GNI Alarm Tickets 1 2 3 4 Opened to Cloud Team GNI Not assigned Mark ticket as resolved or Assign to person in ROTA 5 21/04/2016 HEPiX Spring 2016 Workshop14

16. Missing things… Pass data between jobs (https://trello.com/c/oU8OjZoH)https://trello.com/c/oU8OjZoH Share jobs between Projects (https://trello.com/c/jtFcbflm)https://trello.com/c/jtFcbflm Resume job from where error occurred (https://trello.com/c/o999XAhW)https://trello.com/c/o999XAhW Configure scheduled jobs to be run once (AT jobs) (https://trello.com/c/cmqu4fdW)https://trello.com/c/cmqu4fdW Allow user interaction (https://trello.com/c/LPnsZ019)https://trello.com/c/LPnsZ019 21/04/2016 HEPiX Spring 2016 Workshop15

17. Summary Rundeck is a great platform, easy to use and deploy Active development and increasing user community Adapt its behaviour to your preferences via custom plugins Valid for simple jobs but also for more complicated workflows 21/04/2016 HEPiX Spring 2016 Workshop16

18. Questions? Thank you for your attention! 21/04/2016 HEPiX Spring 2016 Workshop17

20. I like it. Where can start from? Web page: http://rundeck.orghttp://rundeck.org Oficial docs: http://rundeck.org/docs/index.htmlhttp://rundeck.org/docs/index.html Rundeck Trello Board for future ideas: https://trello.com/b/sn3g9nOr/rundeck-development https://trello.com/b/sn3g9nOr/rundeck-development Mailing list: http://groups.google.com/group/rundeck- discusshttp://groups.google.com/group/rundeck- discuss Puppet module: https://github.com/voxpupuli/puppet- rundeckhttps://github.com/voxpupuli/puppet- rundeck 21/04/2016 HEPiX Spring 2016 Workshop19