1. Part 2 – OpsWorks Stacks and Opsworks for Chef Automate
© 2017 Amazon Web Services, Inc. and its affiliates. All rights served. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon Web Services, Inc.

2. What Is OpsWorks Stacks?
Integrated application management solution for ops-minded developers and IT admins
Model, control and automate applications of nearly any scale and complexity
Management Console, SDKs, or CLI
No additional cost
Integrated application management solution for ops-minded developers and IT admins
You can model, control, and automate application stacks of nearly any scale and complexity.
You only pay for the resources you use – OpsWorks itself is free.

3. Why AWS OpsWorks? SIMPLE PRODUCTIVE FLEXIBLE POWERFUL SECURE
Easy to use, get started quickly and improves productive
PRODUCTIVE
Reduces errors with conventions and scripted configuration
FLEXIBLE
Simplifies deployments of any scale and complexity
POWERFUL
Reduce cost and time with automation
SECURE
Enables control with fine grained permissions
Why should you use OpsWorks?
SIMPLE to use
built-in support for common frameworks and tasks,
Makes teams more PRODUCTIVE
Reduces errors with conventions and scripted configuration.
FLEXIBLE
Simplifies deployments and supports a wide variety of application architecture
POWERFUL
You can reduce cost and engineering time with automation and an event-driven configuration system with support for 1) customizable deployments,2) rollback, 3) patch management, 4) auto scaling, and more
And finally, it’s SECURE
You can use the default security settings that come with OpsWorks, as well as extend those and use fine grained permissions to control who has access to your stack, and what they can do with that access.

4. Improve Productivity Scalable infrastructure Flexible architecture
Deploy often
Staging environments
AWS OpsWorks gives us the tools we need to automate operations.
We can scale Monster World, one of the largest Facebook games, to millions of users without ever needing more than two backend developers.
Jesper Richter-Reichhelm head of engineering
Let’s talk about typical 3 use cases that we see with OpsWorks customers. The first is using automation to improve productivity.
Wooga is a gaming customer. Games draw an unpredictable number of players, making it difficult to calculate the number of instances needed at any given time.
But Wooga uses OpsWorks to create high degree of automation the enable the rapid deployment of apps, so they’re able to rapidly adjust to increased usage.
It gives them solid reliability. If a game is offline, players will move on to the next.
Finally, OpsWorks is a solution that can work for small, agile teams, as you can see here Wooga runs a lean engineering organization.

5. Improve Reliability Git Jenkins OpsWorks Deploy Code Build Test
Check in – Build & Test
Tests pass – Deploy
Git
Jenkins
OpsWorks
Deploy
Code
Build
Test
Provision
Monitor
The second use case is integrating a development pipeline to improve reliability. You can automate your entire build process – first deploying into a staging environment that mirrors production, and then when your integration tests pass, deploy to production. You can easily spin up parallel production environments and migrate traffic incrementally so that it’s easy to roll back if there are problems.
So, let’s say you have a Git repo with your code. When new code is checked in, kick off the build process using something like Jenkins.
When build is done, run through automated tests.
Then script the OpsWorks CLI to change the app’s version and deploy to prod.
(Or clone your stack, deploy to the new stack, and flip DNS when ready)

6. Improve Control Versioned config
Control anything you can script with Chef recipes
Lock down console access
Change and deployment logs
Crashlytics uses AWS OpsWorks to support the rapid growth of our mobile crash reporting solution. With AWS OpsWorks, we can focus on the development and growth of our service and not spend developer cycles on infrastructure and operational tasks.
Jeff Seibert CEO of Crashlytics
The third use case is using both recipes and permissions to improve control. We’ll talk more about Chef in a moment, but OpsWorks can use Chef recipes to control anything you can script. With our Identity and Access Management tools, you can specify users or groups of users that are able to take actions against your stack.
So you may have one team responsible for dev / test, but another team responsible for pushing those changes to production.

7. OpsWorks Security
Let’s take a quick look at how OpsWorks Security works for users. You define the users you want to have access to your stacks, and what level of access those users have.
AWS OpsWorks is the only service that integrates IAM with instance logon/credentials.

8. OpsWorks Security
We have some built in security policies for you, or you can customize the level of access to reflect your business requirements.

9. OpsWorks Security
Once you’ve decided on who should have what responsibility, you can easily make those changes.

10. OpsWorks Concepts
Decide how to scale: manually, with 24/7 instances, or automatically, with load-based or time-based instances.
A stack represents the cloud infrastructure and applications that you want to manage together.
A layer defines how to setup and configure a set of instances and related resources.
Then deploy your app to specific instances and customize the deployment with Chef recipes.
Layers: App Server, Load Balancer, Database Layer, Caching Layer,

11. Modeling in AWS OpsWorks: Stacks
In OpsWorks you organize your projects / stages / environments in Stacks.

12. Creating a Stack Via the AWS Management Console Via the AWS CLI
Via an AWS SDK
$ aws opsworks create-stack
As with our other AWS services, OpsWorks gives you several options for creating your stack.

13. Modeling in AWS OpsWorks: Layers
Layers inside the Stack are used as blue prints to configure and manage your instances.

14. 10 Built-In Layers Java App Server HAProxy AWS Flow (Ruby + SWF)
Memcached
Ganglia
MySQL
HAProxy
Static Web Server
Rails App Server
PHP App Server
Node.js App Server
Open Source Cookbooks available at
OpsWorks has 10 built-in layers, and one custom layer, where our customers can define their own resources that make up a layer

15. Modeling in AWS OpsWorks: Instances
Instances are blueprints and might be different EC2 instances over time.
We support the direct integration with other services like ELB, our elasticLoad Balancer

16. Scaling AWS OpsWorks: 24/7 Instances

17. Scaling AWS OpsWorks: Time-based Instances

18. Scaling AWS OpsWorks: Load-based Instances
OpsWorks integrates with CloudWatch to give you detailed layer metrics. If you notice heavy CPU or memory load, you can setup triggers to add “load-based” instances to your layer.

19. Scaling AWS OpsWorks: Load-based Instances

20. Back to Automation: The OpsWorks Agent
talks with
The core of OpsWorks is a software agent we put on every server you start.
That little piece of software talks to our service and can execute commands on the instances.
In addition to that it also sends a heartbeat from the server and host level metrics to OpsWorks.

21. AWS OpsWorks Agent Installed on every instance
Can execute different commands on an instance
Sends keep alive messages for auto healing
Sends 15 host-level metrics every minute
The core of OpsWorks is a software agent that exists on every server you start.
The agent talks to our service lets you execute commands on the instances.
In addition to that it also sends a heartbeat from the server and host level metrics to OpsWorks.
Host-level metrics sample: cpu metrics, load metrics, memory metrics, and active processes

22. AWS OpsWorks and Chef
Lifecycle Events
Recipes
Chef is an open-source framework that automates software deployment and configuration.
Whenever a change happens on your stack, or upon request, all instances are notified and recipes are run.
Metadata
OpsWorks triggers events during the application lifecycle, such as when an instance is provisioned or an application is deployed.
And it does this by using Chef. Chef is an open-source framework that automates software deployment and configuration.
It does this in the form of “recipes”. Chef recipes are attached to these lifecyle events. You can also specify metadata that is passed into the recipes, so that you don’t have to hardcode things like passwords inside your configuration code. Let’s take a look at an example.

23. Chef Recipes in Action Recipe + Metadata = Command
execute “mysql-connect” do
command “/usr/bin/mysql
-u#{node[:deploy][:myphpapp][:database][:username]}
-p#{node[:deploy][:myphpapp][:database][:password]}
#{node[:deploy][:myphpapp][:database][:database]} …
Recipe +
Metadata =
Command
“deploy”: {
“myphpapp”: {
“database”: {
“username”: “root”,
“password”: “abcxyc”, …
“/usr/bin/mysql -uroot –pabcxyz myphpapp …
So this is a recipe that might connect your app to a MySQL server. The recipe would be brittle if we hard coded information like the username and database name, so we define metadata and pass it into the recipes.
So when you take a recipe, and add in the meta data, the system sees the command as if you’d hard-coded it straight into your script.

24. App Environment Variables
user = ENV[“payment_provider_user“]
password = ENV[“payment_provider_password“]
PaymentGateway.new(user, password)
Additionally, you can use the Console to set environment variables. Here, we have specified values for a user and password, and our apps now have access to those variables.

25. AWS OpsWorks Agent: Custom Recipes
Custom recipes can be found in OpsWorks cookbooks, like these AWS provided recipes on Github, or in other 3rd party repos

26. AWS OpsWorks Agent: Custom Recipes
…and those recipes can be used to create custom layers in your stack.

27. Lifecycle Events
The commands the agent reacts on describe important events in an instances life.
There is the setup event to get a server from the plain vanilla state to a become a web server, or a database, or whatever you want it to be.
The configure event, which updates the servers configuration on changes in the Stack. This allows you to react to changes so that you can, for example scale up or switch databases when the primary crashes.
The deploy and undeploy events you use to get code on and off the machines.
The shutdown event to trigger a script to ship your last logs before the instance is gone or similar clean up things.

28. Setup Event Sent when instance boots Includes deploy event
Use for initial installation of software & services
A recipe that kicks off when the instance boots, and it’s useful for Use for initial installation of software & services. You might use it to automatically update your system, install patches, or start services.

29. Configure Event
Sent to all instances when any instance enters or leaves online state
Use for making sure the configuration is up-to-date
So if your autoscaling were to add a new instance, the configure event would run on all your instances, ensuring that they were up to date and matched each other.

30. Deploy Event Sent when you deploy via UI/API; part of each setup.
Use for custom deployment

31. Undeploy Event Sent via UI/API when apps are deleted
Use to remove apps from running instances

32. Shutdown Event Sent when an instance is shut down ~45s to execute
Use for clean shutdown

33. Debugging
1) Create a Stack
2) Add a layer
3) Start layer
4) Create simplejsp.jsp file in a simplejsp directory, archive, upload
4a) zip –r simplejsp.zip simplejsp
4b) aws s3 cp simplejsp.zip s3://bbvademo/ --acl public-read
5) Add the app

34. opsworks-agent-cli
list_commands list the commands get_json show the JSON used run_command re-run a command show_log show the current or an old log
For that you just SSH on to an instance
And there you would use the opsworks-agent-cli
The opsworks-agent-cli is a convenience wrapper around our agent
So how would you use it to develop a cookbook

35. Agent CLI – list what happened
# list event history (up to 10 events) # opsworks-agent-cli list T10:26:58 setup T10:55:13 configure T14:59:05 execute_recipes T18:25:53 deploy T18:34:11 configure T10:05:23 configure T14:43:43 configure
You would probably start listing the most recent events that happened on the machine to figure out which kind of event and therefore which recipes you want to change

36. Agent CLI – get the JSON to debug
# show JSON events (latest or by type/timestamp) # opsworks-agent-cli get { "ssh_users": { "2066": { "name": "thomas", "public_key": "ssh-rsa AAAAB3NzaC1...", "sudoer": true ...
You can also look up the JSON we did hand over for each even by using the get command

37. Make a Change to your Cookbooks
# list current agent folder and cookbooks # ls /opt/aws/opsworks/current cookbooks # OpsWorks cookbooks berkshelf-cookbooks # Cookbooks from Berkshelf site-cookbooks # Your cookbooks merged-cookbooks # Resulting cookbooks for run ...
To do the actual change you would change into opt / aws /opsworks/ current / site-cookbooks
As this is the place were your own cookbooks live
Than just use vi or some other editor to do the change on your cookbook

38. Agent CLI – re-run an event
# run last setup event (changed cookbooks, same JSON) # opsworks-agent-cli run setup # or select specific date # opsworks-agent-cli run T14:43:43 [ :06:41] INFO [opsworks-agent(17817)]: About to re-run 'setup' from T14:43:43 Waiting for process Finished Chef run with exitcode 0
After you saved your change you can use the opsworks-agent-cli again to re-run the last event of a specific type or you can also pass on a timestamp to re run a specific older event if you like
After that you will see the Chef log appear in you console and you can check live on that instance if your change was successful

39. Testing

40. Infrastructure Deserves Tests
Infrastructure as code Test your code Test your infrastructure But how?
What you probably really want is a similar fast experience but also write actual tests for your cookbooks
Because if infrastructure as code is true
Infrastructure should have tests
For this we chose one of many tools out there – Test Kitchen.

41. Test Kitchen
“Test Kitchen is a test harness tool to execute your configured code on one or more platforms in isolation.”
Test Kitchen is a test harness tool to execute your configured code on one or more platforms in isolation.
So test kitchen is Configuration Management-Neutral and not bound to Chef.
A driver plugin architecture is used which lets you run your code on various providers and virtualization technologies such as Amazon EC2, Vagrant, Docker, and many more.
We currently build a driver directly for OpsWorks we hope to release soon.

42. Test Kitchen – Workflow
Destroy
Create
Converge
Setup
Verify
So how does a test kitchen run look like
A test starts by
Cleaning up potential left overs and destroying still running instances
Than it Create as instance in OpsWorks as we will use the OpsWorks driver
Converge does the chef actual run
Setup – setups the used test environment
Verify – uploads the test to your OpsWorks instances and run the tests
Destroy again cleans up running instances
You can repeat any step as often as you like
So how is test kitchen set up

43. AWS OpsWorks for Chef Automate
[ COPY FROM WEBSITE ]

44. Benefits Fully Managed Chef Server Programmable Infrastructure
Scaling Made Easy
Support from Active Chef Community
Simple to Manage Hybrid Environments

45. Fully Managed Chef Server
Easy to use Chef on AWS
Automatic backups
Automatic updates & upgrades
Chef minor versions
OS updates
Security updates
AWS OpsWorks for Chef Automate makes it easy to use Chef sever on AWS. There is no need for Chef server provisioning and installation. OpsWorks for Chef Automate automatically updates the Chef server software and creates backups of the server for you. You can use the AWS Management Console, AWS Command Line Interface (CLI), or AWS SDKs to provision a ready-to-use Chef server.

46. Support from Active Chef Community
Use Chef community cookbooks or create your own
Supports a wide variety of applications out-of-the-box
Control every aspect of your host’s configuration
Interface with your Chef server using native Chef tools - Knife command-line tool or ChefDK
AWS OpsWorks for Chef Automate supports the latest version of Chef server and Chef Automate. You can use any Chef community-built tools or cookbooks with your Chef server. With OpsWorks for Chef Automate, you can interface with your Chef server using native Chef tools, such as the Knife command-line tool or Chef Development Kit (ChefDK).

47. Programmable Infrastructure
Define configuration using Chef recipes written in a Ruby DSL
Maintain and version your configuration
Ensure consistent configuration and compliance
With Chef server, you can define configurations for your servers in a format that you can maintain and version just like your application source code. Chef server ensures your servers are consistently configured and maintained, and handles complex operational tasks when there are interdependencies across your servers. 

48. Get visibility into the state of your nodes
Chef Visibility – a view into convergence, compliance, cookbooks, recipes and more
Premium Chef Features
With AWS OpsWorks for Chef Automate, you receive the full Chef Automate platform which includes premium features that you can use with Chef server, like Chef Workflow, Chef Visibility, and Chef Compliance
View into the state of your cookbooks, recipes, roles and most importantly your nodes,
Whether what is defined in the recipe meets the state of the node.
Chef calls converge status.
Data warehouse platform – use through the UI to get insights into the state of your environment.

49. Go beyond Configuration Management
Chef Workflow – A continuous delivery pipeline for infrastructure and applications
Move a change in your host or application configuration from the developer’s workstation all the way to production.
Has dashboards that let you track each change as it goes through the pipeline

50. Go beyond Configuration Management
Chef Compliance - Discovery and analysis of compliance risks across environments
Chef Compliance lets you generate customizable reports to identify:
Compliance issues, security risks, and even outdated software.
Aggregated in the Visibility console
Write your own compliance rules, or you can get started quickly by using built-in ones.
Use predefined rule-sets for existing security frameworks

51. Any Questions?