1. 2013 Trend Micro 25th Anniversary 2014/6/11 Amazon Web Service AWS Technical Professional

2. 2013 Trend Micro 25th Anniversary 2014/6/12 AWS Technical Professional Batch Processing

3. 2013 Trend Micro 25th Anniversary AWS Technical Professional Batch Processing | Module Overview 2014/6/13 Describe the services provided by AWS that facilitate batch processing. Describe how to create auto scaling groups that are triggered by items in a queue. Describe the Amazon Simple Queue Service (SQS) delivery guarantees. Describe the performance and durability characteristics of SQS. Module Objective

4. 2013 Trend Micro 25th Anniversary Batch Processing Overview Using AWS to Process Heavy Computational Loads Simplified Architecture walkthrough 2014/6/1

5. 2013 Trend Micro 25th Anniversary 2014/6/15 Batch Processing| Overview The execution of a group tasks by a computer, usually without manual intervention. Batch Processing We run a website that showcases independent graphic artists and displays a library of images. Artists submit artwork in the form of digital images and we generate a watermarked version of the image to prevent image theft along with thumbnail images to display in the online catalog. Image processing requests will likely be submitted in chunks so we will store work requests in a queue and scale our computational resources (Amazon EC2 instances) automatically using Auto Scaling to meet the demand. Our Scenario

6. 2013 Trend Micro 25th Anniversary Processing Heavy Computational Loads Scenarios requiring large computational capacity Real world examples Generalized architecture 2014/6/1

7. 2013 Trend Micro 25th Anniversary 2014/6/17 Batch Processing| Heavy Computational Loads Analysis of large datasets Scientific simulations Analysis of financial markets Batch processing Scenarios Data is easily segmented into multiple datasets that can all be processed independently and at the same time. Application can generally consume all compute or processing resources available. Pay only for resources you use. Application Characteristics

8. 2013 Trend Micro 25th Anniversary 2014/6/18 Batch Processing| Heavy Computational Loads Item Classification: Define risk and compliance profile for 9 million units per day across 1+ billion item catalog Affiliates Program: Calculate monthly payments to millions of Amazon affiliates Real-world examples at Amazon.com NASA: Uses the Amazon Simple Workflow Service (SWF) to process images received from the Mars rover Curiosity. DigiChalk: Uses the Amazon Simple Queue Service (SQ) to manage and transcode media files. Real-world example outside Amazon.com

9. 2013 Trend Micro 25th Anniversary 2014/6/19 Batch Processing| Generalized Architecture This diagram shows a generalized batch processing architecture and identifies which AWS services can be used for the various roles.

10. 2013 Trend Micro 25th Anniversary 2014/6/110 Batch Processing| Generalized Architecture – Orchestration Services The orchestration services guide the flow of data or information through the system.

11. 2013 Trend Micro 25th Anniversary 2014/6/111 Batch Processing| Generalized Architecture – Data Services The data and storage services serve as a repository for the data that is consumed and emitted by the batch processing application

12. 2013 Trend Micro 25th Anniversary 2014/6/112 Batch Processing| Generalized Architecture – Compute Services The compute services are responsible for performing the work on the incoming data.

13. 2013 Trend Micro 25th Anniversary 2014/6/113 Batch Processing| Generalized Architecture – Notification Services Notification services are used to provide notifications to observers when the work has been completed.

14. 2013 Trend Micro 25th Anniversary Batch Processing Scenario Simplified architecture Operational considerations 2014/6/1

15. 2013 Trend Micro 25th Anniversary 2014/6/115 Batch Processing| Simplified Architecture This diagram illustrates how a batch processing scenario can be implemented with AWS and how the component services would interact.

16. 2013 Trend Micro 25th Anniversary 2014/6/116 Batch Processing| Simplified Architecture – Ingestion Stage Stage 1 –User submits work to the job manager.

17. 2013 Trend Micro 25th Anniversary 2014/6/117 Batch Processing| Simplified Architecture – Ingestion Stage Stage 2 – Job manager queues the work request

18. 2013 Trend Micro 25th Anniversary 2014/6/118 Batch Processing| Generalized Architecture – Processing Stage Stage 3 – Instances pull work from the queue

19. 2013 Trend Micro 25th Anniversary 2014/6/119 Batch Processing| Simplified Architecture – Processing Stage Stage 4 – Grab work details and do the work

20. 2013 Trend Micro 25th Anniversary 2014/6/120 Batch Processing| Simplified Architecture – Storage and Publishing Stage Stage 5 – Output the work

21. 2013 Trend Micro 25th Anniversary 2014/6/121 Batch Processing| Operational Considerations – Amazon Simple Queue Service Preparing the Queue

22. 2013 Trend Micro 25th Anniversary 2014/6/122 Batch Processing| Operational Considerations – Amazon Simple Queue Service Preparing the Queue

23. 2013 Trend Micro 25th Anniversary 2014/6/123 Batch Processing| Operational Considerations – Amazon Simple Queue Service Preparing the Queue

24. 2013 Trend Micro 25th Anniversary 2014/6/124 Batch Processing| Operational Considerations – Amazon Simple Queue Service Queue Details

25. 2013 Trend Micro 25th Anniversary 2014/6/125 Batch Processing| Operational Considerations – Auto Scaling Name the configuration Specify the EC2 instance type Specify the ID of the AMI to use when launching a new instance Auto Scaling Step 1 – Create the Launch Configuration as-create- launch-config myLaunchConfig --image-id ami-2a31bf1a --instance-type m1.smaLL Command Line Example

26. 2013 Trend Micro 25th Anniversary 2014/6/126 Batch Processing| Operational Considerations – Auto Scaling Name the group Specify the launch configuration (created previously) Specify the minimum and maximum sizes of the group Auto Scaling Step 2 – Create the Auto Scaling Group as-create- auto-scaling-group myAutoScalingGroup --launch-configuration myLaunchConfig --min-size 1 --max-size 20 Command Line Example

27. 2013 Trend Micro 25th Anniversary 2014/6/127 Batch Processing| Operational Considerations – Auto Scaling Name the policy Specify the policy type Specify the associated auto scaling group Specify the auto scaling adjustment Auto Scaling Step 3 – Create the Auto Scaling Group Policies as-put-scaling-policy myScaleUpPolicy --type PercentChangeInCapacity --auto-scaling-group myAutoScalingGroup --adjustment 50 as-put-scaling-policy myScaleDownPolicy --type ChangeInCapacity --auto-scaling-group myAutoScalingGroup --adjustment =-1 Command Line Example

28. 2013 Trend Micro 25th Anniversary 2014/6/128 Batch Processing| Operational Considerations – Auto Scaling Auto Scaling profile can be executed by: A manual action (via the command line tools or API) A time-based schedule Responding to an Amazon CloudWatch alarm Auto Scaling Step 4 – Create the Auto Scaling Triggers

29. 2013 Trend Micro 25th Anniversary 2014/6/129 Batch Processing| Operational Considerations – CloudWatch Creating the Alarm

30. 2013 Trend Micro 25th Anniversary 2014/6/130 Batch Processing| Operational Considerations – CloudWatch Defining the Alarm Properties

31. 2013 Trend Micro 25th Anniversary 2014/6/131 Batch Processing| Operational Considerations – CloudWatch Configuring the Alarm Actions

32. 2013 Trend Micro 25th Anniversary