1. Computing on the cloud B. Ramamurthy 9/20/2018
Rich's Big Data Analytics Training
9/20/2018

2. Outline Golden Era in Computing Data and Computing challenges
Cloud Computing
Popular Cloud Providers
Working with Amazon Web Services (aws)
Identity management: Security strategy
Summary
References
Rich's Big Data Analytics Training
9/20/2018

3. A Golden Era in Computing
Heavy societal involvement
Powerful multi-core processors
Superior software methodologies
Virtualization leveraging the powerful hardware
Wider bandwidth for communication
Proliferation of devices
Explosion of domain applications
Rich's Big Data Analytics Training
9/20/2018

4. Computing for everyone
Explosion in computing accessible to all through emerging models
R packages, data (validated) with documentation, easy to install and use; transformed complex algorithms and statistical models into one-line commands
Proliferation of mobile apps readily available for use
Compute servers loaded with features (thanks to advances in virtualization) : amazon machine images *(AMI), stacks*, data pipelines
API (application programming interfaces) providing data and functions: NYT API: , NOAA:
Web services/ RSS(really simple syndication) for unstructured data (text, images)
Cloud models and services to host most anything with on-demand and elastic capability.
Rich's Big Data Analytics Training
9/20/2018

5. Computing Challenges
Scalability issue: large scale data, high performance computing, automation, response time, rapid prototyping, and rapid time to production
Need to effectively address (i) ever shortening cycle of obsolescence, (ii) heterogeneity and (iii) rapid changes in requirements
Transform data from diverse sources into intelligence and deliver intelligence to right people/user/systems
What about providing all this in a cost-effective manner?
How to make computing available and accessible as a public resource?
Most of all how to store and process the data collected by the numerous devices and processes in our own environment? What will serve as backend for these numerous sources?
Rich's Big Data Analytics Training
9/20/2018

6. Enter the cloud
Cloud computing is Internet-based computing, whereby shared resources, software and information are provided to computers and other devices on-demand, like the electricity grid.
The cloud computing is a culmination of numerous attempts at large scale computing with seamless access to virtually limitless resources.
on-demand computing, utility computing, ubiquitous computing, autonomic computing, platform computing, edge computing, elastic computing, grid computing, …
Rich's Big Data Analytics Training
9/20/2018

7. Cloud Computing
Cloud provides processor, software, operating systems, storage, monitoring, load balancing, clusters and other requirements as a service
Pay as you go model of business
When using a public cloud the model is similar to renting a property than owning one.
An organization could also maintain a private cloud and/or use both.
Cloud transfers capital expenditure into operating expenditure.
Cloud computing models:
platform (PaaS), Eg., Windows Azure
software (SaaS), Eg., Google App Engine
infrastructure (IaaS), Eg., Amazon AWS
Rich's Big Data Analytics Training
9/20/2018

8. Windows Azure Enterprise-level on-demand capacity builder
Fabric of cycles and storage available on-request for a cost
You have to use Azure API to work with the infrastructure offered by Microsoft
Significant features: web role, worker role , blob storage, table and drive-storage
Platform as a service
Azure IDE is similar to Windows environment tools for cloud deployment of applications
Rich's Big Data Analytics Training
9/20/2018

9. Google App Engine
This is more a web interface for a development environment that offers a one stop facility for design, development and deployment Java and Python-based applications in Java, Go and Python.
Google offers the same reliability, availability and scalability at par with Google’s own applications
Interface is software programming based
Comprehensive programming platform irrespective of the size (small or large)
Signature features: templates and appspot, excellent monitoring and management console;
Free version to explore at:
Software as a service: Evolutionary Genetics Testbed
We will work on Google App Engine cloud in a later session (January).
Rich's Big Data Analytics Training
9/20/2018

10. Amazon Web Services (AWS)
Amazon.com, the online market place for goods, has leveraged the services that worked for their own business and has made them available as cloud services, amazon web services (aws)
We will get into the details of the various services aws offers and work on hands-on exercises for a select set of services
Rich's Big Data Analytics Training
9/20/2018

11. Enabling Technologies
Many technologies make computing on the cloud possible
Web services (interface)
Identity management
Virtualization: virtual machine images
Of course, the Internet, bandwidth increases and such other obvious across the board improvements
Rich's Big Data Analytics Training
9/20/2018

12. Identity Management7
A typical system like your personal computer or laptop or your server manages the identity of its users using “username” and “password” model
In the case of accessing a cloud services, we need something that is more robust, secure, and automatic for identity authentication. An important aspect of your data strategy is management of these.
Private key -- public key pair: your identity or your group’s or organization’s is mapped on to a key pair; use that to access cloud services
Credentials: access key and secret access key
Rich's Big Data Analytics Training
9/20/2018

13. What can you do with aws?
You can run nearly anything on AWS that you would run on physical hardware: websites, applications, databases, mobile apps, campaigns, distributed data analysis, media storage, and private networks.
Rich's Big Data Analytics Training
9/20/2018

14. Getting Started with AWS
Amazon Web Services (AWS) provides computing resources and services that you can use to build applications within minutes at pay-as-you-go pricing.
For example, you can rent a server on AWS that you can connect to, configure, secure, and run just as you would a physical server.
The difference is the virtual server runs on top of a planet-scale network managed by AWS.
Using AWS to build your Internet application is like purchasing electricity from a power company instead of running your own generator
Rich's Big Data Analytics Training
9/20/2018

15. AWS: Getting started
Get an account on aws.amazon.com/free -- the free tier for services
Next step is to study the documentation available on various services :
Follow the documentation with hands-on tutorials.
We will look at the list of the services offered by aws.
Extensive documentation is available with excellent contribution form the industry in Amazon Marketplace AMIs and Community AMIs.
Rich's Big Data Analytics Training
9/20/2018

16. AWS Console
Rich's Big Data Analytics Training
9/20/2018

17. Amazon EC2 http://aws.amazon.com/ec2/
EC2 (Elastic Compute Cloud) was the first service that amazon introduced.
Provides resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers.
Amazon EC2 is one large complex web service.
EC2 provides an API for instantiating computing instances with any of the operating systems supported.
It can facilitate computations through Amazon Machine Images (AMIs) for various other models.
Rich's Big Data Analytics Training
9/20/2018

18. Ec2 AMI and instances http://aws.amazon.com/documentation/ec2/
AMI: Amazon Machine Image
Rich's Big Data Analytics Training
9/20/2018

19. Amazon EC2 API
EC2 provides web services API for provisioning, managing, and de-provisioning virtual servers inside amazon cloud.
Applications anywhere on the Internet can launch a virtual server in the amazon cloud with a single web services call (either REST or SOAP WS call)
We will now create an instance on EC2;
This can be done automatically (by other applications) programmatically, remotely, from anywhere through EC2 API.

20. Ec2 Demo
In this demo we will instantiate a Windows machine and connect to it using public/private key pair.
On to Exercise 1
Rich's Big Data Analytics Training
9/20/2018

21. Simple Storage Service (S3)
Amazon Web Service
Free usage tier?
What it does
Amazon Simple Storage Service (Amazon S3)
Yes
5 GB storage
20,000 Get requests
2,000 Put requests
Stores and retrieves digital files.
Rich's Big Data Analytics Training
9/20/2018

22. Simple Storage Service (S3)
Amazon Simple Storage Service (Amazon S3) is storage for the internet. You can use Amazon S3 to store and retrieve any amount of data at any time, from anywhere on the web.
The bucket name you choose must be unique across all existing bucket names in Amazon S3. One way to help ensure uniqueness is to prefix your bucket names with the name of your organization.
Rich's Big Data Analytics Training
9/20/2018

23. Privacy and sharing an object
All objects by default are private. Only the object owner has permission to access these objects. However, the object owner can optionally share objects with others by creating a pre-signed URL, using their own security credentials, to grant time-limited permission to download the objects.
When you create a pre-signed URL for your object, you must provide your security credentials, specify a bucket name, an object key, specify the HTTP method (GET to download the object) and expiration date and time. The pre-signed URLs are valid only for the specified duration.
Anyone who receives the pre-signed URL can then access the object. For example, if you have a video in your bucket and both the bucket and the object are private, you can share the video with others by generating a pre-signed URL.
Rich's Big Data Analytics Training
9/20/2018

24. S3 Demo We will demo using S3Fox utility
We will also demo using Amazon console
Rich's Big Data Analytics Training
9/20/2018

25. Hosting a Web Site
A static website does not require server-side processing and relies only on client-side technologies such as HTML, CSS, and JavaScript.
Rich's Big Data Analytics Training
9/20/2018

26. AWS Services for web site hosting
Amazon Web Service
Free usage tier?
What it does
Amazon Simple Storage Service (Amazon S3)
Yes
5 GB storage
20,000 Get requests
2,000 Put requests
Stores files and can be configured to deliver them to browsers using an Internet URL.
Amazon Route 53 No
Attaches a custom domain name (e.g., to AWS resources.
Amazon CloudFront
50 GB data transfer
2,000,000 requests per month
Speeds up access to the website by caching the files in edge locations around the world.
Rich's Big Data Analytics Training
9/20/2018

27. Static website hosting architecture
Rich's Big Data Analytics Training
9/20/2018

28. Web-site with custom domain (costs money)
Rich's Big Data Analytics Training
9/20/2018

29. Web-site with performance mgt. (costs money)
Rich's Big Data Analytics Training
9/20/2018

30. Choose a domain name
When you host a website on Amazon S3, AWS assigns your website a URL based on the name of the storage location you create in Amazon S3 to hold the website files (called an S3 bucket) and the geographical region where you created the bucket.
For example, if you create a bucket called richs on the east coast of the United States and use it to host your website, the default URL will be
We will not use Route 53 and CloudFront for this proof-of-concept implementation.
Rich's Big Data Analytics Training
9/20/2018

31. Next steps
Open the Amazon S3 console at
Create 3 buckets in s3: richs.com, logs.richs.com
Upload the files of your static web page into richs.com bucket
Set the permissions of richs.com to allow others to view: In the policy edit window enter the code given below and save.
Rich's Big Data Analytics Training
9/20/2018

32. Permissions: policy code
{ "Version":" ", "Statement":[{ "Sid":"Allow Public Access to All Objects", "Effect":"Allow", "Principal": { "AWS": "*" }, "Action":["s3:GetObject"], "Resource":["arn:aws:s3:::richs.com/*" ] }
Rich's Big Data Analytics Training
9/20/2018

33. Enable logging and redirection
In the logging window enter logs.richs.com and /root in the next box
Right click on and redirect it to richs.com
Click on the endpoint address that shows up in properties window of richs.com
You should be able to see the static application.
Enjoy.
Rich's Big Data Analytics Training
9/20/2018

34. High-level architecture
Rich's Big Data Analytics Training
9/20/2018

35. Pricing: Estimated Monthly Bill for this site
If it is active with requests/client accesses is about $5.22/month, may be even less now
Rich's Big Data Analytics Training
9/20/2018

36. AWS Monthly Cost Calculator
Rich's Big Data Analytics Training
9/20/2018

37. Twitter Data Collection Demo
In this demo, we will collect live data using a pre-built stack of amazon services called the “CloudFormation”: this is a typical use of on-demand computing
Twitter data collection using CloudFormation
Note access to instance using PKI keypairs
Storing data in S3
MR computation on EMR: word count
Reference for the demos:
Rich's Big Data Analytics Training
9/20/2018

38. On to the Demo
Suppose your company recently released a new product and you want to assess its reception among consumers who use Twitter.
In this demo we will launch an AWS CloudFormation stack that provides a script for collecting tweets.
We will then store the tweets in Amazon S3.
Then we will run a simple wordcount algorithm to rank the words.
Rich's Big Data Analytics Training
9/20/2018

39. Run a virtual server on AWS: Linux server
Rich's Big Data Analytics Training
9/20/2018

40. Amazon Elastic Compute Cloud (EC2)
We completed an application using S3.
For this exercise we will use the EC2 that is preeminent product of aws
One can use EC2 to run and manage virtual servers on AWS. Your servers are launched as EC2 instances and initialized with a machine image called an AMI.
In the free usage tier, you can launch a t1.micro Amazon EC2 instance. Micro instances provide a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available.
A t1.micro instance is well suited for low-throughput applications and websites that consume significant compute cycles only occasionally.
For this exercise we will launch a Linux server using a Amazon Machine Image that is eligible for the free tier.
Rich's Big Data Analytics Training
9/20/2018

41. Amazon Machine Images (AMI)
Rich's Big Data Analytics Training
9/20/2018

42. Connect to the server
After you launch the instance you need putty to ssh into your server.
Note the dns for the launched server at the bottom half of the console: ec compute-1.amazonaws.com
Convert the richs.pem into .ppk private key
Launch a putty window and use all these items above and username ec2-user to log into the launched server.
Work on some Linux commands to check this new launch out. (sudo yum install emacs)
Make sure you terminate the server when you are done using it. This can be done from the EC2 dashboard
Rich's Big Data Analytics Training
9/20/2018

43. Putty connect to Linux Server
Rich's Big Data Analytics Training
9/20/2018

44. Run a virtual server on AWS: Windows server
Rich's Big Data Analytics Training
9/20/2018

45. Steps to acquire a Windows Server on aws
Similar to Linux instance:
Choose a free tier Windows 64 bit instance
Use the same security keys-pair
Set the security group to allow RDP, SSH
Connect using RDP (remote desktop protocol)
You can see the local system if you enable appropriate privileges when connecting
Transfer software and install and use the server.
Lets go on to the demo. Make sure you stop, and terminate the instance after done.
Rich's Big Data Analytics Training
9/20/2018

46. Summary
We looked at the capabilities of Amazon Web Services through 5 different examples;
Working with S3
Static web app
Twitter data collection using CloudFormation service
Launching an instance of Linux instance + working
Launching an instance of Windows instance + working
You can easily adapt and use any of these or other aws services at your workplace.
Rich's Big Data Analytics Training
9/20/2018

47. References & useful links
Amazon AWS:
AWS Cost Calculator:
Google App Engine (GAE):
B. Ramamurthy. Securing Business IT in the Cloud. Book Chapter in Security, Trust, Regulatory Aspects of Cloud Computing in Business Environments, Edited by S. Srinivasan, IGI Global, Hershey, PA, 2013.
Rich's Big Data Analytics Training
9/20/2018