JamesRH

 7 major AWS Services (  Amazon E-Commerce Service (ECS)  Amazon Elastic Compute Cloud Beta (EC2)  Amazon Historical Pricing  Amazon Mechanical Turk Beta  Amazon Simple Storage Service (S3)  Amazon Simple Queue Service (SQS)  Alexa Web Services  Originally planning to write against both EC2 & S3 but EC2 beta was closed until recently  There are EC2 scaling issues (confirmed by Bezos at Web2.0)  S3 Scaling well:  5B objects  Peak transaction rate:  Day: 923M  Second: 16,67  This demo app was written to learn the S3 interfaces, their strengths, and weaknesses 4/17/20152

 Went to aws.amazon.com  Signed into Amazon account  Same one used for Amazon shopping  Fill in additional contact information  agreed to terms and conditions  AWS provides Access Key & Secret Key  Tested using  Simple web app supporting most AWS operations via SOAP  Failed with: “Client.NotSignedUp”  showed up later complaining about credit card out of date  I corrected the payment issue and was up and running  User provisioning is fast and efficient  Overall, quick and easy process 4/17/20153

 Blob store with per blob metadata  Blobs supported up to 5GB  ~36.4 hour upload via Comcast “broadband”  Metadata up to 4k in name/value pairs (2k of user metadata)  Up to 100 ACLs on each object (part of system metadata)  Read, write, read_ACP, write_ACP, and full_control  Owner has full control and is not changeable  Each blob stored in a bucket  Bucket name space shared by ALL S3 users  All buckets have a single unchanging owner  Each blob has a URI identifier in a flat per-bucket namespace  Directory hierarchy simulated  Get(key=“C:/X/Y/Z.txt”) gets the file  Get(key=“C:/X”) gets all objects with leading “C:/X/” in URI  Get(key=“C:/X/”, separator=“/”) gets all objects with leading “C:/X/” but no subsequent “/” 4/17/20154

 Client libraries offered as samples for:  Java SOAP, Java REST, C# SOAP, C# REST, Ruby REST, Python REST, & Perl REST  BitTorent also supported  Docs at protocol layer rather than APIs  API implementations uneven without docs  C# SOAP has max send of 1M (can use DIME attachment for larger)  C# REST doesn’t support direct streaming  Source provided so quality & functionality issues all have solution 4/17/20155

conn = new AWSAuthConnection(key, secretKey); res = conn.delete(Bucket, key, null); Soap request Headers: - 10T01CX2MA2M9WD25002 NbT52AjYRzhHdA490B+T20KBYYg= T20:10:28.000Z 10T01CX2MA2M9WD25002-Archiver C:/JamesRH/dev/AWS/tst/test.tst*1 SOAP response Headers No Content 4/17/20156 REST Access Key:AWS : Base64(hmac-sha1(VERB, Content-MD5, Content type, Date, AMZ headers, resourceKey )) SOAP Model Doesn’t sign

 S3 Archiver is a version store  Registered directories & sub-directories contents are stored in S3  Each new file version is stored separately  Any version of any file can be restored in place  Multiple versions of a file can be restored with version suffixes on filename  Any file version can be deleted from S3  Roughly 3,500 lines of C# written against C# REST interface 4/17/20157  Performance:  12,220 files, 12,500 versions totaling 5.1 GB synchronized in under 30 seconds (local- only operation)  Actual transfers very slow ranging from roughly 1GB/hour to as low as 100 MB/hour (28 KBs)  Issues mostly network bandwidth-related on the perf side and standard internet flakiness on the reliability side

 Files stored in S3 with version numbers  C:\JamesRH\OpsFriendlyApps.docsx*10  Metadata as name/value pairs with each file:  Version  Size  Hash  Last Write  Create Time  Access Time  Archive Time  Fetching metadata for 1,000s of files slow so have a cache stored in S3  S3_Archiver.xml  Large scale testing required real data so implemented encryption  Single file operations slow and very uneven in performance so need to run all non-metadata-only operations asynchronously  On scale-up to 1000s of files S3_Archiver.xml gets large and slow so implemented compression  On scale-up to 10s of 1000s of files S3_Archiver.xml still slow to transfer so check hash and avoid transfers when possible 4/17/20158 C:/user/file1*1 C:/user/file1*2 C:/user/file2*1 C:/file3*1 cache cache*C:/usr/libary1 cache*C:/usr/library2

4/17/20159

 Error rate ~4 in 1,000 and similar across different networks:  Comcast:  Recv: 43 to KBs  Send: 5 to 52 KBs  Microsoft:  Recv : 317 to 10,935 KBs  Send: 437 to 5,556 KBs  Home Large File:  Send Fast/Slow ratio: 9.8x  Recv Fast/Slow ratio: 12.9x  Recv/Send ratio: 10.8x  Work Large File:  Send Fast/Slow ratio: 12.7x  Recv Fast/Slow ratio: 34.4x  Recv/Send ratio: 1.9x  Conclusion:  Send to receive difference ranges from a factor of 2 to an order of magnitude  2 orders of magnitude spread in throughput  Internet error rate high  5+ orders of magnitude higher latency than local disk 4/17/ Source: one week hourly perf & reliability probe of S3 at 1k, 10k, 100k, 1m, & 25m

4/17/ Kbs 4 errors Errors show as negative throughput

4/17/ Kbs Errors show as negative throughput 1 error

4/17/ Kbs Errors show as negative throughput 2 errors 3 errors

4/17/ Kbs Errors show as negative throughput 1 error

Summary of This Month's Activity as of March 14, 2007 Billing Cycle for this Report: March 1 - March 31, 2007 Usage Charges Rate Usage Totals Amazon Simple Storage Service $0.20 per GB of data transferred (details) GB $3.04 details $0.15 per GB-Month of storage used (details) GB-Mo $0.04 details Total:$3.08 All web services are sold by Amazon Digital Services, Inc. Summary of Last Month's Statement: February 1 - February 28, 2007 Total Charges for February 1 - February 28, 2007: $0.07  Amazingly inexpensive:  Developed a 3,000 line interactive system storing 1,000s of files and several GB stored with many test cycles at $3.11 4/17/201515

 Current bandwidth price (through May 31, 2007)  $0.20 / GB – uploaded  $0.20 / GB - downloaded  New bandwidth price (effective June 1, 2007)  $0.10 per GB - all data uploaded  $0.18 per GB - first 10 TB / month data downloaded  $0.16 per GB - next 40 TB / month data downloaded  $0.13 per GB - data downloaded / month over 50 TB  Data transferred between Amazon S3 and Amazon EC2 will remain free  New request-based price (effective June 1, 2007)  $0.01 per 1,000 PUT or LIST requests  $0.01 per 10,000 GET and all other requests  No charge for delete requests  Storage will continue to be charged at $0.15 / GB-month used 4/17/201516