1. Google and Cloud Computing
王咏刚
Google 资深工程师

2. Agenda The Internet: From Hardware to Community
The Innovation: A Computing Cloud
Breakthroughs for Cloud Computing
Google Apps for Cloud Computing
Google Infrastructure for Cloud Computing

3. From Hardware to Community
The Internet
From Hardware to Community

4. The Internet: From Hardware to Community
MySpace
Facebook
开心网
校内网 ……

5. What Do Today’s Users Want?
Accessibility
Access from anywhere and from multiple devices
Shareability
Make sharing as easy as creating and saving
Freedom
Users don’t want their data held hostage
Simplicity
Easy-to-learn, easy-to-use
Security
Trust that data will not be lost or seen by unwanted parties

6. The Innovation
A Computing Cloud

7. Cloud Computing

8. Attributes of Cloud Computing
Data stored on the cloud
Software & services on the cloud - Access via web browser
Based on standards and protocols - Linux, AJAX, LAMP, etc.
Accessible from any device
Personal PC
Client Server
Cloud Computing
Hardware Centric
Software Centric
Service Centric

9. Breakthroughs for Cloud Computing

10. Breakthroughs for Cloud Computing
User-Centric 1
Task-Centric 2
Powerful 3
Intelligent 4
Affordable 5
Programmable 6

11. User Centric Data stored in the “Cloud”
Data follows you & your devices
Data accessible anywhere
Data can be shared with others
music
preferences
maps
news
contacts
messages
mailing lists
photo s
calendar
phone numbers
investments

12. Example : GMail Just a web browser and your account with password!
San Francisco, Monday
Home, Wednesday
Beijing, on travel
Just a web browser and your account with password!
Once you login, the device is “yours”.
Data stored on remote servers in the “cloud” (with large capacity)

13. Use Google Docs to Solve a Task
Task = “Teachers creating a departmental curriculum”
Changes instantly appear to other collaborators
Access your docs from anywhere
Chat with others in real time

14. Communication Task – Email, Chat, Contacts, Chat History

15. Task: Collaborate on Spreadsheet – Communicate
Chat with others editing the spreadsheet

16. Task: Collaborate on Spreadsheet – Collaborate
Invite others to collaborate on the spreadsheet

17. Task: Collaborate on Spreadsheet – Publish
Invite others to view the spreadsheet

18. You can also easily organize all your common tasks

19. Cloud Computing is Powerful: It can do what no PC can do
Example: Google Search
Is Google Search faster than search in Windows/Outlook/Word?
And Google Search must be much harder….
How much storage does it take to store all of the web pages?
100B pages * 10K per page = 1000T disk!
Cloud computing has at its disposal
Essentially infinite amount of disk
Essentially infinite amount of computation
(Assuming they can be parallelized)

20. Web Page Search  Universal SearchD E
1st Generation: era of single search – not diverse
2nd Generation: era of vertical search – too complex
3rd Generation: an era of Universal Search

21. From vertical search to universal search
Integration of user experience A B C D E

22. Universal Search Example

23. Universal Search Example

24. Cloud Computing Infrastructure

25. … GFS Architecture GFS Master MSN 19% Client Masters Replicas
Enter Title of Presentation Here
GFS Architecture
GFS Master
MSN 19%
Client
Masters
Replicas
GFS Master
Google 48%
Client
Client
Client C0 C1 C1 C0 C5
Yahoo 33% …
Client C5 C2 C5 C3 C2
Chunkserver 1
Chunkserver 2
Chunkserver N
Files broken into chunks (typically 64 MB)
Master manages metadata
Data transfers happen directly between clients/chunkservers 25
Google Confidential 25

26. … Typical Cluster Lock service GFS master Scheduling masters Machine 1
Machine N
User
app1
User
app1
User
app3
User
app3
User app2 …
User app2
Scheduler
slave
GFS
chunkserver
Scheduler
slave
GFS
chunkserver
Scheduler
slave
GFS
chunkserver
Linux
Linux
Linux 26

27. MapReduce 27

28. Programmer specifies two primary methods:
More specifically…
Programmer specifies two primary methods:
map(k, v) → <k', v'>*
reduce(k', <v'>*) → <k', v'>*
All v' with same k' are reduced together, in order.
Usually also specify:
partition(k’, total partitions) -> partition for k’
often a simple hash of the key
allows reduce operations for different k’ to be parallelized 28

29. Distributed multi-level map Fault-tolerant, persistent Scalable
BigTable
Distributed multi-level map
With an interesting data model
Fault-tolerant, persistent
Scalable
Thousands of servers
Terabytes of in-memory data
Petabyte of disk-based data
Millions of reads/writes per second, efficient scans
Self-managing
Servers can be added/removed dynamically
Servers adjust to load imbalance 29

30. BigTable: Basic Data Model
Distributed multi-dimensional sparse map
(row, column, timestamp)  cell contents
Good match for most of our applications
COLUMNS
“contents”
ROWS … t1 … t2
TIMESTAMPS
“<html>…” t3 30

31. BigTable: System Architecture
Bigtable client
Bigtable cell
Bigtable client library
Bigtable master
performs metadata ops, load balancing
Open()
Bigtable tablet server
Bigtable tablet server
Bigtable tablet server
serves data
serves data
serves data
Cluster Scheduling Master
GFS
Lock service
handles failover, monitoring
holds tablet data, logs
holds metadata, handles master-election

32. Thanks
Q&A