1. 1 Scalable Peer-to-Peer Virtual Environments Shun-Yun Hu CSIE, National Central University, Taiwan 2008/06/03

2. 2

4. 4

6. Massively Multiplayer Online Games MMOGs are growing quickly Multi-billion dollar industry 10 million subscribers for World of Warcraft 600,000 concurrent users, but 3,000 per world Can we scale to millions in the same world?

7. Imagine you start with a globe

8. Zoom in…

9. To Trier…

10. and to Universitat Trier

11. Right now it’s flat…

12. But in the near future…

13. Virtual Environments (VEs): A shared space

14. 14 Model for virtual worlds Many nodes on a 2D plane Message exchange with those within Area of Interest (AOI)‏ How does each node receive the relevant messages? Area of Interest

15. 15 A simple solution (point-to-point)‏ But…too many irrelevant messages N * (N-1) connections ≈ O(N 2 )  Not scalable! Source: [Funkhouser95]

16. 16 A better solution (client-server)‏ Message filtering at server to reduce traffic N connections = O(N)  server is bottleneck Source: [Funkhouser95]

17. 17 Current solution (server-cluster)‏ Still limited by servers. Expensive to deploy & maintain. Source: [Funkhouser95]

18. The Problem Client-server: resources limited by provisioning Resource limit [Funkhouser95]

19. The Solution Peer-to-Peer: resources grow with demand Resource limit [Keller & Simon 2003]

20. Outline Overlay management (VON) State management (VSM) Client-assisted service(FLoD)

21. Voronoi-based Overlay Network (VON)‏

22. Design Goals Observation: for VEs, the contents are messages from AOI neighbors Content discovery is a neighbor discovery problem Specific goals: Scalable  Limit per-node message traffic Responsive  Direct connection with AOI neighbors

23. 23 If you talk with your AOI Neighbors directly… games can be built But how to discover new neighbors?

24. 24 Voronoi Diagram 2D Plane partitioned into regions by sites, each region contains all the points closest to its site Can be used to find k-nearest neighbor easily Neighbors Site Region

25. 25 Design Concepts Identify enclosing and boundary neighbors Enclosing neighbors are minimally maintained Boundary neighbors mutually help to discover neighbors boundary neighbor (B.N.)‏L. Blue unknown neighborL. Green normal AOI neighborGreen E.N. & B.N.Pink enclosing neighbor (E.N.)‏Yellow selfWhite Area of Interest (AOI)‏Circle Use Voronoi to solve the neighbor discovery problem

26. 26 Procedure (MOVE)‏ 1)Positions sent to all neighbors, mark messages to B.N. B.N. checks for overlaps between mover’s AOI and its E.N. 2)Connect to new nodes upon notification by B.N. Disconnect any non-overlapped neighbors Boundary neighbors New neighbors Non-overlapped neighbors

27. Dynamic AOI Crowding within AOI can overload a particular node It’s better if AOI-radius can be adjusted in real time

28. 28 Demonstration Simulation demo Random movements (100 nodes, 1200x700 world)‏ Local vs. global view Dynamic AOI adjustment

29. Simulations C++ implementation of VON (open source VAST library) World size: 1200 x 1200 (AOI: 100) Trials from 200 – 2000 nodes Connection limit: 20 3000 time-steps (~ 300 simulated seconds, assuming 10 updates/seconds) Behavior model Random movement:random waypoint Constant velocity: 5 units/step Movement duration: random (until destination is reached)

30. Scalability: Avg. Transmission / sec

31. Scalability: Max. Transmission / sec

32. Voronoi State Management (VSM)‏

33. 33 State Management for VEs Besides positions, object states are important too Three main issues: Consistency control Load balancing Fault tolerance

34. A basic approach Let game states be managed by all clients Each client has two roles: peers & arbitrators i.e., Voronoi partitioning (we can use VON)

35. Problems with basic approach O(n 2 ) connections at hotspots Some cells have large sizes Constant ownership transfer

36. 36 VSM: solution ideas Connection overload→ Aggregators clustering Large cell-size → Virtual peers incremental transfer Constant transfers→ Explicit ownership transfer

37. 37 VSM: Consistency control Managing arbitrator receives and processes events Events are forwarded if necessary Updates sent to affected arbitrators, then peers

38. 38 VSM: Consistency control Managing arbitrator receives and processes events Events are forwarded if necessary Updates sent to affected arbitrators, then peers

39. 39 VSM: Consistency control Managing arbitrator receives and processes events Events are forwarded if necessary Updates sent to affected arbitrators, then peers

40. 40 VSM: Consistency control Managing arbitrator receives and processes events Events are forwarded if necessary Updates sent to affected arbitrators, (then peers)

41. 41 VSM: Load balancing Traditional: high-capacity nodes first, then adjust VSM: low-capacity nodes first, then cluster Overload: ask for aggregator, submit control Underload: disintegrate, release control

42. 42 VSM: Fault tolerance Regular arbitrator: Pick backup arbitrator, backup states Backup transfers ownership to enclosing arbitrators Aggregators: Pick backup aggregators Take over original if failed Choose new backup

43. Peer-to-Peer 3D Streaming

44. 44 Background MMOGs today need DVD installations Too slow and unpractical for: Larger and more dynamic worlds (TBs data) More numerous worlds (Web 3D) Content streaming is needed 80% - 90% content is 3D (e.g., 3D streaming)

45. 3D streaming Continuous and real-time delivery of 3D content to allow user interactions without a full download. base & refinement pieces Base123 Refinements User (Hoppe 96)

46. Scene streaming Multiple objects Object determination & prioritization [Teler & Lischinski 2001]

47. 47 3D streaming vs. media streaming Video / audio media streaming is very matured User access patterns are different for 3D content Highly interactive  Latency-sensitive Behaviour-dependent  Non-sequential How to scale to millions of concurrent users?

48. 48 Observation Limited & predictable area of interest (AOI)‏ Overlapped visibility = shared content

49. 49 overlapped visibility = shared content

50. Challenges for P2P 3D streaming Appropriate peer grouping Matching interests / needs Matching capabilities Dynamic group management Interest groups are dynamic(non-sequential) Real-time constraints(latency-sensitive) Minimal server involvement Visibility determination (object selection) Request prioritization (piece selection)

51. FLoD [Infocom 2008] VE partitioned into cells with scene descriptions Assumes P2P overlay that provides AOI neighbors star: selftriangles: neighbors circle: AOIrectangles: objects

52. 52 Neighbor discovery via VON Boundary neighbors New neighbors Non-overlapped neighbors [Hu et al. 06] Voronoi diagrams identify boundary neighbors for neighbor discovery

53. 53 Prototype experiment Progressive models in a scene (by NTU) Peer-to-peer AOI neighbor requests(by NCU) Found matching client upload / download

54. 54 Simulation setup Environment 1000x1000 world, 100ms / step, 3000 steps client: 1 Mbps / 256 Kbps, server: 10 Mbps (both)‏ Objects Random object placement (500 objects)‏ Object size based on prototype User behavior Random & clustering movement (1.5 * ln(n) hotspots)‏

55. 55 Server bandwidth usage

56. 56 Client bandwidth usage

57. Impacts of P2P VEs… No server as bottleneck  scalable Commodity hardware  affordable 2D web  3D web Earth-scale virtual worlds (millions/billions of people)‏

58. Unresolved issues

59. Issues for creating VEs Consistency Interactivitymultiplayer Security Scalability Persistency massively multiplayer Reliability Interoperability Content 3D web

60. Meshing physical & virtual topologies Client 2 Client 1

61. A generic pub/sub scenario pub sub

62. 62 Q&A VON: A Scalable Peer-to-Peer Network for Virtual Environments IEEE Network, vol. 20, no. 4, Jul./Aug. 2006 FLoD: A Framework for Peer-to-Peer 3D Streaming IEEE INFOCOM, Apr. 2008 Thank you! http://vast.sourceforge.net http://ascend.sourceforge.net

63. Unresolved issues Overlay management Topology-aware, capacity-matching superpeers Flexible publication / subscription Direct vs. forwarding deliveries State management Load balancing (high user density) Persistency Security Client-assisted services (e.g., P2P 3D streaming) Source nodes discovery Visualization vs. networking priority matching LOD considerations

64. Other issues Common API Shared simulator / platform Interoperability

65. Extension: VoroCast Pack reduction via forwarding Headers reduction Data compression & aggregation