Towards Autonomic Adaptive Scaling of General Purpose Virtual Worlds Deploying a large-scale OpenSim grid using OpenStack cloud infrastructure and Chef Matthew Delaney, Eleni Stroulia

OpenSim Architecture and Scalability Issues Types of virtual worlds OpenSim The Software Architecture of OpenSim Scalability issues Distributed Scene Graph

Types of virtual worlds There are two broad classes of virtual worlds  Massively Multiplayer Online Games  General Purpose Virtual Worlds

Massively Multiplayer Online Games  Content and behavior of the world is known beforehand and does not change often  The static nature of this type of virtual world can be exploited to improve performance

General Purpose Virtual Worlds  Content of the world can be altered at any time by some number of agents (users, scripted objects, etc)  The dynamic nature of general purpose virtual worlds make many types of optimizations difficult

Open Simulator  Built as an open source virtual world that implements the protocols used by Second Life  Until recently worked with the Second Life viewer  Has gained a large following and is used by virtual world enthusiasts, educational institutions, the US military, and companies (such as Intel and IBM)

Open Simulator  Virtual world is split into 256m^2 regions  This spatial partitioning makes it easy to assign chunks of land to servers  Issues  Limits user intaraction  Cross-region objects are problematic  What region does the object belong to?  Issues with collision detection and object interaction  Descision to use static partitioning a result of choices made by Second Life developers.

OpenSim Architecture -1

OpenSim Architecture -2

OpenSim Architecture -3  Adding new functionality is easy  New region modules can interact with any part of the simulator  Simulator centric  Most of the simulation and communication processes run on a single server.  When any part of the heartbeat slows the simulation as a whole slows  User experience is severely degraded once the simulator drops below 30 FPS

Scaling Up OpenSim: Objectives  Consistently maintain an immersive user experience for everyone  At least 30fps  Minimal latency  Accommodate new users  Avoid over-provisioning resources  This should happen automatically in a way that is transparent to the end user  Manually administering server nodes is slow and not sustainable

Scalability of OpenSim What does a large scale deployment look like? Each grid service can be split up and run on different machines Normally each simulator will be running on its own machine MySQL databases for grid services

Scalability of OpenSim: Barriers  Simulator-centric architecture prevents scaling by adding additional hardware  Unable to reassign regions to different simulators during run-time without severe impact on user experience  Different regions may be using a different set of region modules that could conflict with each other  If a region has active users, they must be moved to a temporary location while the region is reassigned to a new simulator  As a result, usually each region to a single simulator (over- provisioning)

Scalability of OpenSim: Towards a Solution Intel’s Distributed Scene Graph (DSG)  Split the major computation- and communication- intensive components of the simulator into distinct actors  Actors can run on their own hardware that is optimized for their purposes  Actors communicate with each other using a pub/sub model

OpenSim Architecture with DSG

Scalability of OpenSim with DSG Each grid service can be split into separate instances running on different machines Each service could be on its own machine

OpenSim on the Cloud with OpenStack and Chef What is OpenStack? What is Chef?

 An open source cloud operating system  Very similar to Amazon’s EC2 service  Provides on demand access to compute, storage and networking resources  API  Has its own robust RESTful API  Supports Amazon’s EC2 API  Has significant backing from a large number of large companies  IBM, Rackspace, NASA, RedHat, Intel, Cisco, and more What is OpenStack?

What is Chef?  Open-source systems-integration framework  Cloud automation  Deploy/remove server nodes on cloud infrastructure  Assign a set of roles to a node and automatically install and configure applications based on these roles  Obtain system and performance information for each node  Search for nodes based on role, system info, performance info, and more  Configuration management  Easy to change configuration of an application across hundreds of nodes

Towards Autonomic Adaptive Scaling of Virtual Worlds Proposed OpenSim grid deployment on the cloud using OpenStack First steps

Proposed OpenSim Grid Deployment On The Cloud Edge Clouds Core Cloud

Proposed OpenSim Grid Deployment On The Cloud Core Cloud  Low latency, high bandwidth connection to edge clouds  More computational and storage resources Edge Clouds  Each edge cloud located in geographically different areas  low latency to end users  Virtual world viewers connect to the edge cloud

First Steps  Automate grid deployment using existing architecture with Chef  Manually relocate regions to different simulators using Chef  Collect system usage information to get an idea of where additional resources are needed and where resources are being wasted  Automate grid deployment using DSG architecture with Chef  Manually add/remove/alter simulator actors using Chef  Automatically add/remove/alter simulator actors using Chef based on current system loads  Based on past usage information try to anticipate usage and adjust appropriately

References  Lake, Bowman, Liu; Distributed scene graph to enable thousands of interacting users in a virtual environment; 9 th Annual Workshop on Network and Systems Support for Games; 2010, 1-6  Liu, Bowman, Adams, Hurliman, Lake; Scaling Virtual Worlds: Simulation Requirements and Challenges; Proceedings of the 2010 Winter Simulation Conference  Gupta, Demers, Gehrke et al; Scalability for Virtual Worlds; IEEE 25 th International Conference on Data Engineering; 2009,  accessed on Oct. 18,  accessed on Oct. 20,  accessed on Oct. 20,

Questions? Thank You!