Designing and Delivering Scalable and Resilient Web Services Ron Jacobs Sr. Technical Evangelist, Microsoft

Agenda The ProblemA Possible SolutionNew PossibilitiesWhat to do next

Simple Do the simplest thing that will possibly work

Scalability Able to support the required quality of service as the system load increases -Wiktionary

Typical Web Architecture Need to get routed to same machine (i.e. sticky sessions) Users Each machine round trips for data Data that is cached is stored in the memory of one server Web Tier CPU and disk can get saturated due to traffic Service access is slow Data Tier

Web Explosion Database IIS/ASP.NETIIS/ASP.NET Application Web Site’s too slow!! Database is hot!! Where did my shopping cart go? IIS/ASP.NETIIS/ASP.NET Application IIS/ASP.NETIIS/ASP.NET Servers are crashing Services are slow

Agenda The ProblemA Possible SolutionNew PossibilitiesWhat to do next

Data Near Processing Database Smart Client Browser Cache

Good but… Cache is scoped to machine / process – Machines die – Processes recycle Cache memory is limited

What if? You could have as much cache as you wanted? You could share a giant cache across servers, services and even clients? What if this was something you could simply add to the platform for 1 free? 1 Some features may require certain editions of Windows Server

Windows Server AppFabric AppFabric CACHING MONITORING WORKFLOW HOSTING SERVICE HOSTING SCALE OUTHIGH AVAILABILITYMANAGEMENT AppFabric Cache – Formerly known as Code Name “Velocity”

Unified Cache View What is AppFabric Caching? An explicit, distributed, in-memory application cache for all kinds of data Caching clients can be across machines or processes Clients Access the Cache as if it was a large single cache Cache Layer distributes data across the various cache nodes

AppFabric Cache No need for sticky sessio ns Users Cached data available to all servers Web Tier Multiple machines means scale and high- availability Lots of cache memory Cachin g Tier Reduces load on database Duplicate Service Calls eliminated Data Tier

Data Distribution - Partitioned Cache … Web Tier Cache Tier Cache Service AA CC EEDD BB GGIIHHFF Scale on Data Size - More machines => More memory to cache Scale on Cache Throughput - More machines => keys distributed across more machines => better throughput ASP.Net App Caching Client ASP.Net App Caching Client ASP.Net App Caching Client AACC EEDDBBGG IIHH FF

Scale Test OutputLoadLoad ThroughputThroughput LatencyLatency 1 Cache Server As load increases, throughput fails to scale latency increases Caching Tier

Add Second Cache ServerThroughputThroughput Load Max Throughput increases Latency decreases Caching Tier LoadLoad LatencyLatency

Add Third Cache ServerLoadLoad ThroughputThroughput LatencyLatency Caching Tier

Associated Press Caches metadata and news Serves 16 million hits per day Increased the amount of cached data 6 times.

System.Web.Cache

AppFabric DataCache

Deployment Caching Service Configuration Store Install AppFabric Copy Client DLLs … Application <host name="BL1CDB “ cachePort="22233" cacheHostName="DistributedCacheService"/> ….. <host name="BL1CDB “ cachePort="22233" cacheHostName="DistributedCacheService"/> ….. Caching Access Layer.NET 3.5 SP1 OR.NET 4 Configure AppFabric.NET 4 Update Web.config

Usage Pattern – Cache Aside (Explicit Caching) // Read from Cache Toy toyObj = (Toy) catalog.Get("toy-101"); // Read from Cache Toy toyObj = (Toy) catalog.Get("toy-101"); Application Caching Service // If Not present in the cache if (toyObj == null) { // Read from backend.. toyObj = ReadFromDatabase(); // Populate Cache catalog.Put("toy-101", toyObj); return toyObj; } // If Not present in the cache if (toyObj == null) { // Read from backend.. toyObj = ReadFromDatabase(); // Populate Cache catalog.Put("toy-101", toyObj); return toyObj; } Caching Access Layer

Administration PowerShell cmdlets are used to administer the cache cluster Rich set of cmdlets for – Cache cluster management – Cache creation and monitoring

Hello AppFabric Cache

Using PowerShell Remember – PowerShell can also be called from.NET Code!

AppFabric Cache Codeplex Tool

Security Domain Based Security Option – Domain Account / Local Account based Authentication – Only authorized servers can join the cluster – Only authorized clients can connect to the cluster Transport Level Security – Turn on/off Signing or Encryption Can turn off Cache Security – Use Firewalls, IPSec, VLANs to protect cache grant-cacheallowedclientaccount RedDomain\Machine1$ grant-cacheallowedclientaccount RedDomain\John

AppFabric Caching Logical Hierarchy Host – Physical processes hosting AppFabric Caching instance. Named Caches – Can span across machines – Defined in the configuration file Cache Item – Key, Payload (Object ), Tags, TTL, Timestamps, Version Regions – Physically co-located Container of Cache Items – May be implicit or explicitly created Regions Region A Key Payload Tags 121 xxxx “Toy” “Child” 123 yyyy “Toy” “Chair”.. Machine -> Cache Host -> Named Caches -> Regions -> Cache Items -> Objects AppFabric Caching Service Named Cache : Product Catalog Named Cache : Electronics Inventory AppFabric Caching Service

AppFabric Caching API // Create instance of cachefactory (reads appconfig) DataCacheFactory fac = new DataCacheFactory(); // Get a named cache from the factory DataCache catalog = fac.GetCache("catalogcache"); // Simple Get/Put catalog.Put("toy-101", new Toy("Puzzle",.,.)); // From the same or a different client Toy toyObj = (Toy)catalog.Get("toy-101"); // Simple Get/Put catalog.Put("toy-101", new Toy("Puzzle",.,.)); // From the same or a different client Toy toyObj = (Toy)catalog.Get("toy-101"); // Region based Get/Put catalog.CreateRegion("toyRegion"); // Both toy and toyparts are put in the same region catalog.Put("toy-101", new Toy(.,.), “toyRegion”); Catalog.Put("toypart-100", new ToyParts(…), “toyRegion”); Toy toyObj = (Toy)catalog.Get("toy-101“,"toyRegion"); // Region based Get/Put catalog.CreateRegion("toyRegion"); // Both toy and toyparts are put in the same region catalog.Put("toy-101", new Toy(.,.), “toyRegion”); Catalog.Put("toypart-100", new ToyParts(…), “toyRegion”); Toy toyObj = (Toy)catalog.Get("toy-101“,"toyRegion");

Access APIs – Tagging Items Tag hotItem = new Tag("hotItem"); catalog.Put("toy-101", new Toy("Puzzle"), new Tag[]{hotItem}, “toyRegion”); catalog.Put("toy-102", new Toy("Bridge"), “toyRegion”); // From the same or a different client List > toys = catalog.GetAnyMatchingTag("toyRegion", hotItem); Tag hotItem = new Tag("hotItem"); catalog.Put("toy-101", new Toy("Puzzle"), new Tag[]{hotItem}, “toyRegion”); catalog.Put("toy-102", new Toy("Bridge"), “toyRegion”); // From the same or a different client List > toys = catalog.GetAnyMatchingTag("toyRegion", hotItem);

Types of Data ReferenceActivityResource Primary Read OnlyRead-Write Not sharedRead-Write, Shared Catalog DataShopping CartAuction Data/Seat Assignment Web Tier Distributed Cache Shopping Cart Grocery Catalog Grocery Inventory Grocery Shop

Application AppFabric Caching Client Reference Data – Performance Catalog data doesn’t change often Unnecessary network cost to access from different machines Solution – Local Cache Put(K2, v3) Routing Table Cache2 Cache1 Primary for K2,V2 K2, V2 Primary for K1,V1 K1, V1 Cache3 Primary for K3,V3 K3, V3 AppFabric Caching Client Local Cache Routing Table K2, V2 Get(K2) K2, V3

Reference Data – Bulk Get Bulk Fetch from region – k ops per second – Fewer network calls Catalog.BulkGet( new List (){“toy-101”, “toy-102”}, “toyRegion”); Catalog.BulkGet( new List (){“toy-101”, “toy-102”}, “toyRegion”);

Activity Data – Session Integration … Session State stored in AppFabric Caching Cache Service Caching Service Load Balance Requests No more sticky routing Load Balance Requests No more sticky routing Caching Service Scale your Session Store Dynamically Dynamically Highly Available Drop in AppFabric Caching SessionStoreProvider Allows session state to be shared amongst multiple applications Application Caching Access Layer Application Caching Access Layer <add name="SessionStoreProvider" type=“Microsoft.Data.Caching.DataCacheSessionStoreProvider, ClientLibrary“ cacheName=" "/> <add name="SessionStoreProvider" type=“Microsoft.Data.Caching.DataCacheSessionStoreProvider, ClientLibrary“ cacheName=" "/>

Application (K2, V2) Cache2 Cache1 Cache3 Primary for Activity Data - Availability Get(K2) Primary for K3, V3 AppFabric Caching Client Routing Table K2, V2 PUT Secondary for K2, V2 K1, V1 Secondary for K3, V3 Secondary for K1, V1 AppFabric Caching Client Routing Table K2, V2 Replication Agent K2, V2

Resource Data - Optimistic Locking GetCacheItem returns a version object Every update to an object internally increments it's version Supply the version obtained along with the Put/Remove Put/Remove will succeed only if the passed in version matches the version in the cache Version Based Update TimeClient1Client2 (Different Thread or process) T0 CacheItem item = catalog.GetCacheItem(“PlayerRegion”, ”Zune”); CacheItem item = catalog.GetCacheItem(“PlayerRegion”, ”Zune”); T1((ZuneObject)item.Object).inventory --; T2 catalog.Put(“PlayerRegion”, “Zune”, item.Object, item.Version); T3 catalog.Put(“PlayerRegion”, “Zune”, item.Object, item.Version); // Version mismatch // Client must retry again Two clients access the same item Both update the item Second Client gets in first; put succeeds because item version matches; atomically increments the version First client tries put; Fails because the versions don’t match First client tries put; Fails because the versions don’t match

K1 Resource Data - Pessimistic Locking Take locks on non-existent keys – Allows you to co-ordinate calls for data Client1: GetAndLock ("k1") Client1: GetAndLock ("k1") Client2: GetAndLock ("k1") Client2: GetAndLock ("k1") Client3: Get ("k1") Client3: Get ("k1") Regular Get succeeds GetAndLock gets lock handle Other GetAndLock on same item fails

Data Race GET

Lock Non-Existent Key GET/LOCK Cache Service Caching Service Cache Service

Composite Race CALLWAIT Cache Service Caching Service Cache Service

Composite Race PUT UNLOCK GET Cache Service Caching Service Cache Service

Resource/Activity Data – Tracking Changes Cache Event notifications Register on any client to notify changes Batched Notifications DataCache.RegisterCacheLevelCallback( int filter, DataCacheChangeCallback delegate); DataCache.RegisterRegionLevelCallback( String region, int filter, DataCacheChangeCallback delegate); DataCache.RegisterKeyLevelCallback( String region, String key, int filter, DataCacheChangeCallback delegate); DataCache.RegisterCacheLevelCallback( int filter, DataCacheChangeCallback delegate); DataCache.RegisterRegionLevelCallback( String region, int filter, DataCacheChangeCallback delegate); DataCache.RegisterKeyLevelCallback( String region, String key, int filter, DataCacheChangeCallback delegate);

Application Cache2 Cache1 Primary for K2, V2 Primary for K1, V1 Cache3 Primary for K3, V3 Scalable Notifications AppFabric Caching Client Routing Table Register Notification for Key “K3" Map Keys to Partition Poll Required Nodes Nodes Return List of Changes LSN Order Partition: P2 Last LSN: 19 Partition: P2 Last LSN: 19 Call Delegate Store Last LSN Call Delegate Store Last LSN Change Log Partition P1 1Add K2 2Del K32 Change Log Partition P1 1Add K2 2Del K32 Change Log (Partition P2) 18Del K32 19Del K43 Change Log (Partition P2) 18Del K32 19Del K43 Change Log 33 Add K1 34 Del K22 Change Log 33 Add K1 34 Del K22

Agenda The ProblemA Possible SolutionNew PossibilitiesWhat to do next

Data Center Pre-Fetch Hospital

Data Center Pre-Fetch HospitalRemote Clinic Slow!! WAN

Data Center Pre-Fetch HospitalRemote Clinic Cache Service WAN

Agenda The ProblemA Possible SolutionNew PossibilitiesWhat to do next

Web Platform Installer

Select Enterprise

Install AppFabric

endpoint.tv

AppFabric on MSDN