CACHING: tuning your golden hammer By Ian Simpson from Mashery: an Intel Company

What’s being covered (or not)  Caching within the JVM  Not focusing on external caches like memcached  Not focusing on sharding  Both are important, but we only have an hour

Motivation for this talk  Overall, caching is great!  More performance!  Greater scalability!  When do we have too much of a good thing?  How do we know if/when we’re doing it wrong?

When is more not better?  Ultimately caching isn’t free (but is usually cheap)  Some caches are not effective (low cache/hit ratio)  Some caches hold on to too much data  Some caches cause resource contention  If used incorrectly, caches can destroy performance

Memory: abundant but not free  When we cache, we store data in memory  Generally a replacement for fetching data from (slow) persistent storage  Servers have lots of memory!  The JVM only handles so much memory gracefully* * This varies based on the garbage collection strategy and JVM vendor, which is beyond the scope of this talk

JVM heap  Split between young, tenured, and permanent  Objects start in young gen  Most objects in young gen are short lived and GC’d  Objects in young that survive GCs go to tenured  Objects in tenured are long lived (e.g. cached data)

Heap o’ trouble  Tenured grows as more objects survive  GCs cause “stop the world” events, or pauses  GCs on tenured get more costly the larger it gets  Long GC pauses can cause performance problems  Full heap can cause thrashing (lots of GCs)

Cache structure and data retention  Usually caches are maps of keys to values  Under the covers, often Map  Anything you put in a Map is strongly referenced  Any reachable strong reference can’t be GC’d  Caches need to be reachable to be useful  Caches don’t automagically clean up stale data

How do we solve heap issues?  Don’t cache more than is necessary  Cache/hit ratio should be high (> 80%)  Understand how and when data is used  Estimate how large your data is or will be  Use background thread to clean stale data  Guava has support for this

Can I just not use the heap?  Yes! You can do off heap caching!  No! It’s not particularly straight forward!  On 32-bit JVM, 4GB (or 2) – max heap available  On 64-bit JVM, all available memory – max heap  Space is part of the Java process  Not managed by the garbage collector

What’s so difficult about off heap?  You have to manage your own memory  Allocated as a block via java.nio.ByteBuffer  Can also be allocated via sun.misc.Unsafe  You need to know where things are in the array  Serialization into and deserialization from byte[]  Cost to allocate block of memory

Blocking vs non-blocking cache

 Cache miss means data has to be loaded  Loading data means blocking threads requesting it  Generally unavoidable unless returning nothing is OK  Cache hit on stale data means reloading  Blocking on reloading data can be optional  Return stale data, reload asynchronously

Blocking cache: pros and cons  Pros  Stale data is predictably stale  Less resource intensive than loading asynchronously  Cons  Can have huge latency on popular keys across threads  Read mutexes can result in serial read performance  Better with many keys and low key contention

Non-blocking cache: pros and cons  Pros  Can return stale data and load in the background  No read mutex required!  Cons  Requires a thread pool for asynchronous loading  Data is unpredictably stale  Better with fewer keys and high key contention

Which is which? What do I use?  EhCache is a blocking cache  Guava cache can be either  Neither is the best choice in all cases  Data characteristics will help you decide  Not sure what to use? Talk to your coworkers!

Example cases  Product catalog with millions of items  Blocking  Subset of products on the homepage  Non-blocking  Subset of products on sale for limited time  Questionable – contention and staleness both important  Split data on TTL, static vs dynamic  Refresh price/inventory frequently and asynchronously

Local vs distributed caching

 Local means each server has its own cache  Distributed means multiple servers share cache  Local is very simple  Distributed is very complex  Greater detail of distributed is outside scope

Local: pros and cons  Pros  Basically just a decorated Map  No serialization overhead  Good ones available for free  Cons  Less efficient: more cache misses and memory usage  In large clusters, cache/hit can suffer greatly  In clusters, consistency becomes a problem

Distributed: pros and cons  Pros  Fewer cache misses in clusters; improves with size  Consistency across cluster  Great for splitting up large data sets  Cons  Complex: more data management and network chatter  Serialization overhead for remote retrieval  Enterprise solutions cost $$$, free ones more barebones

Which one do I use?  Again, no one perfect solution; depends on data  Local is quick and dirty  Distributed is better for large data sets  Distributed is usually better for clusters  Distributed comes at a cost of complexity and $$$  Not sure which one? Talk to your coworkers!

Example cases  CMS with large view templates  Distributed (size and consistency)  Display data like greetings, canned messages, etc  Local (small, changes infrequently)  Session store for user behavior  Questionable – consistency and resource concerns  Not mission critical data, but good to have  Local with sticky sessions an option

Getting closer with your data

Questions you should ask  How expensive is it to load?  Size: IO/memory bound; electrons only move so fast  Cost: high DB or CPU load means you must cache  Frequency: death by 1000 cuts  How often will I hit cache?  Example: static content often means high cache hit ratio  Example: highly diverse data means low cache hit ratio

Preparation through estimation  Estimate how data will grow  Number of new records per month?  How much is considered active?  Estimate how diverse data will become  Thousands of unique entries? Millions?  Subset of data that is more popular?  Work with product owners on expected use of data

And now for a recap!

A recap: fundamentals  Caching is great, but it’s not free  Get to know your data  Consider how much memory your app will use  Consider how expensive it is to load data  Take cache/hit ratios seriously  Not everything needs a cache

A recap: choices to make  Non-blocking: great for bounded data  Blocking: great for diverse data  Non-blocking: uses more resources, more stale data  Blocking: bad when keys have high contention  Local: simple, cheap, but can impact memory more  Distributed: great for big data sets & large clusters

Getting runtime info  Track cache statistics via JMX  Cache hits and misses  Total objects in memory  Tools like Nagios and AppDynamics can track JMX  Guava doesn’t have JMX, but it’s easy to add

BUT WHICH ONE DO I USE?!  Ultimately, you’ll have to figure it out  … with the help of your coworkers  Before you can decide on what to use, you need to understand the characteristics of your data

Shameless self promotion  Website:  Has several cache and memory related posts  Github Gists:  Lots of code samples (referenced in blog posts)  Interested in these problems?  Mashery is hiring!  API management for companies big and small  Scalability is core to our business

Obligatory “Questions” slide  Well?