1. Best practices around Dynacache Jonathan Marshall WebSphere Technical Professional marshalj@uk.ibm.com

2. 2 Acknowledgements Based on presentations by:  Geoff Tindall – WebSphere level 2 support  Rama Boggarapu – WebSphere level 2 support

3. 3 Agenda  Why do I need caching?  What is Dynamic Cache?  A peak under the covers of WebSphere Application Server  Replicating the Dynamic Cache  Monitoring the Cache  Troubleshooting Tips  Where do we go from here?

4. 4 Why Cache? A cache allows you to get stuff faster and helps you avoid doing something over and over again (which may be redundant and may not make sense) (far away) (near) (happy) A cache allows you to get stuff faster and helps you avoid doing something over and over again (which may be redundant and may not make sense)

5. 5 Why Cache? Improving the performance Tuning the application? Tuning the environment? Tuning the backend resources? Not doing something at all?

6. 6 Introducing Dynamic Caching Services (dynacache)

7. 7 Enabling Dynamic Cache

8. 8 What does it cache?  Each CacheEntry will have CacheId and CacheData  In addition to CacheId and CacheData, CacheEntry will have other information like dependencyId’s, timeout information, replication type and various other information.  CacheMonitor.ear application can be used to view most of the information about a CacheEntry.

9. 9 Where does it cache data? The Cache Instance  An application uses a cache instance to store, retrieve, and share data objects within the dynamic cache.  Two types of custom cache instances can be configured:  Servlet Cache Cachespec.xml  Object Cache API based cache, com.ibm.websphere.cache.DistributedMap  Three methods to create custom cache instance  Administrative console (under Resources)  Properties file (cacheinstances.properties)  Resource reference in web.xml

10. 10 Dynamic Cache Configuration for Default Cache Instance

11. 11 Implementation inherited from IBM Research Disk cache size can be controlled in terms of space on the disk and # of entries. LRU & Size based Disk eviction algorithms can configured to specify the criteria for eviction. 3 Performance modes HIGH, CUSTOM & LOW depending on the JVM free heap space available. – High..Buffers all disk metadata in memory. – Custom/Balanced… Buffers some/most disk metadata in memory. – Low … Buffers NO metadata in memory. Note this will be deprecated in v8. FlushToDiskOnStop provides for the cache to be persisted. One of the strongest features of Dynacache. Differentiates from other open source solutions as well as eXtreme Scale. Disk Offload

12. 12 Disk Offload  Configure a new cache instance  Resources – Cache Instances – Servlet Cache Instance – new  A jndi name must be given to the cache instance which will be used to refer back to the instance in cachespec.xml.  Check ‘Enable disk offload’ and specify a disk off load location, cache size and cache entry limits.

13. 13 How does it choose what to cache?  Cachespec.xml is a deployable XML cache policy file that contains caching rules  Placed in WEB-INF directory  Specify the cache-instance name where the cache rules to be applied  What to be cached? command/servlet/webservice/JAXRPCClient/static/portlet  How to identify an item in cache? Cache-Id  Where to cache? Memory only/ Disk  When to invalidate? Timebased/invalidation rules  How to handle dependencies? Dependency rules for invalidation

14. 14 Cache ID  A cacheID can be comprised of the following Request parameters and attributes The URI used to invoke the servlet or JSP Session information Cookies Pathinfo and servlet path Http header and Request method Servlet/JSP result caching can be used to cache both whole pages or fragments. Example /reference/Time.jsp:format=analog Web Service – getStockPrice but NOT setStockPrice

15. 15 Example cachespec.xml (1/4) – Define the entry /newscontroller servlet * http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/welc6tech_dyn_dev.html

16. 16 Example cachespec.xml (2/4) – Define the cache id /newscontroller servlet view true true false * http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/welc6tech_dyn_dev.html

17. 17 Example cachespec.xml (3/4) – Define the dependencies /newscontroller servlet view true true false category true * http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/welc6tech_dyn_dev.html

18. 18 Example cachespec.xml (4/4) – Define the invalidation /newscontroller... category update true true * http://publib.boulder.ibm.com/infocenter/wasinfo/v7r0/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/welc6tech_dyn_dev.html

19. 19 Fragment Caching The content of A.jsp is composed of its own content plus 3 jsps(fragments) B.jsp,C.jsp and D.jsp Often a mix of static and dynamic consume-subfragments tells the cache to store fragments allows fragments to explicitly not be cached

20. Going under the covers: Data Replication Service

21. 21 Data Replication Service - DRS  Data replication service (DRS) is an internal WebSphere Application Server component that replicates data.  Transport for sending data from one managed server to another.  Uses HAManager and DCS data stack frameworks to accomplish replication.  Has the notion of replicas  Data transfer channel can be encrypted. In practice no-one does this.  Benefits  Used by DynaCache, HTTPSession, Stateful Session Beans and SIP  Challenges  DRS Bootstrap will be expensive, if aggressive replication occurs during startup.  DCS uses a star topology for synchrony, can result in scalability bottlenecks.

22. 22 High Availability Manager New feature in V6 Collection of services Highly Available Singleton Services Low-level Replication Abstraction Shared State Bulletin Board Several runtime features depend on HA Manager services HA Manager runs in every single JVM in the cell HA Manager services are only available between JVMs that are part of the same core group High Availability Manager Distribution and Consistency Services (DCS) Reliable Multicast Messaging (RMM) Transaction Service Workload Management (WLM) Data Replication Services (DRS) Messaging Engine On-Demand Configuration (ODC)

23. 23 HA Core Group Message Links JVM N^2-N links N=2, 2 links N=3, 6 links N=4, 12 links N=5, 20 links N=6, 30 links With each additional JVM: Geometrically greater messages flowing More connections per JVM More memory per JVM to hold messages Optimizations to reduce message frequency and size are available in WAS 6.1

24. 24 DRS – Important Tuning DRS shares the HA Manager with other services Under load, you may see large numbers of any of the following Distribution and Consistency Services (DCS) congestion messages in your SystemOut.log file: DCSV1051W, a high severity congestion event for outgoing messages DCSV1052W, a medium severity congestion event for outgoing messages DCSV1054W, a medium severity congestion event for incoming messages Tuning directions are given in the InfoCenter: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/trun_ha_cfg_replication.html http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/com.ibm.websphere.nd.doc/info/ae/ae/trun_ha_cfg_replication.html

25. 25 DRS – Important Tuning Consider increasing the size of the default thread pool – In larger configurations, the Default thread pool size should be increased. – Doubling the thread pool size to 40 will likely be sufficient. – However, when the number of application servers in a replication domain is greater 10 and the number of replication domain consumers in each application server exceeds 2, it may need to be increased more. – This should keep the DCS traffic moving and avoid the timeouts. The transport buffer size out of the box value may also be insufficient – For all the appservers doing replication, – Click on Servers -> Application Servers -> -> Coregroup service – Change the transport buffer size to 100MB (or larger in more extreme loads). This changes the RMM buffer size. We also recommend changing the IBM_CS_DATASTACK_MEG memory config parameter – Servers --> Core Groups --> Core group settings -->then the settings for your core group (e.g. such as DefaultCoreGroup). Choose "Additional Properties" to specify a "Custom Properties". Key would be "IBM_CS_DATASTACK_MEG" with value in MB. Default value is 50.

26. Cache Replication

27. 27 Replicating data across servers in a cluster Currently only dealt with the scenario where data is cached per JVM The Data Replication Service replicates data throughout the cell for various functions Dynacache HTTP Session Stateful Session Beans The scope of replication, the “Replication Domain” allows multiple deployment patterns Client/Server Client-only/Server-only Creating replication domains: http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/co m.ibm.websphere.nd.doc/info/ae/ae/trun_drs_replication.html http://publib.boulder.ibm.com/infocenter/wasinfo/v6r1/index.jsp?topic=/co m.ibm.websphere.nd.doc/info/ae/ae/trun_drs_replication.html

28. 28 Replication Topologies – Peer to peer For dynacache and HTTP session replication

29. 29 Replication Topologies – Client/Server For HTTP session replication only

30. 30 Replication Domain configuration Single replica – Limits replicated data to a single backup copy within the Replication Domain – Reduces the total amount of redundant data held throughout the domain – E.g. HTTP Session replication Entire Domain – Stores a backup copy within each DRS instance in the Replication Domain – Maximises the redundancy of the data held throughout the domain – E.g. Dynamic cache needs to use this Specify – Stores a specified number of backup copies within the Replication Domain

31. 31 Configuring Replication for Default Cache Instance

32. 32 Replication Types  In all of the following modes, the invalidations will be sent across servers for data consistency.  PUSH mode - cacheId and Cache Data will be sent to all servers in cluster  PULL mode (Not recommended) - Server requests the data from other servers in cluster, when not present in its cache  PUSH/PULL mode - Only cacheId is sent to all servers in cluster. When a server needs an entry in the Push/Pull table, it requests it from the server that has the copy.  Not Shared - Cache data will not be shared across servers. But invalidation events are sent to all servers.  Recommendation  Try PUSH and move to PUSH/PULL or even Not Shared if struggling to replicate

33. Monitoring the Cache

34. 34 Monitoring the Cache – The CacheMonitor Application  Shipped with WebSphere application server install in the installableApps dir

35. 35 Monitoring the Cache  Cache contents can also be displayed.  In our example, the cacheID is /referenceWEB/Time.jsp:requestType=GET

36. 36 Disk Offload

37. 37 Extended Cache Monitor on developerWorks  http://www.ibm.com/developerworks/websphere/downloads/cache_monitor.html http://www.ibm.com/developerworks/websphere/downloads/cache_monitor.html  Provides two functions that are not available with the cache monitor Monitor and Manage contents of object cache Monitor cache statistics for cache instances across all members of the cluster.  Enhancements made for the cache monitor in WebSphere Application Server V7:  look at the push-pull table associated with a cache instance  search memory contents, disk contents, and push-pull table using a regular expression  compare cache instances

38. 38 Cache Statistics Collector and Visualizer Statistics from DynaCache JMX MBean provide insight into the state, health, performance, composition & efficiency of the cache. Collect cache statistics using DynaCacheStatisticsCSV.py wsadmin jython script. Outputs statistics in CSV file Statistics can be charted with Microsoft Excel or OpenOffice SpreadSheet. http://www.alphaworks.ibm.com/tech/cacheviz/

39. Performance & Troubleshooting

40. 40 Dynamic Cache Common Sense Tips  WebSphere Application Server uses JVM memory to cache objects. Therefore, know how much memory can be allocated for the cache and set the cache size to the proper value.  Increase the priority of cache entries that are expensive to regenerate.  Modify timeout of entries so that they stay in memory as long as they are valid.  If the estimated total size of all cached objects is bigger than the available memory, you can enable the disk offload option.  Increase the cache size if memory allows.

41. 41 Essential Dynacache Tuning  Read the following document for cache replication tuning http://www-01.ibm.com/support/docview.wss?uid=swg27006431  Set the following Dynacache JVM Custom Properties  Name: com.ibm.ws.cache.CacheConfig.ignoreValueInInvalidationEvent Value: true  Name: com.ibm.ws.cache.CacheConfig.filterTimeOutInvalidation Value: true  Name com.ibm.ws.cache.CacheConfig.filterLRUInvalidation Value: true  Name: com.ibm.ws.cache.CacheConfig.cacheEntryWindow Value: 50  Name: com.ibm.ws.cache.CacheConfig.cacheInvalidateEntryWindow Value: 50  Name: com.ibm.ws.cache.CacheConfig.useServerClassLoader Value: true  Assume to use replication type of NOT_SHARED  Read the following Technote for Portal Server Cache Replication issues http://www-01.ibm.com/support/docview.wss?uid=swg21322640  Read the following Technote for Commerce server replication issues http://www-01.ibm.com/support/docview.wss?uid=swg21358672

42. 42 Objects placed in cache are not replicated  Make sure the replication instance is launched (see SystemOut.log below)  Test with PUSH mode  If you are using PUSH_PULL mode, only CacheId will be pushed. Cache data will be pulled when needed  The following messages in SystemOut.log file tells Dynamic cache instance is initialized and replication service is launched. Notice a small delay, if application is looking for replicated data during the delay, it may not find it [2008-07-10 18:56:01:828 EST] 0000002c CacheServiceI I DYNA1001I: WebSphere Dynamic Cache instance named /cache/instance_one initialized successfully. [2008-07-10 18:56:02:828 EST] 00000042 DRSBootstrapM A CWWDR0001I: Replication instance launched : /cache/instance_one  See the following Technote for more information:  http://www-01.ibm.com/support/docview.wss?uid=swg21313480 http://www-01.ibm.com/support/docview.wss?uid=swg21313480

43. 43 Dynacache Congestion  Signs of Congestion:  (1) DCSV1051W/DCSV1052W: DCS Stack DefaultCoreGroup.MyCluster at Member CustomServer1: Raised a high severity congestion event for outgoing messages. Internal details are Total stored bytes: 67701476, Red mark is 41943040, Yellow mark is 37748736, Green mark is 8388608.  (2) HMGR0152W: CPU Starvation detected. Current thread scheduling delay is 109 seconds.  Congestion normally occurs when there is lot of data stored in cache and a member joins a HAManager view or if there are lot of invalidation events across servers.  Workarounds:  Use the NOT_SHARED or PUSH_PULL replication mode  Filter out of unnecessary cache invalidation events by configuring custom properties: Name: com.ibm.ws.cache.CacheConfig.filterTimeOutInvalidation Value: true Name: com.ibm.ws.cache.CacheConfig.filterLRUInvalidation Value: true

44. 44 Dynacache: Controlling DRS message size  When processing BatchUpates, the DRS message could be huge, which can lead to fragmentation issues and OutOfMemory errors.  The following custom properties introduced in PK32201 and PK35284 helps to control the size of DRS messages: com.ibm.ws.cache.CacheConfig.cachePercentageWindow – 2% com.ibm.ws.cache.CacheConfig.cacheEntryWindow – 50 com.ibm.ws.CacheConfig.batchUpdateMilliseconds - 1 sec com.ibm.ws.cache.CacheConfig.cacheInvalidateEntryWindow – 50 com.ibm.ws.cache.CacheConfig.cacheInvalidatePercentWindow -2%

45. Advanced topologies

46. 46 Dynamic Cache Next Generation

47. 47 WebSphere eXtreme Scale WebSphere eXtreme Scale provides distributed object caching essential for elastic scalability and next-generation cloud environments. It processes massive volumes of transactions with extreme efficiency and linear scalability. The WebSphere eXtreme Scale dynamic cache provider: It uses cheaper system memory instead of SAN or storage solutions Provides a scalable replicated cache with a configurable number of replicas. This eliminates the need to use broadcast data everywhere with DRS. Scales “elastically”. Additional WebSphere eXtreme Scale containers can be added at runtime. WebSphere eXtreme Scale automatically redistributes data partitions as new containers are added to the grid. Provides better caching qualities of service and control, than the default cache provider. Uses the same runtime monitoring and administration tools as the classic dynamic cache.

48. 48 WCS JVM 1.5Gb heap WCS App 90% - 1.35Gb WAS internals 10% - 0.15Gb Benefits With WXS, we offload the dynacache data store to WXS “grid” WCS estate needs 25% less memory for dynacache - potentially reduce WCS estate Performance improvement from not needing disk I/O Disk not needed – cost savings WCS throughput improvement through reduced “chatter” between JVMs and less GC overhead WXS can now provide much larger in-memory cache if desired by adding more JVMs (disk often constrained by size and contention limits Replica WXS JVM gives the cache resilience WCS JVM 1.5Gb heap WCS App 90% - 1.35Gb WAS internals 10% - 0.15Gb WCS JVM 1.5Gb heap WCS App 90% - 1.35Gb WAS internals 10% - 0.15Gb WCS JVM 1.5Gb heap WCS App 90% - 1.35Gb WAS internals 10% - 0.15Gb WCS JVM 1.5Gb heap WCS App 90% - 1.35Gb WAS internals 10% - 0.15Gb WCS JVM 1.5Gb heap WCS App 90% - 1.35Gb WAS internals 10% - 0.15Gb WebSphere eXtreme Scale Example with Commerce WXS Grid 18Gb

49. Summary

50. 50 Summary Dynamic Cache is a core service provided by WebSphere Application Server that can provide significant benefits to application developers for – Web content – Servlets/JSPs, AJAX requests, Web Services – DistributedMap API – Java command objects Dynamic Cache provides a comprehensive monitoring support infrastructure Higher performance environments will need to do some tuning to ensure Dynamic Cache performs at its best

51. 51 Reference  IBM Redbooks  WebSphere Application Server V6 Scalability and Performance Handbook http://www.redbooks.ibm.com/abstracts/SG246392.html  Mastering DynaCache in WebSphere Commerce http://www.redbooks.ibm.com/abstracts/sg247393.html  WebSphere Dynamic Cache: Improving J2EE application performance http://www.research.ibm.com/journal/sj/432/bakalova.pdf

52. Backup

53. 53 Cache Invalidation  It is essential that timely invalidations of cached content take place for the integrity of the website.  Mechanisms for invalidation are:  timeout or inactivity directives within cachespec.xml  group-based invalidation mechanism through dependency IDs.  Programmatic invalidation via the cache API 'com.ibm.websphere.cache.*’  The CacheMonitor

54. 54 “Data Oriented” “Application Oriented” Session management Elastic DynaCache Web side cache Event Processing In-memory OLTP Worldwide cache Petabyte analytics Data buffer In-memory SOA eXtreme Scale DataPower XC10 Appliance Ultimate flexibility across a broad range of caching scenarios In-memory capabilities for application oriented scenarios Drop-in cache solution optimized and hardened for data oriented scenarios High density, low footprint improves datacenter efficiency Elastic caching for linear scalability High availability data replication Simplified management, monitoring and administration

55. 55 Caching Rules in cachespec.xml  Consider the simple JSP, Display.jsp.  The request to Display.jsp returns itself, as the parent, plus the included fragment Time.jsp.

56. 56 Caching Rules in cachespec.xml consume-subfragments The consume-subfragments property tells the cache not to stop saving content when it includes a child servlet. The parent entry will include all the content from all fragments in its cache entry, resulting in one big cache entry. Use the element to tell the cache to stop consuming for the excluded fragment and instead, create a placeholder for the include or forward. For example, exclude Time.jsp from the consume-subfragment, as follows…

57. 57 Caching Rules in cachespec.xml servlet /Display.jsp true /Time.jsp h 30

58. 58 Caching Rules in cachespec.xml do-not-consume  As discussed, the consume-subfragments property tells the cache to save all content including child fragments.  A fragment can be excluded from the rules of the parent by using the do-not-consume property.  In this cachespec, Time.jsp is labeled do- not-consume and can specify its own timeout or other rules. servlet /Display.jsp true 30 servlet /Time.jsp true 10

59. 59 Caching Rules in cachespec.xml Inactivity.  While the timeout directive dictates how long content can remain in cache before being refreshed, the Inactivity directive can cause a refresh prior to the timeout when a page is not used frequently. 600 30

60. 60 Dependency IDs  Dynamic Cache provides a group-based invalidation mechanism through dependency IDs.  A dependencyID identifies a cache entries dependency based on certain factors, such that when those factors occur they trigger an invalidation of all the cache entries that share this dependency.  An example of such a dependency could be the invalidation of a page which lists customer names. (Customer.jsp) Cached entries for this list should be invalidated when a customer is added to or removed from the list.

61. 61 Troubleshooting Dynamic Cache  Dynamic Cache Trace  Use the WebSphere trace facility to review key trace points and verify expected caching behavior.  Enabling Trace  Dynamic Cache issues can be traced using the following trace specification: Dynacache replication is disabled −*=info:com.ibm.ws.cache.*=all Dynacache replication is enabled −*=info:com.ibm.ws.cache.*=all:com.ibm.ws.drs.*=all  For information regarding trace setting please refer to the WebSphere information center or see this this link: http://www-1.ibm.com/support/docview.wss?rs=180&uid=swg21254706

62. 62 Troubleshooting Dynamic Cache Binding to cache instance ResourceMgrIm I WSVR0049I: Binding services/cache/diskoffload as services/cache/diskoffload First request CacheHook 3 handleServlet: absoluteUri = /referenceWEB/Display.jsp EntryInfo 3 set id=/referenceWEB/Display.jsp:name=Fred:requestType=GET FragmentCompo 3 setConsumeSubfragments: /Display.jsp consumeSubfragments=true FragmentCompo 3 setDoNotConsume: /Display.jsp doNotConsume=false CacheStatisti 3 CACHE: Cache Miss: /referenceWEB/Display.jsp:name=Fred:requestType=GET CacheHook 3 CACHE MISS id: /referenceWEB/Display.jsp:name=Fred:requestType=GET CacheHook 3 servicing /referenceWEB/Display.jsp:name=Fred:requestType=GET FragmentCompo 3 saveCachedData uri=/Display.jsp Next request: CacheStatisti 3 CACHE: Local Cache Hit: /referenceWEB/Display.jsp:name=Fred:requestType=GET CacheHook 3 CACHE HIT id:/referenceWEB/Display.jsp:name=Fred:requestType=GET Invalidation due to cache timeout Cache > internalInvalidateById() cacheName=baseCache id=/referenceWEB/Display.jsp:name=Fred:requestType=GET Entry Cache < internalInvalidateById() id=/referenceWEB/Display.jsp:name=Fred:requestType=GET Exit  If a problem is to be submitted to IBM Support, please collect trace and configuration by following this Mustgather document  http://www-1.ibm.com/support/docview.wss?uid=swg21193837