1. Bringing Apache Cassandra® to JDK 9
Yingqi (Lucy) Lu Software Performance Engineer Intel Corporation
Robert Stupp Sr. Software Engineer – Escalations DataStax Inc. @snazy

2. Legal Disclaimer & Optimization Notice
INFORMATION IN THIS DOCUMENT IS PROVIDED “AS IS”. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO THIS INFORMATION INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. Software and workloads used in performance tests may have been optimized for performance only on Intel microprocessors. Performance tests, such as SYSmark and MobileMark, are measured using specific computer systems, components, software, operations and functions. Any change to any of those factors may cause the results to vary. You should consult other information and performance tests to assist you in fully evaluating your contemplated purchases, including the performance of that product when combined with other products. Copyright © 2016, Intel Corporation. All rights reserved. Intel, Pentium, Xeon, Xeon Phi, Core, VTune, Cilk, and the Intel logo are trademarks of Intel Corporation in the U.S. and other countries.
Optimization Notice
Intel’s compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice.
Notice revision # 2

3. Disclaimer Talking about OSS Nothing promised
Things may change without prior notice
Apache Cassandra, Apache and Cassandra are trademarks of the Apache Software Foundation or its subsidiaries in Canada, the United States and/or other countries.

4. What is Apache Cassandra?
A distributed NoSQL database
Google BigTable ✖ Amazon Dynamo
Scale out with linear performance
Just add nodes
Continuously available
Disaster avoidance, not disaster recovery
Run on commodity hardware
In the cloud, on premise, or hybrid
No time to explain the details
Trust me

5. Distributed Architecture
Node 1 1st copy
Node 4
Node 5
Node 2 2nd copy
Node 3 3rd copy
Fully distributed
Data spread over multiple nodes
All nodes participate in a cluster
Configurable data replication
Masterless
All nodes are equal
Read from or write to any node
No Single Point of Failure
No transaction manager
Single transaction manager for the whole world???

6. Scale Out Linearly Need more storage? Need greater throughput?
Add more nodes.
Need greater throughput?
Predictable, linear performance gains

7. Changes in C* codebase for Java 9 (not perfect, but good enough)

8. jamm + OHC
1.8.0_144-b01
vs.
9+181
Library to measure actual object memory use including JVM overhead - v0.3.2 got change to recognize Java 9 version number
Off heap cache (C* row cache backend) - v0.4.5 got change to recognize Java 9 version number

9. Java options Different JVM options for 8 + 9
conf/cassandra-env.sh
Java options
Different JVM options for 8 + 9
--add-exports
Generalized logging options -Xlog:gc vs --Xloggc
-XX:ThreadPriorityPolicy=42 no longer works
Needs to “magically” work everywhere
jvm.options + jvm8.options + jvm9.options (chosen in conf/cassandra-env.sh)
CI environments running on 8 + 9

10. Java UDFs Classloader issues Library ecj (Java Compiler) updated
UDFs use a wrapping class loader in the security sandbox
Delegating ClassLoader.getResourceAsStream() changed be Class.getModule().getResourceAsStream() for Java 9 (for ecj)
Library ecj (Java Compiler) updated
--> USER DEFINED FUNCTIONS
Compilation
Sandbox
- Threads
- I/O
- Locks

11. Even if you want to use JavaScript UDFs, use Java UDFs, seriously.
Nashorn security issues
Requires even more packages (java.lang.ref java.io java.util.function jdk.dynalink.linker jdk.internal.org.objectweb.asm jdk.internal.reflect jdk.nashorn.internal.scripts)
Requires even more privileges (nashorn.createGlobal suppressAccessChecks dynalink.getLookup getClassLoader)
Nashorn scripts are difficult to secure (CPU time limit, heap usage)
Even if you want to use JavaScript UDFs, use Java UDFs, seriously.
Sandbox

12. AtomicBTreePartition
Concurrent memtable partition updates
Optimistic & fast vs. pessimistic & slow
Usually short time interval required to hold the lock
Up to Java 8 (Unsafe.monitorEnter/monitorExit)
monitorEnter called when heap usage limit exceeded
Spin-lock using AtomicLongFieldUpdater on a long containing the “owning” thread
Locked when heap usage limit exceeded
Elaborate about memtable
partition key
partitions
updates to a partition are atomic

13. ((Cleaner)DirectBuffer. cleaner()).clean()
Why that?
Off-heap memory region is a resource, guys
A resource needs to be closed (deallocated)
We do not want to rely on a GC to happen to free off-heap memory.
File and network I/O (NETTY!)

14. Bits.reserveMemory() Followed by a while (true) … Thread.sleep()
Can block for 0.5s !
Awesome…

15. ((Cleaner)DirectBuffer. cleaner()).clean()
Abstracted in FileUtils
java.lang.reflect.Method + java.lang.invoke.MethodHandle
Object cleaner = buf instanceof DirectBuffer ? mhDirectBufferCleaner.bindTo(buf).invoke() : null; if (cleaner != null) mhCleanerClean.bindTo(cleaner).invoke();

16. JMXServerUtils
com.sun.jmx.remote.internal.RMIExporter (Java 8) vs. com.sun.jmx.remote.internal.rmi.RMIExporter (Java 9)
Class.forName() test using the class names, use good, old java.lang.reflect.Proxy#newProxyInstance
Why? The primary benefit we gain is that it doesn't trigger the scheduled full GC that is otherwise incurred by programatically configuring the management server. CASSANDRA-2967, CASSANDRA-10091

17. Java 8 + 9 Interoperability
Not necessarily the class file format
Requirement to build on Java 8 (java.nio) Exception encountered during startup java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer; YAY :):
javac --release didn’t work for some reason
Need to build on Java 8 to support both Java 8 and 9 runtimes

18. It Is Time to Upgrade! References:
No public updates Oracle JDK 8 after September
JDK9 offers 2
New functionalities and new tools for development/debugging
Security enhancements
Lots of performance features
Checksum acceleration with new CRC32C
Lexicographic Array Compare
Many more
References: 1. 2.

19. Checksum Background
A checksum is a small-sized datum derived from a block of digital data for the purpose of detecting errors which may have been introduced during its transmission or storage (from Wikipedia1)
Performed on I/O blocks.
Frequently invoked method in Cassandra
Every 8KB for uncompressed dataset
Every 64KB (default) for compressed dataset
Cassandra uses CRC32 as default checksum
Reference: 1.

20. Java Built-In Checksum
Prior to JDK8 java.util.zip.CRC32 implemented as a JNI call
Hadoop community implemented a pure-Java version to eliminate JNI overhead (org.apache.hadoop.util.PureJavaCRC32) 10x faster that default
In JDK8 Intel introduced an underlying java.util.zip.CRC32 intrinsic 10x faster than the PureJavaCRC32
In JDK9 Intel contributes to the java.util.zip.CRC32c API 2x faster than the previous CRC32
Proposed adding CRC32C into Cassandra (CASSANDRA-13908)
References:

21. Lexicographic Array Compare Background
Compares two arrays letter by letter
Commonly used across applications comparing keys for lookups or search. Ex: Cassandra, HBase, Spark, etc.
Measured that up to 20% of runtime spent on lexicographic compare
Open source community implemented an unsafe version comparing 8 bytes in a single operation with an 8x speedup
Unsafe code shared across many Java middleware

22. How Casandra Uses Lexicographic Array Compare
Key comparison happens in Read, Write and Compaction
Keys are implemented as ByteBuffers
Partition Key
Transfer into byte array and compare
Clustering Key
Comparator implementation varies with data type: NOT_COMPARABLE, BYTE_ORDER and CUSTOM
Majority uses BYTE_ORDER, internally transfer ByteBuffer into byte array and compare

23. Java’s Lexicographic Array Compare
JDK9 introduces a standard Lexicographic array comparator based on Apache community feedback (VectorizedMismatch)
Implements underlying intrinsic comparing up to 64bytes as a single operation (on supported hardware)
12x speedup over standard Java 8 and older implementation
2x over unsafe implementation
Future hardware proof
References:

24. Proposed Changes to Cassandra
FastByteOperations
UnsafeOperation
PureJavaOperation
8 bytes as a compare unit
Compare byte by byte
Is unsafe available?
Yes No
Is JDK9?
Java.util.Array.compare
Up to 64 bytes as a compare unit (use UnsafeOperation on unsupported hardware)

25. Key Takeaways Apache Cassandra is JDK9 ready now
Not reviewed yet – not before C* 4.0
We are in progress of enabling new JDK9 features/APIs into Cassandra for performance improvement
Checksum acceleration with new CRC32C API
Lexicographic Array Compare
It is time to upgrade your JVM 

26. Acknowledgment
Thanks to everyone who has contributed to CASSANDRA-9608
Robert Stupp
Yingqi (Lucy) Lu
Paul Sandoz
Carlos Abad
Dave Brosius
Alan Bateman
Mandy Chung
Mark Paluch
Branch on github: goo.gl/SGGb21

27. Thank You!