1. Java on z/OS: A fresh look
Scott Chapman
American Electric Power

2. Important notes I don’t really like Java as a language
I’m not a Java expert
Results presented herein may be installation-dependent
There’s a lot of moving parts here
I understand there’s zAAP on zIIP
“zAAP” used generically here
All trademarks of IBM, Oracle, and everybody else hereby recognized

3. Why Java on z/OS? Because programmers want to use it

4. Why Java on z/OS
Because it enables open source projects that are cool/useful/interesting
Key trick: run the JVM in ASCII
-Dfile.encoding=ISO8859-1
Many things will just run with that run-time option!

7. What about a GUI? Turns out that that just works too!
Start Xming X server on your PC
Check the “No Access Control” option
Set the DISPLAY environment variable
Run the code
S147774:/u/s147774: >export DISPLAY= :0
S147774:/u/s147774: >java -Xmx320m -jar ga33.jar

8. Debugging Javascript code
running in Helma on the
mainframe with the GUI
connected to Xming on
my laptop
Works better than
I expected

9. Why Java on z/OS Because it enables more programming language choices
Javascript built in to Java 6
Rhino interpreter from Mozilla
In theory, should be able to run any JVM- based language (I haven’t tested these)
Jython
Groovy
Clojure
Scala
Ruby (via JRuby)

10. Why Java on z/OS It may perform better It may save you money
If you are on a sub-capacity machine
It may save you money
Pretty unlikely
Only if you can take some work away from your peaks

11. Which job is better?

12. How cheap are zAAP/zIIPs?
$100K/SE (z196, zEC12)
How much is $100K?
Consider adding 1 engine to z :
710 = 10,250 MIPS, 1191 MSUs
711 = 11,073 MIPS, 1286 MSUs
710+1 zIIP = 10,302+1,000 MIPS
z/OS (base) at this level costs $62/MSU
Scenario B, z/OS base goes up almost $6K/month
zIIP costs < 17 months of z/OS Base
Not to mention features, DB2, CICS, etc.

13. What about accessing z/OS services?
JZOS Classes to easily access z/OS specific constructs
z/OS datasets
RACF
Respond to operator commands
Access JES Spool

14. Ways to Run Java on z/OS WebSphere CICS DB2 Stored Procedures Batch
Started Tasks
Unix shell

15. Batch / Started Task options
BPXBATC
BPXBATCH (traditional alias)
BPXBATSL (local spawn alias)
Traditional approach
Difficulty with 100-byte JCL Parm
JZOS
Ships with z/OS
Avoids 100-byte parm limit
Adds a lot of flexibility

16. Measuring Java

17. zAAP vs. GCP time Watch the normalization factor!
Most SMF values not normalized
Tools/reports may normalize for you
Consider IFAHONORPRIORITY=NO
Avoid using GCPs to help zAAPs
Can result in >99% of Java CPU time executed on zAAP

18. SDSF zAAP vs. GCP columns
This data comes from RMF
JOBNAME CPU-Time GCP-Time zAAP-Time zACP-Time zAAP-NTime
P3SR01BS
P3SR01AS
P3SR01B
P3SR01A
P3SR02A
P3SR02B
P3SR01AS
P3SR02BS
P3SR01BS
P3SR02AS
RTMSERVE
TCB + SRB
real
zAAP on GCP
normalized

19. SMF 30 Accounting BPXBATCH vs. BPXBATSL vs. JZOS
Important due to spawned OMVS tasks
Single step job results:
BPXBATSL: 1 step, 1 job record
BPXBATCH: 6 step, 4 job records
CPU time collected on type OMVS records
JZOS: 2 step, 2 job records
CPU time almost completely on JOB types

20. Some interesting calculations
zAAPn = SMF30_TIME_ON_IFA * SMF30ZNF / 256
percent work done on zAAP =
zAAPn / (zAAPn + SMF30CPT + SMF30CPU)
(“Generosity” or “offload” factor)
percent zAAP sent to GCP =
SMF30_TIME_IFA_ON_CP / (SMF30_TIME_ON_IFA+SMF30_TIME_IFA_ON_CP)
(“Fallback” percentage—can be <1%, although some fallback is normal and expected)

21. Other SMF records RMF records WAS 120 records HIS type 113 records
Look for breakdown of processor types for both hardware and report / service classes
WAS 120 records
New subtype 9s for WAS 7+ much better!
HIS type 113 records
GCP vs. zAAP vs. zIIP

22. Java Performance

23. What about performance?
Java on the mainframe has a history of performance problems
Java is inherently “heavy” due to the JVM
Scott’s Law: “The easier you make it on the programmer, the harder it is on the system”
Today’s z hardware and software are up to the task!
(But you probably want zAAPs!)

24. Heard at WAS Week 200x…
“Our goal is to get JVM startup time down to about 1 second.”
Seemed like a stretch at the time!
WAS startup took several minutes

25. Today: WAS Servant Startup <1 min
STC MONDAY, APR
STC $HASP373 P3SR02AS STARTED
STC IEFUSI BPXBATSL-P3ASRU ABOVE REGION SET TO 1536MB
STC IEF403I P3SR02AS - STARTED - TIME=
STC BBOO0004I WEBSPHERE FOR Z/OS SERVANT PROCESS
P3CELL/P3NODEA/P3SR02/P3SR02A IS STARTING.
STC BBOO0239I WEBSPHERE FOR Z/OS SERVANT PROCESS p3cell/p3nodea/p3sr02a IS
STARTING.
STC BBOO0308I SERVANT PROCESS P3CELL/P3NODEA/P3SR02/P3SR02A IS EXECUTING
IN 64-BIT ADDRESSING MODE.
STC BBOM0007I CURRENT CB SERVICE LEVEL IS build level
(cf ) release WAS70.ZNATV date 07/09/10 11:02:02.
...
STC BBOO0222I: WSVR0001I: Server SERVANT PROCESS p3sr02a open for
e-business
STC BBOO0020I INITIALIZATION COMPLETE FOR WEBSPHERE FOR Z/OS SERVANT
PROCESS P3SR02A.
STC BBOO0248I INITIALIZATION COMPLETE FOR WEBSPHERE FOR Z/OS SERVANT
PROCESS P3CELL/P3NODEA/P3SR02/P3SR02A.
Not much in that
particular servant

26. Today: HelloWorld in <2 seconds
JOB IEF403I S147774B - STARTED - TIME=
JOB TIMINGS (MINS.) PAGING COUNTS---
JOB JOBNAME STEPNAME PROCSTEP RC EXCP CPU SRB CLOCK SERV PG PAGE SWAP VIO
JOB S147774B RUNOMVS
JOB IEF404I S147774B - ENDED - TIME=
JOB S147774B ENDED. NAME-BPXBATCH TEST TOTAL CPU TIME= TOTAL ELAPSED TIME= .02
JOB $HASP395 S147774B ENDED
z10 EC 504
with zAAP
Output
Hello Scott
Java runtime: IBM Corporation 1.6.0, vm version 2.4
Running on: s390 z/OS
Running for: S147774
Classpath: /usr/lpp/java/J6.0/lib:/usr/lpp/java/IBM/J1.3/l
JCL
//RUNOMVS EXEC PGM=BPXBATCH,
// PARM='SH java -Xms32M -Xmx32M HelloWorldApp Scott'
//SYSOUT DD SYSOUT=*
//SYSPRINT DD SYSOUT=*
//SYSUDUMP DD SYSOUT=*
//STDENV DD *
//STDOUT DD SYSOUT=*
//STDERR DD SYSOUT=*

27. Small machine Not surprising that ~50 MIPS engines can’t keep up with
JOB IEF403I S147774B - STARTED - TIME=
JOB TIMINGS (MINS.) PAGING COUNTS---
JOB JOBNAME STEPNAME PROCSTEP RC EXCP CPU SRB CLOCK SERV PG PAGE SWAP VIO
JOB S147774B RUNOMVS
JOB IEF404I S147774B - ENDED - TIME=
JOB S147774B ENDED. NAME-BPXBATCH TEST TOTAL CPU TIME= TOTAL ELAPSED TIME= .18
JOB $HASP395 S147774B ENDED
z10 BC E02
without zAAPs
Not surprising that ~50 MIPS engines can’t keep up with
450 / 900 MIPS engines

28. What about doing real work?
Days of assuming it will run faster on your PC are over
Have seen H2 perform better on z/OS
Still, it is Java, it’s not CPU-free
Performance may depend on:
zAAP and GCP capacity
System settings (USS, zFS, WLM)
Application code
Java Settings (heap size, GC policy)
Random luck

29. Application code Application code is always important
Regardless of the language!
BufferedReader or ZFile?
Classic “it depends”
BufferedReader seems like it should be faster
But they provide different results: byte array vs. string
What you want to do with the result may impact which is best for any given situation
Java has lots of similar but slightly different ways of doing things

30. Heap settings Heap settings always seen as an issue
Size is the usual suggestion
Is bigger always better?
Does anybody know how much heap they really need? (no)
Min / Max sizes same or different?
Garbage collection policy options

31. Memory is an issue Java’s memory usage can be an issue
“Requirements” for 100s of MBs are not unusual
Often “requirements” seem to be a SWAG
Java heap size can’t be reliably predicted from the code & expected volumetrics
Test with reasonable numbers before assuming the requirements are real
Be sure to get all processing scenarios!

32. Garbage Collection Options (IBM Java 6)
optthruput – default
Probably best for batch
gencon – generational / concurrent
maybe good for large heap, transactional workloads (WAS)
optavgpause – reduces long pauses
subpool – “improved” object allocation
For important workloads, may want to test all of them at various size
Lots of other heap/gc options too
See IBM JDK Diagnostics Guide!

33. heap size may not matter
For some workloads,
heap size may not matter

34. Too small of a heap can
cause CPU increase

35. There might be a slight
benefit to a fixed
heap size

36. Heap size most important,
but GC Policy also
can be significant

37. Don’t mess
with the JIT!

38. Could be good
for certain
workloads

39. So what’s the random thing?
Much more variation in CPU time measurements with today’s CPUs
Superscalar pipeline and cache issues
Seems to impact my Java work more than I expected
Consistently ran same workload
Extremely lightly utilized LPAR
Lightly utilized zAAPs
Same variability over time
So I tried some more tests…

40. One zAAP
Two zAAPs
Zero zAAPs

41. Why is this?
I don’t know, but best guess is CPU cache and memory access effects
But I thought I’d look at the 113 records to see if I could find anything interesting….

42. Data from
Test period 1
(One zAAP)
Proc 0 = GCP
Proc 2 = zAAP

43. Proc 0 = GCP
Proc 2 = zAAP
Seems to confirm
our SMF30 data

44. Proc 0 = GCP
Proc 2 = zAAP

45. L1.5 Improvement
corresponds to dip
in machine usage
Proc 0 = GCP
Proc 2 = zAAP

46. Dip in GCP TLB Miss
overhead due to
machine less busy
Proc 0 = GCP
Proc 2 = zAAP

47. Proc 0 = GCP
Proc 2 = zAAP

48. My Guesses…
My test Java workloads were too cache and superscalar friendly
Perhaps makes it more susceptible to pipeline hazards
But:
Wouldn’t the REXX workload be even more superscalar and cache friendly?
Why were the 113 measurements so consistent?
Or Java is really doing variable amounts of work?
Or… something isn’t right someplace?
Take away: Java CPU measurements might be more variable than you expect

49. Most recent testing Repeated testing later in the year
z/OS 1.12 vs. 1.10
1 Year more recent Java 6 (Fall 2010 vs. Fall 2009)
Still saw variability, but worst of it was closer to % instead of upwards of 75%
Saw similar variability when testing on a z9 with zAAPs
Saw at least one instance in a production LPAR with similar variability: (in 3 executions of the same job, 1st consumed just over half as much CPU of the later runs)
Could not readily replicate on a WSC system running under z/VM

50. Summary
Java enables all sorts of cool things you might not have thought could run on the mainframe
Mainframe’s Java performance not significantly worse than any other platform
(Assuming adequate zAAP capacity)
Lots of tuning knobs for Java
Java CPU time measurements might be more variable