1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 1

2. DTrace for Oracle Linux Avi Miller Principal Program Manager, Oracle Linux

3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 3 Overview  Setting the stage…  State of the work.  What can you do with it… (and how)?  A look ahead…

4. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 4 Setting the stage…  DTrace is a runtime observer across the software stack.  Static tracing using instrumentation compiled into kernel and applications.  Dynamic tracing using probes inserted at runtime.  ‘D’ scripts execute when probes fire (under predicate control).  Decision to record events can be deferred using speculative tracing.  DTrace detects illegal memory accesses and reports an error rather than crashing the system! What is DTrace?

5. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 5 Setting the stage…  A plethora of tracing tools exists for Linux, for varying use cases: – Inconsistent syntax, scripting languages, and output formats. – Lack of integration of both kernel and application tracing in a single tool.  DTrace provides an integrated solution.  Administrators and developers “know” DTrace from Oracle Solaris.  Existing ‘D’ scripts can be used.  Customers ask for it! Why do we want DTrace for Oracle Linux?

6. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 6 Setting the stage…  It is… DTrace!  Full (1) port of the Oracle Solaris implementation.  Tailored to the Linux kernel.  Brings new features to Linux (page fault notifiers, CTF, …).  Source code for the DTrace core and providers is available at oss.oracle.com. (1) Standard disclaimers apply. What is DTrace for Oracle Linux?

7. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 7 State of the work

8. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 8 State of the work  Project start: summer/fall 2010  First public release: 0.1.0 (October 2011)  Current release: 0.3.1 (October 2012)  Based on Oracle Linux Unbreakable Enterprise Kernel 2.6.39 (UEK2)  Available as technology preview kernel  64-bit only (x86_64) Work in progress; not feature complete quite yet.

9. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 9 High level overview  Comprises several packages: – Kernel RPM: UEK2 with patches that augment the tracing infrastructure – Kernel modules: DTrace core and standard providers – Compact Type Format (CTF) library for DTrace – DTrace user space utility, incl. translators  Supports standard extensions: – Custom providers (provider-to-framework API) – Custom consumers (ioctl()-based API) Modular framework

10. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 10 Kernel  Writable system call table.  Tracing specific fields in the task_struct structure.  Notifier support for page faults.  Generic iterator over kallsyms data, and handle compiled-in modules.  Kernel data type and external variable data stored as CTF data in a separate kernel module.  SDT probe locations linked into the kernel.  Very minimal implementation of cyclics. Oracle Linux Unbreakable Enterprise Kernel modifications

11. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 11 Probes  Direct call probes – Direct calls to the dtrace_probe() function. – Either from regular execution flow or timer-based.  Trap-based probes – Single byte instruction inserted to trigger a trap. – Statically Defined Tracing (SDT) – Function Boundary Tracing (FBT) – … Two flavours

12. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 12 Probes  dtrace:::BEGIN, dtrace:::END, dtrace:::ERROR – Triggered by the DTrace core.  profile:::tick- where is an integer followed by a length of time specifier (ns, ms, s, …) or “hz” – Triggered by a high resolution timer.  syscall:: :entry, syscall:: :return for each system call – Triggered upon entering the kernel for system call, and right before returning from the system call. Direct call probes (provider:module:function:name)

13. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 13 Probes  proc::: where is one of: – create, lwp-create, start, lwp-start, exec, exec-failure, exec-success, exit, lwp-exit – signal-send, signal-discard, signal-handle, signal-clear  sched::: where is one of: – change-pri, enqueue, dequeue, tick – on-cpu, off-cpu, remain-cpu, preempt, surrender, sleep, wakeup  io::: where is one of: – start, wait-start, wait-done, done Statically Defined Tracing (SDT) probes (provider:module:function:name)

14. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 14 Dare to compare? Utterly useless comparison Operating system (version)Number of probes MacOS X154918 (incl. pre-created per-process probes) Delphix OS77320 Oracle Linux (DTrace 0.1)574 Oracle Linux (DTrace 0.2)617 Oracle Linux (DTrace 0.3)636 Oracle Linux (DTrace Dev)41935

15. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 15 What can you do with it… (and how)?

16. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 16 High level workflow a.k.a. What we would like to do 1234 Select probes that we are interested in Process events and associated actions Listen for probe fire events on selected probes Generate output

17. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 17 What happens when a probe fires? Linear tracing Loop: Retrieve buffer Process events Show output DTrace coreKernelConsumer Process event: Predicate eval Record event Execute script Probe reached: Direct call Trap Timer-based: Direct call

18. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 18 What happens when a probe fires? Aggregation: f(f(x 1 ), f(x 2 )) ≡ f(x 1, x 2 ) Loop: Retrieve buffer Aggregate data Show output DTrace coreKernelConsumer Process event: Predicate eval Aggregate data Probe reached: Direct call Trap Timer-based: Direct call

19. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 19 What happens when a probe fires? Speculative tracing DTrace core (t 1 )Kernel (t 1 )Point Predicate eval Store event Probe fires Retrieve buffer Process events DTrace core (t 2 )Kernel (t 2 )Consumer (t 2 ) Commit/DiscardProbe fires

20. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 20 syscall::: { @[probefunc] = count(); } tick-1s /i++ >= 10/ { exit(0); } syscalls.d Basic example (Aggregation, global variable, predicate)

21. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 21 syscall::read:entry { self->t = timestamp; } syscall::read:return /self->t != 0/ { printf(“%s spent %d nsecs in read()\n”); self->t = 0; } rtime.d Cross-probe example (Thread variable, formatted output)

22. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 22 io:::done, io:::start, io:::wait-done, io:::wait-start { printf(“%8s %10s: %d %16s (%s size %d @ sect %d)\n”, args[1]->dev_statname, probename, timestamp & 1000000000, execname, args[0]->b_flags & B_READ ? “R” : args[0]->b_flags & B_WRITE ? “W” : “?”, args[0]->b_bcount, args[0]->b_blkno); } io1.d Fancy example (Multi-probe action, xlators, formatted output)

23. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 23 Workflow matters Little details make all the difference 1234 Enable probes that we are interested in Process events until done: retrieve, process, output Commence recording of probe events Stop recording events and disable probes CPU 1 CPU 2 CPU n Sleep

24. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 24 Output is not time-sequential  DTrace consumer retrieves CPU buffers every N ns.  CPU buffers are retrieved sequentially.  Not necessarily by CPU ID!  Within a CPU buffer, events are time-sequential.  Across multiple CPUs, there are no guarantees.  Solution: use timestamps! Why?

25. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 25 Details that matter  When a probe fires, all actions associated with that probe will: – Execute in the order they appear in the script. – Execute on the CPU that triggered the probe.  Kernel implementation details matter (e.g. IO completion is typically not associated with the task that started it.)  If you do not print aggregates, they will be printed automatically after the END probe fires.  If you want fancy output, consider post-processing. a.k.a. Good things to keep in mind

26. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 26 Tracing incurs overhead  Cost of calling an probe that has not been enabled: ~75 ns  Cost of calling a probe with empty action: ~95 ns  Cost of triggering an SDT probe: ~3600 ns (38x direct probe) – SDT probe is not called unless enabled – SDT probe uses a trap, which is relatively expensive  Tracing = data collection! There is no such thing as free tracing…

27. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 27 A look ahead…

28. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 28 A look ahead…  Function boundary tracing – Probe at entry and exit of any function  More SDT probes – Cover more Oracle Linux subsystems  Userspace probing – Statically Defined Tracing of applications – Function Boundary Tracing of applications – Arbitrary address probes Features for future releases

29. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 29 A look ahead…  Release as a supported feature.  Integration into the Oracle Linux Unbreakable Enterprise Kernel.  New DTrace features.  Stability and correctness remains the top priority! Coming of age…

30. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 30 Graphic Section Divider

31. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 31 Visit our partners and don’t miss these events sponsored by QLogic  Smoothie Bar on Monday, Oct 1 st, 2:30- 5:30pm  Ice Cream Social on Wednesday, Oct 3 rd, 1- 2pm ORACLE LINUX PAVILION

32. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 32 NEW: Oracle Linux Curriculum Footprint Oracle Linux System Administration Instructor-led and Live virtual Unix/Linux Essentials Instructor-led and Live virtual This Oracle Linux System Administration course teachesOracle Linux System Administration you all the essential system administration skills and includes key information specific to Oracle Linux: Unbreakable Enterprise Kernel, Ksplice, ULN, and other key features Visit: oracle.com/education/linux Oracle Linux Training from Oracle University

33. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 33 @ORCL_Linux Facebook.com/ OracleLinux Blogs.oracle.com /linux Oracle Linux Experts Group YouTube.com/oraclelinuxchannel Join our communities Resources Visit Oracle.com/linux Download for FREE edelivery.oracle.com/linux

34. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 34 The preceding is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

35. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 35

36. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 36