1. 1 Generic logging layer for the distributed computing by Gene Van Buren Valeri Fine Jerome Lauret

2. 2 Logging API for HENP application Most HENP experiment software includes a logging or tracing API allowing for displaying in a particular format important feedback coming from the core application. However, inserting log statements into the code is a low-tech method for tracing the program execution flow and often leads to a flood of messages in which the relevant ones are occluded. In a distributed computing environment, accessing the information via a log-file is no longer applicable and the approach fails to provide runtime tracing

3. 3 Practical Grid case example A “job” is often a complex workflow or Meta-job –A Meta-Job can be composed of step1 = several sub-jobs running in parallel, the result collected and N jobs in step2 process the result etc …. –We want to have a consistent picture of the Meta-Job (not only individual ones) A given job may be re-directed elsewhere (pre-emption, re-submit) An individual job log may warrant the triggering of some action (abort the entire task or Meta-job, recovery service take action, workflow is altered etc …) To accommodate for the needed components, we first need to provide the necessary tools to collect the information => logger

4. 4 Consistency Running a job involves a chain of events where many components are involved often written in diverse languages and not offering a consistent and easily adaptable interface for logging important events. Meta-Job requires a Meta-log –Workflow should be represented –All steps may be inspected in details (individual steps) –An overview of the a global log (Meta-log) MUST be possible

5. 5 STAR meta-job STAR GRID job scheduler GRID realm GRID jobsJob log file Meta-log of meta-job Job log fileGRID jobs workflow

6. 6 The job steps … Job initiation –– local OS shell script –Job generation/ submission – Java code (for example, STAR Unified Meta-Scheduler, SUMS) Job initialization at the remote node – Shell script Application (user) job –Starting, application preparation – C++, Fortran, … –Running the application itself – C++, Fortran, … –Job finishing (from the application user point of view) – C++/Fortran/etc Job normal completion –transferring the log files and clean up – Shell script, Services (SRM) –file registration, … job abnormal completion at remote site – Perl script, … job abnormal completion at submitting side – Java code,… … etc …

7. 7 Jakarta log4j project An approach based on a new generic layer built on top of a logger family derived from the Jakarta log4j project that includes log4cxx, log4c, log4perl packages may provide a consistency across packages and framework A community supported project

8. 8 Main components Log4cxx has 3 main components: –loggers, –appenders and –layouts. These three types of components work together to enable developers to log messages according to message type and level, and to control at runtime how these messages are formatted and where they are reported.

9. 9 STAR Scheduler Log4cxx.xml Current job logging configuration Production Manager Job Abstract STAR logger List of appenders List of layouts Hierarchy of the loggers “ Console” Appender “HTML” Appender “MySQL Appender “ File” Appender “ Custom” Appender

10. 10 Event severity & verbosity The first and foremost advantage of any logging API over plain stdout resides –in its ability to disable certain log statements while allowing others to print unhindered. –This capability assumes that the logging space, that is, the space of all possible logging statements, is categorized according to some developer-chosen criteria –Repetitive events can be filtered Selection of log event severity or verbosity level

11. 11 Run-time log configuration Additionally, the power of using log4j design, is the possibility to enable logging (or features) at runtime without modifying the application binary or the wrapper layers. The picture shows a dedicated layer implemented as C++ abstract class library that serves as a proxy between the application framework and the distributed environment. The approach is designed so that the debugging statements can remain in shipped code without incurring a heavy performance cost. Logging equips the developer with as detailed context as necessary for application failures, from testing, quality assurance to a production mode limited amount of information.

12. 12 STAR logger configuration file

13. 13 The implementation of the STAR logger layer via log4cxx API allows to independently set the level of the verbosity for each STAR production “module”. It is convenient for the debug purposes of some particular piece of the code. Each developer knows his/ her module and can set the message level” for that module. Capability to include the special appenders like “MySQL”and “HTML” for certain critical points does allow to get the job status on-line (via remote Db or Web page) before the GRID deliveries the entire usually huge log file to dig through. If the needs of the special treatment arisen the custom dynamically loaded appender filter can be deployed also.