1. Job Scheduling: History and Evolution
November 17, 2010
Welkin Associates

2. Agenda The Scheduler Study Overview Brief History of Job Scheduling
Job Scheduler Terminology and Components
The Scheduler Study Overview
Brief History of Job Scheduling
Job Scheduler Functions
Job Scheduler Architectures
Evaluation of Job Schedulers and Vendors
Q/A

3. Job Scheduler Terminology and Components
Job: An executable unit. Can be a systems command, a script, an application program, or a collection of executable units. Jobs can have one or more steps, each containing an executable unit.
Job-Dependencies: Logical events or conditions that must be satisfied in order for jobs to be scheduled for execution. Predecessor and Successor.
Job-Flow: A collection of two or more jobs related by job-dependencies.
Schedule: The current active set of scheduled jobs and job-flows ready to be released for execution and monitored by the Master Scheduler.
Components
Master Scheduler: Central software component that defines, modifies, stores job and job-flow definitions, dependencies, security information, events, and schedules. Provides automatic submission of executables, monitors status of all jobs and job-flows, provides alerts, controls starting, stopping, restarting of jobs and job-flows. Creates Audit trail of all events.
Agent: A program that resides on each server that is controlled by the Master. The agent monitors jobs on its server, communicates with the Master about the status of jobs, and executes commands relayed by the Master.
GUI Console: Console display for Administrators and Operators.
Relational Database: The central repository used by the Master to store all job, job-flow, dependencies, security information, schedules and audit trails.

4. Job Scheduler Terminology and Components (Cont.)
Administrator
Operator
Agent
Job-Flows
Administrator GUI console
Operator GUI console
Agent
Job-Flows
Master Scheduler
Agent
Job-Flows
Logs
RDMS

5. Scheduler Study Overview
OPUS: Legacy Job Scheduler used for Satellite data processing
Developed by Space Telescope Science Institute (STScI ) in 1994 for HST
Very fast and efficient pipeline processor. Powerful job-flow features and GUI console
Adopted and customized for NOAA as a Job Scheduler on AIX p-series for migration from z/OS Mainframe (2005)
OPUS requires replacement by a COTS Scheduler:
STScI no longer supports OPUS
OPUS GUI written in Java has compatibility issues with latest version of Java
The Scheduler Study
Evaluated using two in-house Schedulers (NDE and STARS)
Evaluated COTS Schedulers and vendors.

6. Brief History of Job Scheduling
In the beginning… there were 80 column Hollerith cards
The term “batch” was derived from stacks of punched cards
The first true Job Schedulers were people and card readers
Therefore, originally Job Scheduling known as “batch processing”
1964 IBM OS/360 JCL presented first job dependency functions

7. Brief History of Job Scheduling (Cont.)
Early Mainframe era: Batch processing using card decks
Later Mainframe era: JCL on IBM Mainframes.
Jobs: Dependencies/Priorities/Queues/TOD-Calendar/Alerts/Reruns
Evolution of 3rd party vendor Mainframe Job Schedulers
IBM Tivoli Workload Scheduler (TWS)
CA 7 Mainframe Scheduler
BMC Control-M
ASG Zeke Scheduler
Unix systems: crons
Today: Job Scheduling has greatly matured, with a number of software vendors that have developed and marketed their Job Scheduler products for over 25 years

8. Brief History of Job Scheduling (Cont.)
Workload Automation Maturity Pyramid

9. Brief History of Job Scheduling (Cont.)
Gartner: The future automation model will be “IT Workload Automation Broker” (ITWAB)

10. Job Scheduler Functions
Create and edit jobs, job-flows, users, groups, schedules
Calendar scheduling
File-Watchers (Trigger file events)
Job parameters within job-flows available
Job Queues, Classes, Priority
Virtual resources to control jobs and files
High Availability / Failover capability
Reliable and fast Scheduler database with open schema
Scheduler commands to start, stop, suspend, resume, and restart jobs
Alerts generated for job errors and sent to Operators (GUI console, , cell)
Reporting tools
GUI console:
Windows, Web-based, Unix/Linux
Flexible and configurable for Administrators and Operators, easy to use
Graphical display of jobs and job-flows. Click and drag highly preferable
Detailed display of job-flow and job status, dependencies, errors/alerts

11. Job Scheduler Architectures
Architecture: Master-Agent vs. Peer-to-Peer

12. Job Scheduler Architectures (Cont.)
Architecture: Master – Agent

13. Research/Advisory Firms Job Scheduler Evaluations
EMA Criteria and Methodology for Evaluating Job Schedulers

14. Research/Advisory Firms Job Scheduler Evaluations (Cont.)
EMA Radar Report for Job Schedulers 01/2010

15. Research/Advisory Firms Job Scheduler Evaluations (Cont.)
November 2009 “Market Overview: Workload Automation, Q3 2009”
Workload Automation Vendors
Forrester List of Evaluated Job Schedulers

16. Research/Advisory Firms Job Scheduler Evaluations (Cont.)
The Gartner Magic Quadrant is copyrighted (04/28/2009) by Gartner, Inc., and is reused with permission. The Magic Quadrant is a graphical representation of a marketplace at and for a specific time period. It depicts Gartner’s analysis of how certain vendors measure against criteria for that marketplace, as defined by Gartner. Gartner does not endorse any vendor, product or service depicted in the Magic Quadrant, and does not advise technology users to select only those vendors placed in the “Leaders” quadrant. The Magic Quadrant is intended solely as a research tool, and is not meant to be a specific guide to action. Gartner disclaims all warranties, express or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
Source: Gartner, Inc.
Magic Quadrant for Job Scheduling
28 April 2009, Report Id Number: G
Milind Govekar, Gartner Analyst
Gartner Job Scheduler Magic Quadrant Report 04/2009

17. Q/A

18. Additional Slides for reference…

19. Workload Automation Example

20. Future Growth of WLA and Job Scheduling
Forrester: Forecast for Growth of WLA and Job Scheduling

21. Evaluation of Job Schedulers and Vendors
Scheduler Product Name
Open Source, Freeware, COTS
ASCI
ActiveBatch
COTS
Absyss
Visual TOM
ActivMindz
TaskForest
Open Source
APM Software GmbH
APX/PCC
Apple
XGRID
Argent
Job Scheduler
ASG
Zena for Distributed Systems
Zeke for z/OS Mainframes
Automation Anywhere
Ayehu
eyeShare
BMC
Control-M
Camellia Software
Batch Job Server (BJS)
CISCO
Tidal Enterprise Scheduler (TES)
Cluster Resources
Maui Cluster Scheduler
Computer Associates
AutoSys

22. Evaluation of Job Schedulers and Vendors
Scheduler Product Name
Open Source, Freeware, COTS
Corporate Practical Solutions
Bqplus
COTS
Cronless.com
cronless
Open Source
David G. Walker
php JobScheduler
Free Ware
Exact
JobBOSS
Exspans
Automan
Flux Corp.
Flux
Grand Logic
JobServer
Help/Systems
Robot/SCHEDULE Enterprise
Honico
Batchman
Hux Software
HQ Scheduler
IBM
Tivoli Workload Automation Suite
IndependIT
BICsuite Scheduler
ISE
EnterpriseSCHEDULE
JOX IT
JOX
LXI
LXI Scheduler

23. Evaluation of Job Schedulers and Vendors
Scheduler Product Name
Open Source, Freeware, COTS
MJTNet
Macro Scheduler
COTS
MVP Systems
JAMS
Neteject.com
VisualCron
Network Automation
Automate7
Nguyen Anthuan
SetCronJob
Free Ware
OAR Team
OAR
Open Source
Open Systems Mgmt
Cosbatch
OpsWise
Automation Center
ORSYP
Dollar Universe
OW2 Consortium
ProActive Parallel Suite
Platform Computing
Load Sharing Facility (LSF)
PTC Software
PTC Scheduler
Redwood Software
Cronacle
ROC Software
Maestro Scheduler
Runtime Design Automation
Network Computer (NC)
SMA
OpCon/xps
SoftTree Tech
24x7 Scheduler

24. Evaluation of Job Schedulers and Vendors
Scheduler Product Name
Open Source, Freeware, COTS
Sourceforge
Jcrontab
Open Source
Stonebranch
Indesca
COTS
Sun Microsystems
N1 Grid Engine
Sypsoft
Job2Do
Tenth Generation
Job Scheduler for Windows
Terracotta
Quartz Enterprise Job Scheduler
The Coens Group
Jobstream
UC4
Workload Automation Suite
Univa UD
Grid MP
University of Warwick
TITAN Grid Scheduler
University of Wisconsin
Condor
Vinzant
Global ECS (GECS)
Xi Software
GNUbatch
XuiS Software
EnterpriseSchedule