1. Dror Feitelson Hebrew University
The Forgotten Factor: FACTS on Performance Evaluation and its Dependence on Workloads
Dror Feitelson
Hebrew University
Thanks toparticipants and progam committee; thanks to Monien; abuse hospitality – talk about agenda

2. Performance Evaluation
In system design
Selection of algorithms
Setting parameter values
In procurement decisions
Value for money
Meet usage goals
For capacity planing
Important and basic activity

3. The Good Old Days… The skies were blue
The simulation results were conclusive
Our scheme was better than theirs
Focus on system design. Widely different designs lead to conclusive results.
Feitelson & Jette, JSSPP 1997

4. Their scheme was better than ours!
But in their papers,
Their scheme was better than ours!
But literature is full of contradictory results.

5. How could they be so wrong?
Leads to question of what is the cause for contradictions.

6. Performance evaluation depends on:
The system’s design
(What we teach in algorithms and data structures)
Its implementation
(What we teach in programming courses)
The workload to which it is subjected
The metric used in the evaluation
Interactions between these factors
Next: our focus is the workloads.

7. Performance evaluation depends on:
The system’s design
(What we teach in algorithms and data structures)
Its implementation
(What we teach in programming courses)
The workload to which it is subjected
The metric used in the evaluation
Interactions between these factors

8. Outline for Today
Three examples of how workloads affect performance evaluation
Workload modeling
Research agenda
In the context of parallel job scheduling
Job scheduling, not task scheduling

9. Example #1
Gang Scheduling and
Job Size Distribution

10. Gang What?!?
Time slicing parallel jobs with coordinated context switching
Ousterhout
matrix
Ousterhout, ICDCS 1982

11. Gang What?!?
Time slicing parallel jobs with coordinated context switching
Ousterhout
matrix
Optimization:
Alternative
scheduling
Ousterhout, ICDCS 1982

12. Packing Jobs Use a buddy system for allocating processors
Feitelson & Rudolph, Computer 1990

13. Packing Jobs Use a buddy system for allocating processors
Start with full system in one block

14. Packing Jobs Use a buddy system for allocating processors
To allocate repeatedly partition in two to get desired size

15. Packing Jobs
Use a buddy system for allocating processors

16. Packing Jobs Use a buddy system for allocating processors
Or use existing partition

17. The Question: The buddy system leads to internal fragmentation
But it also improves the chances of alternative scheduling, because processors are allocated in predefined groups
Which effect dominates the other?

18. The Answer (part 1): Feitelson & Rudolph, JPDC 1996
Answer as function of workload, but not full answer because workload unknown. Dashed lines: provable bounds.
Feitelson & Rudolph, JPDC 1996

19. The Answer (part 2):
Note logarithmic Y axis

20. The Answer (part 2):

21. The Answer (part 2):

22. The Answer (part 2): Many small jobs Many sequential jobs
Many power of two jobs
Practically no jobs use full machine
Conclusion: buddy system should work well

23. Verification
Using Feitelson workload
Feitelson, JSSPP 1996

24. Parallel Job Scheduling
Example #2
Parallel Job Scheduling
and Job Scaling

25. Variable Partitioning
Each job gets a dedicated partition for the duration of its execution
Resembles 2D bin packing
Packing large jobs first should lead to better performance
But what about correlation of size and runtime?
First-fit decreasing is optimal

26. Scaling Models Constant work Constant time Memory bound
Parallelism for speedup: Amdahl’s Law
Large first  SJF
Constant time
Size and runtime are uncorrelated
Memory bound
Large first  LJF
Full-size jobs lead to blockout
Question is which model applies within the context of a single machine
Worley, SIAM JSSC 1990

27. “Scan” Algorithm
Keep jobs in separate queues according to size (sizes are powers of 2)
Serve the queues Round Robin, scheduling all jobs from each queue (they pack perfectly)
Assuming constant work model, large jobs only block the machine for a short time
But the memory bound model would lead to excessive queueing of small jobs
Important point: schedule order determined by size
Krueger et al., IEEE TPDS 1994

28. The Data
Data: SDSC Paragon, 1995/6

29. The Data Data: SDSC Paragon, 1995/6
Partitions with equal numbers of jobs; many more small jobs.
Data: SDSC Paragon, 1995/6

30. The Data Data: SDSC Paragon, 1995/6
Similar range, different shape; 80th percentile moves from <1m to several h.
Data: SDSC Paragon, 1995/6

31. Conclusion Parallelism used for better results, not for faster results
Constant work model is unrealistic
Memory bound model is reasonable
Scan algorithm will probably not perform well in practice

32. User Runtime Estimation
Example #3
Backfilling and
User Runtime Estimation

33. Backfilling
Variable partitioning can suffer from external fragmentation
Backfilling optimization: move jobs forward to fill in holes in the schedule
Requires knowledge of expected job runtimes

34. Variants EASY backfilling Make reservation for first queued job
Conservative backfilling
Make reservation for all queued jobs

35. User Runtime Estimates
Lower estimates improve chance of backfilling and better response time
Too low estimates run the risk of having the job killed
So estimates should be accurate, right?

36. They Aren’t Mu’alem & Feitelson, IEEE TPDS 2001
Short=failed; killed typically exceeded runtime estimate, ~15%
Mu’alem & Feitelson, IEEE TPDS 2001

37. Surprising Consequences
Inaccurate estimates actually lead to improved performance
Performance evaluation results may depend on the accuracy of runtime estimates
Example: EASY vs. conservative
Using different workloads
And different metrics
Will focus on second bullet

38. EASY vs. Conservative
Using CTC SP2 workload

39. EASY vs. Conservative Using Jann workload model
Note: jann model of CTC

40. EASY vs. Conservative
Using Feitelson workload model

41. Conflicting Results Explained
Jann uses accurate runtime estimates
This leads to a tighter schedule
EASY is not affected too much
Conservative manages less backfilling of long jobs, because respects more reservations
Relative measure: more by EASY = less by conservative

42. Conservative is bad for the long jobs Good for short ones that are respected Conservative EASY

43. Conflicting Results Explained
Response time sensitive to long jobs, which favor EASY
Slowdown sensitive to short jobs, which favor conservative
All this does not happen at CTC, because estimates are so loose that backfill can occur even under conservative

44. Verification
Run CTC workload with accurate estimates

45. But What About My Model?
Simply does not have such small long jobs

46. Workload Modeling

47. No Data Innovative unprecedented systems Use an educated guess
Wireless
Hand-held
Use an educated guess
Self similarity
Heavy tails
Zipf distribution

48. Serendipitous Data Data may be collected for various reasons
Accounting logs
Audit logs
Debugging logs
Just-so logs
Can lead to wealth of information

49. NASA Ames iPSC/860 log 42050 jobs from Oct-Dec 1993
user job nodes runtime date time
user cmd /10/93 10:13:17
user cmd /10/93 10:19:30
user nqs /10/93 10:22:07
user cmd /10/93 10:22:37
sysadmin pwd /10/93 10:22:42
user cmd /10/93 10:25:42
sysadmin pwd /10/93 10:30:43
user cmd /10/93 10:31:32
Feitelson & Nitzberg, JSSPP 1995

50. Distribution of Job Sizes

51. Distribution of Job Sizes

52. Distribution of Resource Use

53. Distribution of Resource Use

54. Degree of Multiprogramming

55. System Utilization

56. Job Arrivals

57. Arriving Job Sizes

58. Distribution of Interarrival Times

59. Distribution of Runtimes

60. Job Scaling

61. User Activity

62. Repeated Execution

63. Application Moldability
Of jobs run more than once

64. Distribution of Run Lengths

65. Predictability in Repeated Runs
For jobs run more than 5 times

66. Recurring Findings Many small and serial jobs Many power-of-two jobs
Weak correlation of job size and duration
Job runtimes are bounded but have CV>1
Inaccurate user runtime estimates
Non-stationary arrivals (daily/weekly cycle)
Power-law user activity, run lengths

67. Research Agenda

68. The Needs New systems tend to be more complex
Differences tend to be finer
Evaluations require more detailed data
Getting more data requires more work
Important areas:
Internal structure of applications
User behavior

69. Generic Application Model
Iterations of
Compute
granularity
Memory working set / locality
I/O
Interprocess locality
Communicate
Pattern, volume
Option of phases with different patterns of iterations
compute
I/O
communicate
At least for SPMD; not for pipeline

70. Consequences Model the interaction of the application with the system
Support for communication pattern
Availability of memory
Application attributes depend on system
Effect of multi-resource schedulers
Common occurrence in benchmarking that different benchmarks rank systems differently

71. Missing Data
There has been some work on the characterization of specific applications
There has been no work on the distribution of application types in a complete workload
Distribution of granularities
Distribution of working set sizes
Distribution of communication patterns

72. Effect of Users Workload is generated by users
Human users do not behave like a random sampling process
Feedback based on system performance
Repetitive working patterns

73. Feedback User population is finite
Users back off when performance is inadequate
Negative feedback
Better system stability
Need to explicitly model this behavior

74. Locality of Sampling
Users display different levels of activity at different times
At any given time, only a small subset of users is active
These users repeatedly do the same thing
Workload observed by system is not a random sample from long-term distribution
Can this be exploited by a scheduler?

75. Final Words…

76. We like to think that we design systems based on solid foundations…

77. But beware:
the foundations might be unbased assumptions!

78. Computer Systems are Complex
We should have more “science” in computer science:
Collect data rather than make assumptions
Run experiments under different conditions
Make measurements and observations
Make predictions and verify them
Science = experimental scince, like physics, chemistry, biology

79. Advice from the Experts
“Science if built of facts as a house if built of stones. But a collection of facts is no more a science than a heap of stones is a house”
-- Henri Poincaré

80. Advice from the Experts
“Science if built of facts as a house if built of stones. But a collection of facts is no more a science than a heap of stones is a house”
-- Henri Poincaré
“Everything should be made as simple as possible, but not simpler”
-- Albert Einstein

81. Acknowledgements Students: Ahuva Mu’alem, David Talby, Uri Lublin
Larry Rudolph / MIT
Data in Parallel Workloads Archive
Joefon Jann / IBM
CTC SP2 log
SDSC Paragon log
SDSC SP2 log
NASA iPSC/860 log