1. Kay Ousterhout, Christopher Canel, Sylvia Ratnasamy, Scott Shenker
Monotasks Architecting for Performance Clarity in Data Analytics Frameworks
Kay Ousterhout, Christopher Canel, Sylvia Ratnasamy, Scott Shenker

2. How can I make this faster?
Starting abstractly: have a user, she’d running a workload on some system and getting some result. Pretty quickly asks: how can I make this faster?

3. How can I make this faster?
She has lots of choices for how to do so. Lots of knobs to turn! To give a few examples…

4. Should I use a different cloud instance type?
Change instance type! Can have significant impact (Shivaram’s work: 1.9x)

5. Should I trade more CPU for less I/O by using better compression?
Change software config. How do I know what effect of different tradeoff will be?

6. How can I make this faster?
???
User has no way to know how to turn these knobs – yet could get big performance improvements from doing so!

7. How can I make this faster?
???
What’s more, this is getting worse: with each new optimization, there are more knobs, and the systems are more complicated, so it’s harder to know how to turn them

8. How can I make this faster?
???

9. Architect for performance clarity
Make it easy to reason about performance
We should architect for performance clarity. What do I mean by clarity? Much more than just “what’s the bottleneck”. Want to know how to change the knobs; this requires knowing how system performs under different conditions. No measure of utilization will tell me what happens if network is twice as fast. This goes way beyond measurement. Not aware of any system that was designed to provide this kind of clarity.
Enabling the user to turn the existing N knobs is much more useful than adding an N+1th one!

10. For data analytics frameworks: Is it possible to architect for performance clarity? Does doing so require sacrificing performance?
Intuitively: making performance easier to reason about might mean making system simpler, undoing performance improvements, and making it slower as a result

11. Key idea: use single-resource monotasks
Reasoning about performance Today: why it’s hard Monotasks: why it’s easy Does using monotasks hurt performance? Using monotasks to predict job runtime

12. Split input file into words and emit count of 1 for each
Example Spark Job
Word Count:
spark.textFile(“hdfs://…”) \
.flatMap(lambda l: l.split(“ “)) \
.map(lambda w: (w, 1)) \
.reduceByKey(lambda a, b: a + b) \
.saveAsTextFile(“hdfs://…”)
Split input file into words and emit count of 1 for each
User writes a high level description of job. Two parts, first…

13. Example Spark Job Word Count:
spark.textFile(“hdfs://…”) \
.flatMap(lambda l: l.split(“ “)) \
.map(lambda w: (w, 1)) \
.reduceByKey(lambda a, b: a + b) \
.saveAsTextFile(“hdfs://…”)
Split input file into words and emit count of 1 for each
For each word, combine the counts, and save the output
Second part

14. Spark Word Count Job: … …
spark.textFile(“hdfs://…”)
.flatMap(lambda l: l.split(“ “))
.map(lambda w: (w, 1))
.reduceByKey(lambda a, b: a + b)
.saveAsTextFile(“hdfs://…”)
Map Stage: Split input file into words and emit count of 1 for each
Reduce Stage: For each word, combine the counts, and save the output …
Worker 1
Worker n …
Worker 1
Worker n
Tasks
The framework’s job is to handle the details of executing the job: break it into tasks that can be parallelized on a cluster, handle data transfer, etc.
Aside: this is how Spark works, but lots of other frameworks too!

15. Reduce Stage: For each word, combine the counts, and save the output
Spark Word Count Job:
.reduceByKey(lambda a, b: a + b)
.saveAsTextFile(“hdfs://…”)
Reduce Stage: For each word, combine the counts, and save the output …
Worker 1
Worker n
Focus on just reduce stage

16. Reduce Stage: For each word, combine the counts, and save the output
Spark Word Count Job:
.reduceByKey(lambda a, b: a + b)
.saveAsTextFile(“hdfs://…”)
Reduce task:
Reduce Stage: For each word, combine the counts, and save the output
Network …
Worker 1
Worker n
CPU (aggregate)
Disk write
Tasks use pipelining to parallelize the use of many resources

17. Challenge: Tasks pipeline multiple resources,
Reduce task
Network read
CPU (filter)
Disk write
Task only using disk
Task only using the network
Task only using the CPU
Challenge: Tasks pipeline multiple resources,
resource use changes at fine time granularity
Fine grained pipelining makes it hard to reason about performance, because the task is doing different things at different times, and this can change at the granularity of the pipelining.
This is how things have always been built: fine-grained pipelining seen as fundamental to performance! But it’s also a fundamental challenge to reasoning about performance. This results in a few issues…

18. 4 concurrent tasks on a worker
First, tasks running concurrently!
time

19. Concurrent tasks may contend for
the same resource
(e.g., network)
Task 7
Task 3
Task 4
Task 8
A single scheduler is responsible for scheduling these multi-resource tasks! Can’t avoid contention that occurs at fine time granularity
time

20. Challenge: Resource use controlled by operating system
Task 18
Task 19
Reduce task
Network read
CPU (filter)
Disk write
Disk write controlled by OS buffer cache
Alternate Disk write
(data in buffer cache)
Disk write controlled by OS! OS might reasonably hold all data in the cache, and actually write it to disk much later, when it will contend with other tasks running on the machine.

21. Time t: different tasks may be bottlenecked on different resources
What’s the bottleneck?
Single task may be bottlenecked on different resources at different times
Task 1
Task 2
Task 5
Task 3
Task 4
Task 7
Task 6
Task 8
Time t: different tasks may be bottlenecked on different resources
Now given what I’ve said so far, say you wanted to answer a simple question: what’s the bottleneck?
This starts to look pretty hairy pretty quickly. Can *you* answer this from looking at this picture?
If you look at one slice in time, different tasks may be bottlenecked on different things!
You could think about this for one task: but the task has different bottlenecks at different times!

22. How much faster would my job be with 2x disk throughput?
Task 1
Task 2
Task 5
Task 3
Task 4
Task 7
Task 6
Task 8
How much faster would my job be with 2x disk throughput?
How would runtimes for these disk writes change?
How would that change timing of (and contention for) other resources?
What if you wanted to ask a more complicated question?
Emphasize: no amount of additional logging is going to make it easy to answer this question today.

23. Fundamental challenge: tasks have non-uniform resource use
Concurrent tasks on a machine may contend
Resource use is controlled outside of the framework
No model for performance
Contention exacerbated by the fact that resource use controlled outside of framework

24. Reasoning about performance Today: why it’s hard Monotasks: why it’s easy Does using monotasks hurt performance? Using monotasks to predict job runtime

25. Today: tasks use pipelining to parallelize multiple resources Proposal: build systems using monotasks that each consume just one resource

26. Today: Tasks have non-uniform resource use
Concurrent tasks may contend
Resource use outside of framework
No model for performance

27. Monotasks: Each task uses one resource
Today:
Network read
Today’s task:
Tasks have non-uniform resource use
CPU
Disk write
Task 1
Concurrent tasks may contend
Monotasks: Each task uses one resource
Network monotask
Compute monotask
Resource use outside of framework
Disk monotask
Monotasks don’t start until all dependencies complete
No model for performance
Dependencies: key to uniform resource use! Don’t want tasks to block.

28. Today: Monotasks: Dedicated schedulers control contention
Tasks have non-uniform resource use
Monotasks for one of today’s tasks:
Each task uses one resource
Network scheduler
Concurrent tasks may contend
CPU scheduler:
1 monotask / core
Resource use outside of framework
Disk drive scheduler:
1 monotask / disk
No model for performance

29. Per-resource schedulers have complete control Today: Monotasks:
Tasks have non-uniform resource use
Each task uses one resource
Network scheduler
Concurrent tasks may contend
Dedicated schedulers control contention
CPU scheduler:
1 monotask / core
Resource use outside of framework
writes buffered
Disk drive scheduler:
1 monotask / disk
All writes flushed to disk
No model for performance

30. Monotask times can be used to model performance
Today:
Monotasks:
Monotask times can be used to model performance
Tasks have non-uniform resource use
Each task uses one resource
Ideal CPU time: total CPU monotask time / # CPU cores
Concurrent tasks may contend
Dedicated schedulers control contention
Resource use outside of framework
Per-resource schedulers have complete control
No model for performance

31. Monotask times can be used to model performance
Today:
Monotasks:
Monotask times can be used to model performance
Tasks have non-uniform resource use
Each task uses one resource
Ideal CPU time: total CPU monotask time / # CPU cores
Modeled job runtime:
max of ideal times
Concurrent tasks may contend
Dedicated schedulers control contention
Resource use outside of framework
Per-resource schedulers have complete control
Ideal network runtime
Ideal disk runtime
No model for performance

32. How much faster would the job be with 2x disk throughput?
Today:
Monotasks:
How much faster would the job be with 2x disk throughput?
Tasks have non-uniform resource use
Each task uses one resource
Ideal CPU time: total CPU monotask time / # CPU cores
Concurrent tasks may contend
Dedicated schedulers control contention
Ideal network runtime
Resource use outside of framework
Per-resource schedulers have complete control
Ideal disk runtime
No model for performance

33. How much faster would the job be with 2x disk throughput?
Today:
Monotasks:
How much faster would the job be with 2x disk throughput?
Tasks have non-uniform resource use
Each task uses one resource
Ideal CPU time: total CPU monotask time / # CPU cores
Concurrent tasks may contend
Dedicated schedulers control contention
Modeled new job runtime
Ideal network runtime
Resource use outside of framework
Per-resource schedulers have complete control
Ideal disk runtime
(2x disk concurrency)
Monotask times can be used to model performance
No model for performance
Emphasize: model simple (and not perfectly accurate!). But as I’ll show later, sufficiently accurate to answer what-if questions.

34. How does this decomposition work?
Today:
Monotasks:
4 multi-resource tasks run concurrently
Tasks have non-uniform resource use
Each task uses one resource
Concurrent tasks may contend
Dedicated schedulers control contention
How does this decomposition work?
Resource use outside of framework
Monotasks scheduled by per-resource schedulers
Per-resource schedulers have complete control
Monotask times can be used to model performance
No model for performance
Note that simple performance questions (e.g., bottleneck, time on each resource) are trivial

35. How does this decomposition work?
.reduceByKey(lambda a, b: a + b).saveAsTextFile(“hdfs://…”)
Network monotasks: request remote data
Today’s reduce task:
Network
CPU
Disk write
Disk monotask: write output …
CPU monotask: deserialize, combine counts, serialize
Framework can do this under the hood!
Key insight: possible for the same reasons Spark is widely-used: high-level abstraction
API-compatible with Spark, implemented at application layer

36. Reasoning about performance Today: why it’s hard Monotasks: why it’s easy Does using monotasks hurt performance? Using monotasks to predict job runtime

37. Does using monotasks hurt performance?
3 benchmark workloads: Big data benchmark (10 queries run with scale factor 5) Sort (600GB sorted using 20 machines) Block coordinate descent (ML workload, 16 machines) For all workloads, runtime comparable to Spark At most 9% slower, sometimes faster

38. How much faster would jobs run if…
Each machine had 2 disks instead of 1?
Sort 600GB of key-value pairs on 20 machines
Takeaway: monotask model correctly predicts new runtimes. Note that this is non-trivial (e.g., no runtimes increased by a factor of 2 because something else became the bottlencek), and monotasks model gets it correct in both cases.
Predictions within 9% of the actual runtime

39. How much faster would job run if...
4x more machines
Input stored in-memory
No disk read
No CPU time to deserialize
Flash drives instead of disks
Faster shuffle read/write time
10x improvement predicted with at most 23% error

40. Leveraging Performance Clarity to Automatically Improve Performance
Schedulers have complete visibility over resource use
Network scheduler
CPU scheduler:
1 monotask / core
Can configure for best performance
Disk drive scheduler:
1 monotask / disk

41. Leveraging Performance Clarity to Automatically Improve Performance
Monotask schedulers automatically select ideal concurrency
Monotasks allow you to get performance improvements that it would be hard to get any other way

42. Why do we care about performance clarity?
By using single-resource monotasks, system can provide performance clarity without sacrificing performance
Why do we care about performance clarity?
Typical performance eval: group of experts
Practical performance: 1 novice
Insight from this talk: it *is* possible to architect for performance clarity, can do so with single-resource tasks
Why do we care? We need to build systems that perform well for the average user

43. Reflecting on Monotasks
Painful to re-architect existing system to use monotasks
Pipelining deeply integrated (>20K lines of code changed)
Implemented at high layer of software stack
Should clarity be provided by the operating system?

44. Goal: provide performance clarity
Only way to improve performance is to know what to speed up
Using single-resource monotasks provides clarity without sacrificing performance
With monotasks, easier to improve system performance
only way to know how to make things faster is to know what you need to speed up!
Insight from this talk: it *is* possible to architect for performance clarity, can do so with single-resource tasks
With this one architectural change, can make lots of things easy – both for user and via automated tools.
github.com/NetSys/spark-monotasks