1. Toward Replayable Research in Networking and Systems Eric Eide eeide@cs.utah.edu University of Utah, School of Computing May 25, 2010

2. “Replayable” Research the goal of the networking and systems communities should be replayable research the goal of the networking and systems communities should be replayable research repeatable repeatable network testbeds network testbeds data repositories data repositories experiment-management systems experiment-management systems replayable replayable fully encapsulated and easily re-executable fully encapsulated and easily re-executable 2

3. 3 Emulab machines machines 500+ PCs, and more 500+ PCs, and more time- & space-shared time- & space-shared loads OS and software loads OS and software network network config. topology & quality config. topology & quality services services file storage, email, … file storage, email, … available to researchers and educators worldwide available to researchers and educators worldwide

4. 4 Emulab Experiments expt. DB topology + SW (by reference) + events

5. 5 Resources Alone: Not Enough testbeds give you lots of resources… testbeds give you lots of resources… …but offer little help in using those resources …but offer little help in using those resources package / distribute / configure / instrument / init / execute / monitor / stop / collect / analyze / archive / revise / repeat package / distribute / configure / instrument / init / execute / monitor / stop / collect / analyze / archive / revise / repeat

6. 6 unmanned aerial vehicle receiver automatic target recognition images → ← images alerts → Example a distributed, real-time application a distributed, real-time application evaluate improvements to real-time middleware evaluate improvements to real-time middleware vs. CPU load vs. CPU load vs. network load vs. network load 4 research groups 4 research groups x 19 experiments x 19 experiments x 56 metrics x 56 metrics how to manage this? how to manage this?

7. 7 Needs my experiment needs… my experiment needs… encapsulation encapsulation automation automation instrumentation instrumentation preservation preservation benefits benefits verify previous results verify previous results establish base for new research establish base for new research my own, or someone else’s my own, or someone else’s package / distribute / configure / instrument / init / execute / monitor / stop / collect / analyze / archive / revise / repeat package / distribute / configure / instrument / init / execute / monitor / stop / collect / analyze / archive / revise / repeat

8. Emulab “Workbench” an Emulab-integrated environment for replayable research an Emulab-integrated environment for replayable research expt. management expt. management expt. execution expt. execution encapsulation encapsulation automation automation output data capture output data capture exploration exploration provenance provenance metadata metadata 8 n = 2 n = 4

9. Are We Replayable Yet? encapsulated expts + testbed = replay? 9 expt. DB +

10. Challenges to Replayability packaging is still too hard packaging is still too hard interactive use interactive use capturing semantics & metadata capturing semantics & metadata partly address “not worthwhile” perception partly address “not worthwhile” perception 10

11. Challenges to Replayability external events & nondeterminism external events & nondeterminism cost of capture cost of capture need to replay even when experiment is modified need to replay even when experiment is modified 11

12. Challenges to Replayability long-term access to physical resources long-term access to physical resources physical devices needed by systems research physical devices needed by systems research physical resources wear out [and/or lose funding] physical resources wear out [and/or lose funding] virtualized resources virtualized resources possible? useful? possible? useful? 12

13. Challenges to Replayability better definition of “replayable” better definition of “replayable” replayable with respect to chosen properties only replayable with respect to chosen properties only let experimenters be explicit about this let experimenters be explicit about this potential benefits potential benefits better design and instrumentation of expts better design and instrumentation of expts expts designed with replay in mind expts designed with replay in mind separating what matters from accidental detail separating what matters from accidental detail 13

14. Summary the goal of networking and systems researchers should be replayability the goal of networking and systems researchers should be replayability not just repeatable not just repeatable also easily re-executable also easily re-executable challenges challenges easier packaging easier packaging designing experiments for replay & reuse designing experiments for replay & reuse 14

15. http://www.emulab.net/ Thank you! Questions?