1. View from Experiment/Observation driven Applications Richard P. Mount May 24, 2004 DOE Office of Science Data Management Workshop

2. Richard P Mount View from Experiment/Observation Driven Science 2

3. 3

4. 4

5. 5

6. 6

7. 7

8. 8

9. 9

10. 10

11. Richard P Mount View from Experiment/Observation Driven Science 11

12. Richard P Mount View from Experiment/Observation Driven Science 12

13. Richard P Mount View from Experiment/Observation Driven Science 13

14. Richard P Mount View from Experiment/Observation Driven Science 14

15. Richard P Mount View from Experiment/Observation Driven Science 15

16. Richard P Mount View from Experiment/Observation Driven Science 16

17. Richard P Mount View from Experiment/Observation Driven Science 17

18. Richard P Mount View from Experiment/Observation Driven Science 18

19. Richard P Mount View from Experiment/Observation Driven Science 19

20. Richard P Mount View from Experiment/Observation Driven Science 20

21. Richard P Mount View from Experiment/Observation Driven Science 21

22. Richard P Mount View from Experiment/Observation Driven Science 22

23. Richard P Mount View from Experiment/Observation Driven Science 23

24. Richard P Mount View from Experiment/Observation Driven Science 24

25. Richard P Mount View from Experiment/Observation Driven Science 25

26. Richard P Mount View from Experiment/Observation Driven Science 26

27. Richard P Mount View from Experiment/Observation Driven Science 27

28. Richard P Mount View from Experiment/Observation Driven Science 28

29. Richard P Mount View from Experiment/Observation Driven Science 29

30. Richard P Mount View from Experiment/Observation Driven Science 30

31. Richard P Mount View from Experiment/Observation Driven Science 31

32. Richard P Mount View from Experiment/Observation Driven Science 32

33. Richard P Mount View from Experiment/Observation Driven Science 33

34. Richard P Mount View from Experiment/Observation Driven Science 34

35. Experiment/Observation Common Characterisitcs (Mildly Provocative)

36. Richard P Mount View from Experiment/Observation Driven Science 36 Experiment/Observation Common Characteristics Dominated by large, expensive devices and projects Correct project planning includes data- management hardware and software development –Not acceptable to build a $1Billion device and then face a Data-Management crisis –Development might be much more valuable if performed in a wider context Often hundreds or thousands of users Geographically distributed users

37. Richard P Mount View from Experiment/Observation Driven Science 37 Consequences of Common Characteristics Less worry about workflow management – part of the project from the start Multi-user concerns: –Keeping track of millions of data products (files?) created by people you barely know –Performance issues due to many concurrent queries –Data movement, grids and networks really matter to international collaborations Visualization can be a useful tool but rarely a major issue Responsiveness is a key issue –Taking months or years to answer a simple question is almost deadly

38. Final Comments and Pet Project Peddling

39. Richard P Mount View from Experiment/Observation Driven Science 39 Characterizing Scientific Data My petabyte is harder to analyze than your petabyte –Images (or meshes) are bulky but simply structured and usually have simple access patterns –Features are perhaps 1000 times less bulky, but often have complex structures and hard-to-predict access patterns

40. Richard P Mount View from Experiment/Observation Driven Science 40 Hydrogen Bubble Chamber Photograph 1970 CERN Photo

41. Richard P Mount View from Experiment/Observation Driven Science 41 Storage Issues Disks: –Random access performance is lousy, unless objects are megabytes or more independent of cost deteriorating with time at the rate at which disk capacity increases (Define random-access performance as time taken to randomly access entire contents of a disk)

42. Richard P Mount View from Experiment/Observation Driven Science 42 Latency and Speed – Random Access

43. Richard P Mount View from Experiment/Observation Driven Science 43 Latency and Speed – Random Access

44. The End