1. Network Computing Laboratory Experiment Tutorial Network Computing Lab. 2005. 5. 19

2. Network Computing Laboratory | 2 Korea Advanced Institute of Science and Technology Table of contents What is experiment? What is experiment design? Case study - CIGMA Three keys for experiment Step by step for experiment

3. Network Computing Laboratory | 3 Korea Advanced Institute of Science and Technology What is experiment? Proof of your proposal Convince people that your goal is well achieved Make your contribution to be clear and true Support that your technical problems are well solved

4. Network Computing Laboratory | 4 Korea Advanced Institute of Science and Technology What is experiment design? Experiment design 실험 계획표 작성 (Pre-report) 실험 목적 1. Decide key aspects you want to show 2. Choose the proper verification method Demo, implementation and measurement, simulation, and analysis 3. Identify metrics (dependent variables) 4. Classify parameters (independent variables) 실험 환경 설정 1. Decide key system components to be implemented Plan to test your implemented component 2. Prepare workload (benchmark, trace, synthetic workload) 3. Plan to prepare test-bed (machine, OS, software, network link, simulation tools) 실험 예상 1. Draw the expected graphs which you want to show (imaginary experiment w/ paper and pencil) Assignment Presentation slides of experiment design for your own experiment

5. Network Computing Laboratory | 5 Korea Advanced Institute of Science and Technology Verification methods 1. Analysis 예 ) Quick sort = O(N log N)) Deterministic and Statistical Analysis 2. Simulation 예 ) Network protocol simulation (topology, bandwidth, link delay) 실험환경과 시스템을 모두 모델링 해서 검증하는 방법 Simulation 은 언제 하는가 ?? 변수가 많고 시나리오가 복잡해서 분석이 불가능 할 때 구현 및 측정하기가 어렵거나 불가능 할 때 구현 및 측정하기 앞서서 미리 메커니즘 등을 검증하고 싶을 때 장점 : Easy to control, 단점 : Unrealistic 3. Implementation and measurement 예 ) 웹 서버 구현 후, 초당 처리 수 측정

6. Network Computing Laboratory | 6 Korea Advanced Institute of Science and Technology Case study – CIGMA: Motivation CIGMA: aCtive Inventory service in Global e-MArket based on efficient catalog management Highly dynamic e-commerce environment Constantly changing sales item information Flooding information: a lot of shopping sites “I want to know how I can purchase goods with the best condition!!”

7. Network Computing Laboratory | 7 Korea Advanced Institute of Science and Technology Case study – CIGMA: Goal CIGMA: aCtive Inventory service in Global e-MArket based on efficient catalog management CIGMA service and system Provide aggregated catalog information from a lot of sites Assure up-to-date information Serve large number of customers with fast response

8. Network Computing Laboratory | 8 Korea Advanced Institute of Science and Technology Case study – CIGMA: Design and Implementation

9. Network Computing Laboratory | 9 Korea Advanced Institute of Science and Technology Case study – CIGMA: Experiment Questions!! What kinds of experiments do you want to do? Why do you want to do those experiments?

10. Network Computing Laboratory | 10 Korea Advanced Institute of Science and Technology Case study – CIGMA: Experiment What and Why - “Remind your goal and contribution” CIGMA should deal with a lot of shopping sites. So it should process a large number of update requests from multiple sites Catalog update throughput CIGMA should assure up-to-date information. So update delay should be small Catalog update delay from shopping site to CIGMA CIGMA should handle a lot of customers requests. So it should process a number of customer requests at the same time Catalog browse throughput CIGMA receives update requests and customer requests concurrently. So it should process both of requests Mixed throughput

11. Network Computing Laboratory | 11 Korea Advanced Institute of Science and Technology Case study – CIGMA: Experiment One more Question!! How do you do those experiments?

12. Network Computing Laboratory | 12 Korea Advanced Institute of Science and Technology Case study – CIGMA: Experiment What to do after you decide metrics such as Catalog update throughput Catalog update delay Catalog browse throughput Mixed throughput What do you need in order to measure those metrics?? Do you have machines and software needed for implementation? Do you have your implementation which is working well? Do you have something to generate the workload? What kinds of parameters affect the experimental results?? What is your expectation about your experimental results??

13. Network Computing Laboratory | 13 Korea Advanced Institute of Science and Technology Case study – CIGMA: Experiment What do you need in order to measure those metrics?? Do you have machines and software needed for implementation?  Prepare your machines and software. Pentium III 1GHz, 2GB memory machine, 100M LAN Link, Linux OS, Tomcat server, FastDB Main Memory DB … Do you have your implementation which is working well?  Implement your system’s core functionality to be verified and test your system.

14. Network Computing Laboratory | 14 Korea Advanced Institute of Science and Technology Case study – CIGMA: Experiment What do you need in order to measure those metrics?? Do you have something to generate the workload?  Use existing benchmark tools. If not available, implement your own workload generator. Browse throughput: Httperf benchmark used (c.f. TPC-W, SPEC-Web) Update throughput: Update request generator implemented Update delay: WAN emulation tool used

15. Network Computing Laboratory | 15 Korea Advanced Institute of Science and Technology Case study – CIGMA: Experiment What kinds of parameters affect the experimental results??  Find possible parameters  Plan to choose and vary just one parameter at a time What is your expectation about your experimental results??  Draw your expected graphs which show not only a pattern of result (linearly increasing, logarithmically increasing, decreasing, or keeping constant) but also specific numbers Help you to analyze your measured results  Calculate your expected results mathematically

16. Network Computing Laboratory | 16 Korea Advanced Institute of Science and Technology Case study – CIGMA: Experiment And then?? Experiment!! Experiment!! Experiment!! Analyze your results Verify your expectations Modify your parameters, implementations, even design

17. Network Computing Laboratory | 17 Korea Advanced Institute of Science and Technology Case study – CIGMA: Experiment Example result graphs

18. Network Computing Laboratory | 18 Korea Advanced Institute of Science and Technology Three keys for experiment What Why How

19. Network Computing Laboratory | 19 Korea Advanced Institute of Science and Technology Three keys for experiment What What do you want to show to people? Strongly connected to what you propose as your system goals and your contributions Quantitative Q. What kinds of metrics can you use? E.g. Throughput, latency, execution time, storage usage, # of Disk I/O, # of lookups, % of failed lookups, BW consumption, etc. vs. Qualitative Q. What are core functionalities to show? E.g. Demo, screenshots

20. Network Computing Laboratory | 20 Korea Advanced Institute of Science and Technology Three keys for experiment Why Why do you want to show it to people? Also strongly connected to your system goals and your contributions Q. Does it really prove your contribution? Q. Does it show your achievement more effectively than others?

21. Network Computing Laboratory | 21 Korea Advanced Institute of Science and Technology Three keys for experiment How How can you show it to people? Experiments using real implementation and benchmark tool Simulations Demos

22. Network Computing Laboratory | 22 Korea Advanced Institute of Science and Technology Step by step for experiment Determining What and Why Setting experimental environment Implementation Benchmark tools While (hypothesis verified to be true || your approach turn to be wrong || give-up) { Making hypothesis Expect results and draw your expected graphs Verifying hypothesis Do experiment and get results Analyze results Giving feedback from your analysis To experimental parameters, system tuning To your implementation To your system design } Explaining your results and lessons if you succeed