1. What is Computer Science?  Three paradigms (CACM 1/89) Theory (math): definitions, theorems, proofs, interpretations Abstraction (science): hypothesize, model, experiment, analyze data, results Design (engineering): state requirements, design, implement, test

2. How to choose a method, tool, technique?  Common sense  Consultants  Theoretical proof  Benchmark  Case study  Survey  Experimentation in the Small  Experimentation in the Large?

3. Experiment Design – Define/Describe  Subjects  Independent Variables and Levels  Dependent Variables  Confounding Variables (noise) and how controlled  Group Selection  Treatment vs Measurement

4. Analyzing Results  Descriptive statistics averages ranges standard deviations correlations  Inferential statistics t-test ANOVA

5. Correlations  from -1 to +1  “explains” variance  should include statistical significance  NOT causality  see examples …

6. Useful Experimentation in the Small  The Magic Number Seven, Plus or Minus Two; Miller 1956  Exploratory Experimental Studies Comparing Online and Offline Programming Performance; Sackman 1968  A Controlled Experiment in Program Testing and Code Walkthroughs / Inspections; Myers 1978  Reliability of Function Points Measurement – A Field Experiment; Kemerer 1993

7. To read a paper with empirical contributions ask the following:  What was the author’s primary interest?  How rigorous is the work? Internal validity External validity  What was the STRENGTH of the statistical results reported?  What serendipitous results were reported?

8. Lederer and Prasad Article  Discuss methodology questionnaire sent to 400 computing managers and professionals 115 responded responses analyzed for correlations between approaches used and success  Careful of cause and effect see footnote 2, page 53

9. Some interesting guidelines:  assign initial estimating task to the developers  delay finalizing initial estimate until after thorough study  monitor progress (and use external auditors)  use the estimate to evaluate personnel  don’t rely on cost estimating software!