S519: Evaluation of Information Systems Analyzing data: Causation Ch5

Step5: Analyzing data Dealing with the causation issue, basically be able to answers following questions: How certain does the client need us to say that the evaluand „caused“ a certain change? What are the basic principles for inferring causation? What types of evidence do we have available to help us identify or rule out possible causal links? How should we decide what blend of evidence will generate the level of certainty needed most cost- effectively?

Certainty about causation (D- ch5) Each decision-making context requires a different level of certainty Quantitative or qualitative analysis All-quantitative or all-qualitative Sample choosing Sample size Mix of them

Inferring causation: basic principles Two basic principles: Look for evidence for and against the suspected main cause (i.e., evaluand) Look for evidence for and against any important alternative causes (i.e., rival explanations) Too many evidences or causes, which are the primary causes All based on the level of certainty you need for your evaluation Stepwise process Put yourself in the hardest critics, gather enough evidence to support your explanation  Repeat it until all remaining alternative explanations are ruled out. Critical multiplism: triangle The harder people attack, the more solid your answers need to be.

Inferring causation: 8 strategies 1: ask observers Two possibilities Ask actual or potential impactees Ask indirect impactees (i.e., co-worker, parents...) Design your interview questions to include causation questions E.g., how much has your knowledge increased as a result of attending this program? – get primary cause E.g., did anything else besides the program increase your knowledge in this area over the same period of time? – get other causes E.g., please describe anything else that has happened to you or someone you know as a result of participating in this program? – get the causes which people know or believe were caused by the program.

Inferring causation: 8 strategies 1: ask observers Causation-rich questions tend to be leading (direct the respondent to answer in a particular way). Be careful about the wording when designing interview questions The causation question is not just whether the program produced the effect but also what other factors enabled or inhibited the effect. Individual might not be a reliable witness to answer the causation question, other evidence will be required to make justifiable causal inferences.

Inferring causation: 8 strategies 1: ask observers Methods Questionnaires to identify the targeted groups (people who experienced substantial changes)  Using open-end to get more opinions In-depth interview with the targeted groups  To find out causation.

Inferring causation: 8 strategies 2. Check whether the content of the evaluand matches the outcome Alcoholics treatment program  alcoholics avoid relapses Check whether the strategies which alcoholics use to avoid relapses after the program, are the same as the strategies taught in the program

Inferring causation: 8 strategies 3. Look for other telltale patterns that suggest one cause or another Modus operandi method – look for evidence -- detective metaphor to describe the way in which potential causal explanations are identified and tested. A silly example Evidences: a naked man, dead; in the middle of the desert; personal belongings near by; half match at hand Cause of his death

Inferring causation: 8 strategies 4. Check whether the timing of outcomes makes sense Common sense: an outcome should appear only at the same time as or after whatever caused it – a considerable delay. A further downstream the outcomes, the longer they should take to appear Using timing to confirm or disconfirm causal links: Is the outcome happened before the evaluation? Or Other downstream outcomes too early? Is the timing of the outcomes logical to possible causes? Do the further downstream outcomes in the logic model occur out of sequence? More on Lipsey, M. W. (1989). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage.

Inferring causation: 8 strategies 4. Check whether the timing of outcomes makes sense Example – a community health education on diet and exercise Fairly immediate knowledge and skill gain: during or immediately after the intervention A short delay (days or weeks) before the knowledge and skills are transformed into changing behavior A moderate delay (weeks or months) before we see changes in individual health indicators (weight, cholesterol, blood pressure, etc.) A long delay (months or years) to see changes on improvement on diabetes and heart diseases

Exercise Take grantsmanship workshop as one example List the timeline potential outcomes (fairly immediate, a short delay, a moderate delay, a long delay) Using timing strategies to confirm or disconfirm the cause links, state one page for how and why: One month after the workshop, 3 proposals got grants One year after the workshop, 3 proposals got grants Three months after the workshop, some people write good proposals, but some are not. Think about your own solution Form a group and discuss Lab

Inferring causation: 8 strategies 5. Check whether the „dose“ is related logically to the „response“. The dose-response idea The more dose of drug, the better response later on If more A (dose), then better B (response) Compare the less dose with more dose (not overdose) to confirm or disconfirm the cause links  E.g., for performance evaluation project, if we found that performance had been improved dramatically in the unit where the system has been poorly implemented,  this system is not the cause of the performance improvement.

Inferring causation: 8 strategies 6. Make comparisons with a „control“ or „comparison“ group Divide the participants into different groups control group (receive the evaluand) vs. Comparison group (receive no evaluand) Sampling should be done carefully to make sure no systematic differences between groups Sample size randomization

Inferring causation: 8 strategies 7. Control statistically for extraneous variables When using control and comparison groups, try to exclude external variables and make two groups no systematic differences: Statistical methods  Regression analysis Try to identify other potential systematic differences  E.g., math improvement for students, how to sample students and think about other potential existing difference. Is the random sampling enough?

Inferring causation: 8 strategies 8. Identify and check the underlying causal mechanism(s) Try to look for an underlying mechanism to make the case for causation more or less convincing Cigarette smoking  lung cancer  Correlation studies  Carcinogenic in cigarette causes cancer Normally coming from literature.

Put them together Do we need all the evidences we collect from 8 strategies? How to select them? Put yourself in the shoes of a tough critic, identify the most potential threatening rival explanation, then chose the types of evidence that will most quickly and cost-effectively confirm or dispel that rival explanation. Go to next less tough rival explanation,... Continue, until you have amassed a body of evidence to provide you enough certainty to draw causal inferences

Exercise Grantsmanship workshop (p57) Grantsmanship workshop strengthen local communities For (evidences) Against (evidences) Other alternative causes (i.e., rival explanations) Using strategies to confirm or disconfirm these evidences or causes Putting them together Form a group to discuss Lab