How do you know your product “works”? And what does it mean for a product to “work”?

Purposes and Uses of Evidence  Common Uses of Evaluation Evidence  In Process Product Design and Modification  Gathering “proof” that the product works  Understanding how teachers and students use the product  Complying with grant or foundation giving requirements  Supporting product adoption decisions from piloting  Understand your purpose for collecting information at the beginning.  Design your evidence collection in a way that is consistent with those goals. SEG Measurement

Sources of Evidence and Ways to Collect It  Sources of Evidence  Students  Teachers  Administrators  Parents  Experts or other third parties  How to collect evidence  Pre/Post Testing (Cognitive and Affective  Anecdotes  Surveys  Interviews  Observation  Focus Groups  Product Usage Data  Amount its used  How its used SEG Measurement

Are you looking for “proof” that use of your product is the “cause” of improved learning outcomes?  “Proof” and “Causality” (The scientific model)  The “Medical model”  If we want to know if a drug works, we know you can’t just ask a bunch of people.  Random Assignment to treatment and control groups or matched groups designs  Pre and post testing  Sound statistical models to eliminate any potential initial differences or “interfering” variables SEG Measurement

Challenges  Getting results quickly  Desire for rapid cycle, iterative improvement  Costs (One good adoption won pays the bill)  Lack of Research Literacy (We offer pro bono assistance) SEG Measurement

How to prove your product is effective  Treatment and Control Group  Sufficient Sample Size  Statistical Power  Generalizability  Pre post assessment of desired outcomes (external, reliable and valid)  Fidelity of implementation  Qualitative and quantitative data  Third party conducted  Peer reviewed  Multiple studies SEG Measurement