1. Unités de recherche éducationnelle/Educational Research Units L’adulte et l’apprentissage en milieu de travail/ Adult and Workplace Learning Mesure et évaluation des apprentissages/ Measurement, Evaluation and Assessment Séminaire d’après-midi/Afternoon Seminar 30 janvier 2013/January 30 th, 2013 De 16h à 17h15/4:00 to 5:15 p.m. LMX 388 La présentation sera en anglais/Presentation will be in English Des rafraichissements seront servis/Refreshments will be served RSVP avant le 28 janvier à : RSVP before January 28 th at: claire.duchesne@uottawa.ca Tips on how to develop measurement tools and collect data Ivana Pavic, Maurice Taylor, Sait Atas and David Trumpower

2. Tips on how to develop measurement tools Involve your learners and instructors in the development (Ivana) Ensure the tool has a strong conceptual development (Maurice) Pilot test the tool to obtain evidence of reliability, validity, utility (Sait) Revise, as necessary, following psychometric analysis (David)

3. Let’s talk about the networks in our lives Maurice Taylor & Ivana Pavic

4. Talking about the social networks in our lives is very much related to a term called SOCIAL CAPITAL

5. Social capital relates to the connections we have with people …..connections to family …..connections to friends …..connections to the workplace …..connections to the neighbourhood …..connections to the local community

6. Why are these connections important? 1) they help us advance our interests in the society we live in 2) they help us co-operate within groups 3) they promote trust, goodwill and a desire to act in a supportive manner

7. Some examples The Ottawa East Community Association Immaculata Parent Council The Art of Running Club The Ottawa Newcomers Club Sandy Hill Community Health Centre Volunteer One Parent Families Association Jamaican Ottawa Community Association

8. What are the benefits of social capital? Glue- holds society together by facilitating co- operation within groups of people Oil- keeps the economic and social wheels of society rolling smoothly

9. Social capital is like: a resource that people can accumulate and use to their benefit and advantage having a bank account of all of the networks in our lives

10. The lack of social capital also can hold back people preventing them from achieving their goals which can affect the quality of life

11. Social capital is an important part of the learning process for 1) individuals 2) the training outcomes of a program 3) the community that we live in

12. How do we know if we’ve got social capital? How do we measure it?

13. Conceptual development of the tool How robust is the concept of social capital? OECD: 2000-2012 – framework for social outcomes of learning Australian Bureau of Statistics (2004) – indicators for a social capital framework A concerted effort to empirically understand adult literacy and social capital outcomes in Scotland, the UK, Australia and New Zealand

14. Conceptual development Interest on Canadian soil Canadian Institute for Policy Research Initiatives; National Centre for Literacy Is the timing right for developing a measure Do you have access to a target population?

15. Conceptual development 1 st Subscale: Network Qualities (NQ): including sub-concepts of trust levels, efficacy, diversity and inclusiveness 2 nd Subscale: Network Structure (NS): including sub-concepts of size, communication mode and power relationships 3 rd Subscale: Network Transactions (NTS): including sub-concepts of sharing support and sharing knowledge 4 th Subscale: Network Types (NTY): including sub-concepts of bonding, bridging, and linking

16. What are the size of your networks? How do you communicate in your networks? How do you share support? How do you share knowledge? What are your levels of trust? What is your level of self- confidence? How do you bond, bridge, and link with your networks? 4. Network Types 1. Network Qualities 2. Network Structures 3. Network Transactions

17. Lets talk about the reliability, validity, and utility of the Social Capital Inventory

18. Sample Items from SCI Sub- scales Item Strongly Disagree Disagree Neither agree nor disagree Agree Strongly Agree MSD 1.NQ I have the self-confidence to influence things in my work and community life. 23.645.13 2. NS When I am in a problem situation I have a network consisting of family, friends, and neighbours. 22.854.84 3. NTr I believe I am a valued member of my group because of the support that I give in class. 23.105.19 4. NTy I believe the program has provided opportunities for me to interact with new outside groups and services. 22.975.03

19. Reliability …refers to the stability or consistency of scores (over time, across raters, across items). When a test is reliable, it provides dependable and consistent results.

20. Methods for Estimating Reliability Involves administering the same test to the same group of participants on two different occasions and then correlating the two sets of scores. Test-Retest Reliability Involves two raters independently scoring the same set of tests and then correlating the two rater’s sets of scores. Inter-rater Reliability Involves administering the test once to a single group of participants and determining if items on the test are correlated with one another (as indicated by Cronbach's alpha). Internal Consistency Reliability

21. Let’s have a look at the Social Capital Inventory’s reliability To test SCI ’ s reliability internal consistency reliability was determined. *The SCI was found to have adequate reliability. SubscalesNumber of items Cronbach's alpha ( α) 1. Network Qualities 6.86 2. Network Structures 6.81 3. Network Transactions 6.88 4. Network Types 6.88 Total Scale24.96

22. Is reliability good enough for a test to be used A reliability coefficient only indicates whether the attribute measured by the test— whatever it is—is being assessed in a consistent way. Whether the test is actually assessing what it was designed to measure is addressed by an analysis of the test's validity.

23. Validity The term validity refers to whether or not the test measures what it claims to measure. On a test with high validity the items will be closely linked to the test’s intended focus.

24. SCI Validity Three major types: Content validity Construct validity Criterion- related validity Involves systematic examination of the test content to determine whether it covers a representative sample of the domain to be measured Typically conducted by panel of subject matter experts

25. SCI Validity Three major types: Criterion- related validity Content validity Construct validity Involves demonstrating that a measurement tool varies systematically with another measure of the same construct But, requires the existence of another valid measure…

26. SCI Validity Three major types: Construct validity Criterion- related validity Content validity Involves empirical testing of hypotheses about the construct, using the test as a measure of the construct e.g., if we know that one group should have higher levels of construct X than another group, then comparison of the mean level of construct X (as measured by the test) for the two groups should reveal a significant difference

27. Ongoing research -N=115 adult learners seeking literacy and essential skills in formal, non-formal, and informal learning settings -1 Adult high school (Ontario) -6 Workplace programs (4 in Manitoba, 2 in Nova Scotia -Mixed methods study (SCI, SDLRS, interviews)

28. Ongoing research - results -Relatively high SCI means are consistent with qualitative data obtained from instructors and learners -Those with average to above average scores on the SCI have significantly higher scores on the SDLRS -Disproportionately higher percentage of females than males had average to above average scores on the SCI, if they do not belong to any clubs

29. Take-home Tips 1. Start with theory -Read, read, read!

30. Take-home Tips 2. Seek feedback from Subject Matter Experts -Teachers -Colleagues -Committee members?!

31. Take-home Tips 3. Pilot test -Small, representative sample -Use multiple raters (if applicable) -Evaluate descriptive statistics (frequency distributions, means, SDs) -Evaluate reliability (Cronbach alpha, item- total correlations, inter-rater agreement)

32. Take-home Tips 4. Revise, if necessary -Delete unreliable items -Clarify wording -Modify scale -Consider difficulty/discriminability

33. Take-home Tips 5. Include validity checks -Collect demographic data -Include other measures related to construct -Compare with prior studies/theory

34. To Summarize: Read Listen to your committee Pilot test Revise Include validity checks