Presentation is loading. Please wait.

Presentation is loading. Please wait.

Basic Data Collection and Analysis TAU Breakfast Session 14 April 2011 Fia van Rensburg 1.

Similar presentations


Presentation on theme: "Basic Data Collection and Analysis TAU Breakfast Session 14 April 2011 Fia van Rensburg 1."— Presentation transcript:

1 Basic Data Collection and Analysis TAU Breakfast Session 14 April 2011 Fia van Rensburg 1

2 Purpose To highlight aspects with regard to monitoring, data collection and data analysis that are relevant to TAU on an operational level To build capacity of support staff within TAU to perform basic monitoring, data collection and data analysis functions To encourage TAU support staff to request assistance form the KM unit to advise on monitoring, data collection and data analysis 2

3 What this presentation is not about This presentation does not give detailed information on RBM It is not a technical lecture on research methodology and statistics 3

4 What is happening here? 4

5 What is happening in the picture? We can see different things in this picture: how many children; how many girls and boys; colours of their clothes; colours of the sacks; one fell; how many have long hair, short hair; they are indoors etc. This means that in any situation there is a lot of potential data that can be collected, but we are not going to collect all – we need to focus on what it is that we need, what is important to monitor. We need to decide up front what it is that we need to know in order to measure performance and inform decisions. The caption of the picture was: “5 out of 6 children like sack-racing” This statement can be questioned, because it is based on an assumption that the child that fell does not like sack racing. This may not be true. Therefore, we need to think very carefully of our interpretation of the data. LEARNING POINT: Data collection can be simple, but be clear on what is data and what is interpretation. 5

6 Why do we collect data? To get feedback on how we are doing in relation to the results we want to achieve (e.g. performance monitoring) To know where we can improve our services (e.g. customer satisfaction) To find out how and where we can work more cost-effectively (e.g. cost-benefit analysis) To learn from our experiences and to inform our future planning (e.g. programme evaluation) What is it that we don’t we know, that if we know it, will make a difference if we know it? 6

7 TAU Context Building capacity of our clients to deliver services to citizens How is capacity defined? (individual, organisational, environmental) Focus on capacity in: OD, PPM, Strategic Planning, SCM and Finance Management etc. Capacity is built through process consulting and sometimes workshops or capacity building sessions What is our theory of change? (strategy, and logic model linking inputs>outputs>immediate, intermediate, ultimate outcomes – Results Based Management) 7

8 Let’s pause for a moment 8

9 Background on Results-Based Management Before we look at data management on capacity building, we need to take note of how Results Based Management (RBM) works. The next four slides (with recognition to Peter Brook – TAU) gives an overview of: –basic terminology; –the results chain; –levels of measurement on the logic model; –indicator definition; and –baselines, targets and measures 9

10 10 Activities, Outputs and Outcomes are the building blocks of Results Based Management Describes a collection of functions (actions, jobs, tasks) that consume inputs/resources required to produce outputs The direct products and services generated through processes or activities. The effects, benefits or consequences that occur (either in the short, intermediate, or long-term) due to the produced outputs Every programme undertakes activities that produce outputs that contribute to the achievement of outcomes.

11 11 How do activities, outputs and outcomes link? Single Results Chain Results Hierarchy Outcome Output Activity Input

12 12 FINAL OUTCOME INTERMEDIATE OUTCOMES OUTPUTS ACTIVITIES All levels of the Logic Model should be measured INPUTS IMMEDIATE OUTCOMES Indicators

13 Indicator measurement has four different elements Indicator definition Baselines Targets Measures The first measurement of an indicator For comparing later measurements with For setting targets against The score we would like to achieve Must be time-bound, challenging and achievable Actual score for each measurement Must be measured regularly, so that trends can be established Identifies what should be measured, over what time period, and in what units, by who, how frequently, but does not contain a score or a target

14 Baselines, Targets and Measure are all expressed in the same units

15 So…now that we have refreshed our memories on RBM…what is the Programme logic/Theory of Change related to Capacity Development in TAU? 15

16 Chain of events/levels of evidence Programme chain of events (Theory of Action) Matching levels of evidence End Results (Outcomes) 7Measures of impact on overall problem, ultimate goals, side effects, social and economic consequences Practice and behaviour change 6Measures of adoption of new practices and behaviour over time Knowledge, skill and attitude changes 5Measures of individual and group changes in knowledge, skills, attitudes and learning Reactions4What participants and clients say about the programme, satisfaction, interest, strengths, weaknesses Participation3Characteristics of Programme participants and clients, numbers, nature of involvement, background Activities2Implementation data on what the programme actually offers or does Inputs1Resources expended, number and types of staff involved, time extended ( Adapted from Patton, 1997: p 235) 16

17 How do we know we are successful with capacity building? 17 Ultimate: Behaviour change – actual application and change in systems, practice – CSS, MTR, Evaluation Intermediate: Knowledge skills and attitude Immediate: Reaction to presentation content, delivery and format Outcome Indicators will relate to REACTION; KNOWLEDGE, SKILLS AND ATTITUDE; and BEHAVIOUR CHANGE Number of workshops/capacity building sessions held Number of participants attending Sphere(s) of government covered Number of departments provinces attending Level of participants Output Indicators will relate to PARTICIPATION

18 How do we measure the indicators? IndicatorDataData source Number of workshops held Simple count, dates, topics and themes covered Attendance registers and feedback forms Number of participants attending Simple count, disaggregated i.t.o. sex, sphere of government, department/province, level/ designation of participants Attendance registers and feedback forms Reaction to presentation Participants’ rating of content, relevance delivery and format, overall satisfaction Feedback forms Extent to which knowledge skills and attitudes have improved Participants’ rating of improvement in knowledge, skills and personal effectiveness Comparative post-intervention assessment against baseline/ stated project outcomes Feedback forms - also Project Closure Reports, CSS, MTR Behaviour change over time Self-reported or others’ reports/observation providing evidence of change in relevant behaviour (comparison pre-and post), level of implementation of proposals, systems, practice in relation to stated objectives Feedback forms (related to a series of capacity building initiatives implemented over time), SS, MTR, Evaluation 18

19 How do we collect the data? Hard copy/electronic questionnaire Telephonic, face-to-face or focus group interviews When and how to administer questionnaires/use interview guides Questionnaires: –How many questions? (Not too many, but cover the indicators) –Open-ended or fixed response categories –Options for fixed responses - even or uneven number, ranking scales, naming –The rhythm of the questionnaire 19

20 How do we capture and analyse and present the data ? Quantitative data Use MS Excel Draw up a framework and decide on codification (remember o response, not applicable) Descriptive statistics Graphs Qualitative data Capture in exact wording Thematic analysis Summary 20

21 …and we need to pause again… to consider a few technicalities… 21

22 What level of measurement/types of data are we working with? Nominal: Lowest, least precise level of measurement. Different types of categories of a variable (e.g. colour: red or blue or yellow/sex: male or female/religion: protestant/catholic). Categories are mutually exclusive, not ordered. Ordinal: Identifies difference among categories available, and allows categories to be rank ordered (e.g.: level of education: primary, secondary, tertiary) Interval: Categories are ranked, and distance between categories is measured. No true zero (e.g. IQ) Ratio: Most precise level, rank ordered, distance is precisely measured, and there is an absolute zero (e.g. age). 22

23 Please note - the level of measurement determines what calculations we can do with the data… The mode is the most frequent score. Can be use for all levels of measurement. (Most The median is the middle point – half of cases are above this point and the other half below. Can be used for ordinal, interval and ratio data. The mean is the average. Can be used only when we have interval and ratio data. 23

24 What you can and cannot do… 24 OK TO COMPUTE.... N OMINAL O RDINAL I NTERVAL R ATIO frequency distribution Yes median and percentiles NoYes add or subtract No Yes mean, standard deviation, standard error of the mean No Yes ratio, or coefficient of variation No Yes

25 How can we present the data analysis? The most common ways of presenting analysed data are: –Pie charts –Bar graphs –Stacked bar graphs –Line charts 25

26 Pie Charts Circles divided into a number of slices, each slice representing the relative proportion data points falling into a given category. Suitable for graphing nominal data and percentages. 26

27 Pie Charts Example: Attendance of TAU Breakfast Session 9 September

28 Bar Graphs Used to show umber of proportion of nominal or ordinal data according to a specific attribute (nominal data). Often shows # or % of observations per data points Switching rows and columns gives different perspectives (see next slide) –it is important to decide if you want to focus on the project phase or what the picture is per portfolio (see example in next slide) 28

29 Bar Graphs Projects per Project Phase (fictional data) 29 Project phase per Portfolio (fictional data)

30 Stacked Bar Graphs Can be used for representing multiple categories of data in a single bar, and each bar is made up of sections that represents the proportion of data falling into a category. Can be used to show comparative contributions of different groups or ratings to a total. Use the switch rows and columns function to see which display makes sense – in the example in the next slide Participant Feedback (B) may be better to use. 30

31 Stacked Bar Graphs Participant Feedback (A) 31 Participant Feedback (B)

32 Line Graphs A single line is used to connect data points (interval and nominal data) Used to present progress over time and trends More than one variable can be presented on a single chart – but check that it works. See examples on next slide: The presentation of # of website visits, together with page views per visit and # of seconds per page view does not work. The presentation of Male/Female breakdown over time works well. 32

33 Line Graphs A single line is used to connect data points (interval and nominal data) Used to present progress over time and trends 33

34 Line Chart Website report Staffing Table - Gender Breakdown per Quarter 34

35 Qualitative Data Qualitative data in questionnaires is mostly gathered through open-ended questions In the next slide a basic example is given of how qualitative data can be analysed: (1) Capture the text; (2) Group related themes/comments together; (3) count; (4) analyse; and (5) present the analysis in an appropriate format. NB Qualitative date provides more in-depth information, but if applicable, it can be counted, as in the example on the next slide 35

36 Qualitative data Future PMIG sessions should focus on…. Verbatim The project charter How to set up a project office Stakeholder management Communication in projects How to manage a Project office Project Office Analysis Three priorities were identified for future PMIG sessions: Project office set-up and management (50%) Communication and stakeholder management (33.3%) Project Charter (16.7%) 36

37 You can do it It is not as difficult as we sometimes think to analyse data, and to represent it in graphs There are, however a few points of caution and if you are not sure about something, rather consult a colleague that has a deeper understanding of data analysis. 37

38 It’s easy? 38

39 Really? Our potential President of the Math Club does not really qualify for this position – he has seriously under-estimated his support. Why? …see the next slide 39

40 Are you sure? TotalSupportPercentage Boys % Girls % Total /200= 26% Boys % Girls % Total /400=23% Boys % Girls % Total /300=27% 40

41 What happened here? This is a very common mistake Percentages cannot be added up as percentages A percentage represents a proportion of a universe, and the size of the universe needs to be taken into account. To calculate his support base correctly, our clever friend had calculate the total number of people that supported him in relation to the total number of boys an girls. LEARNING POINT: We need to know what calculations we can do with certain types of data. Always double-check. (see notes on types of data) 41

42 Thank You References:


Download ppt "Basic Data Collection and Analysis TAU Breakfast Session 14 April 2011 Fia van Rensburg 1."

Similar presentations


Ads by Google