1. Formative Evaluation of Workbooks and Textbooks Version 13 March 2013

2. Formative evaluation of textbook and workbooks, 2012 The evaluation of the DBE Workbook and Textbook initiative is formative. It aims to contribute towards the review and improvement of the effectiveness (including quality) and utilisation of workbooks and textbooks in schools in South Africa; The key research questions addressed by the evaluation are: – How are the textbooks and workbooks being utilised? – What is happening in schools where the workbooks and textbooks are not being utilised? – Do the workbooks and textbooks have the characteristics of quality text/workbooks? DBE workbooks & textbooks – Grade 3: Mathematics – Terms 1 and 2, Home Language – Terms 3 and 4, English as a First Additional Language – Terms 1 and 2 – Grade 6: Mathematics – Terms 1 and 2, Home Language – Terms 1 and 2, English as a First Additional Language – Terms 1 and 2 – Grade 9: Mathematics – Terms 1 and 2 – Grade 10 Mathematics and Physical Science Siyavula textbooks

3. Formative evaluation of textbook and workbooks, 2012 National sample survey (Learner and educator questionnaires and classroom observation) 324 schools, 969 teachers, 455 learners, 335 HoDs, 269 SGB members Three schools in case studies to provide insights into the perceptions held by stakeholders (parents, SGB member, student groups) of the performance and utilisation of workbooks. The case studies used focus group discussions and in-depth interviews to gather data. Desk review evaluated the textbooks and workbooks using criteria, that were developed drawing on information from a literature review on the characteristics of quality workbooks and textbooks. All three components of the study focused on Grades 3, 6 and 9 for the DBE workbook evaluation. Data related to the Grade 10 textbooks came from classroom observations and the desk review.

4. Formative evaluation of textbook and workbooks, 2012 A Quality assessment rubric was developed (from literature review) to assess the quality of DBE workbooks and Siyavula textbooks An observation tool was used to observe and report on utilisation aspects including presence, storage and use of DBE workbooks, Siyavula books as well as other books. – Intention of study is not to study pedagogic practice but usage & quality The tool also collected subject specific data regarding the curriculum coverage of the workbooks under observation. The depth and volume of curriculum coverage was also observed and reported. The tool was Grade and Subject specific, best learner and learner needing more support were used to asses curriculum coverage and scope of learning and any changes in progress since the introduction of workbooks.

5. Sampling and weightings Sampling Oversampling of schools from smaller provinces was undertaken and an equal number of schools was selected from each province. Fair representation of the country - Schools were sampled with probability proportional to size from a list stratified by province, wealth quintile, and school type. From each sample school five teachers were randomly selected from a list of all teachers from the school who teach at the focus year levels (3, 6 and 9). Weighting When designing the sample for the SA textbook evaluation study, it was agreed to approach the same number of schools (36) from each province so that enough data were collected from each province to allow for some provincial comparisons as part of the analysis.

6. Findings of the evaluation of DBE workbooks and Siyavula textbooks

7. Better pacing, coverage, assessment and remediation Teacher reports and observations of DBE workbook use indicate that, – Typically, topics are being covered in sequence for better learners, poorer learners and the classes as a whole, four standard exercises are being covered per week by classes, and teachers are assessing and correcting the work of both the better and the poorer learners in classrooms. A far cry from situation projected by JET in 2009. – On average only 24% topics covered in Grades 4 and 5 – 88% teachers covered no more than 40% of 89 topics in Grade 5 curriculum – 58% had covered no more than 20 topics in Grade 4 making only 22% of curriculum covered.

8. Research findings How many schools are using the DBE workbooks? In around 80 per cent of schools, teachers were using the DBE workbooks. How are DBE workbooks used in classrooms? Generally, at all three case study schools, teachers and school leaders saw the workbooks as an effective tool in teaching and learning. However, most indicated that they use the DBE workbooks to supplement the curriculum and that the workbooks are perceived as an ‘add on’ to the textbooks that were being used prior to the introduction of the workbooks.

9. Research findings 2 Lesson planning using DBE workbooks At the three case study schools teachers and school managers indicated that formalised workbook planning rarely took place and that teachers tend to go through the DBE workbooks before going to class or preselect the exercises learners will be asked to complete in class. This is a generalised indication of lesson planning in the system How often are DBE workbooks used in class? According to teachers, on average the DBE workbooks were used 3.6 times in the previous week. The majority (70%) of teachers used the workbooks 5 times (on average daily) 1 percent of teachers used the workbooks more than 5 times in the previous week

10. Barriers for workbook utilisation Delivery The greatest reason cited by classroom teachers as to why the books are not being used in classrooms is no delivery. Within a number of provinces the sharing of workbooks was reported more frequently; however this did not appear to impact on utilisation Communication Qualitative data from case study interviews and focus groups indicate that schools, teachers and parents were not notified of the program or informed about how to use the workbooks. Teachers and school leaders also indicated that prior consultation, introduction and involvement from schools and teachers could improve the quality of teaching and lead to better learner outcomes.

11. Barriers for workbook utilisation 2 Lack of formal assessment and monitoring of learner performance In teacher interviews and focus groups teachers expressed difficulty with monitoring learner performance through the workbooks. Teachers perceived the workbooks as allowing for informal assessment, but found the workbooks were not able to support the recording and tracking of learner performance over time. Errors and perceived errors in the workbooks Assessments of the quality of workbooks indicate there is a need for further editing. In interviews and focus groups teachers also expressed concern with a number of errors within the workbooks.

12. Research questions and quality rubric

13. Research questions and quality rubric 2

14. Teachers: Frequency of use of learning materials in class including DBE workbooks

15. Curriculum coverage and pacing Teachers report that the DBE workbooks provide wide coverage of the curriculum %generally that the DBE workbooks cater to a diversity in the classroom 3 in 4 teachers indicate that the pace set for the completion of the work is about right for most students. Learners also coped with the pace and level of work Topics are typically being covered in sequence in classrooms where the DBE workbooks are used. HODs and learners also reported that the sequencing of topics within the workbooks allows for knowledge and skills development. Teacher and learner responses and classroom observations indicate that Volume 1 DBE workbooks are completed before Volume 2 more than half of the time. Between-book differences were not observed, with the exception of Grade 3 Home Language, where Volume 1 was completed before Volume 2 more often than some other workbooks

16. Teachers indication of DBE workbook ease of use

17. Over 60% of learners agree that DBE workbooks address basics before progressing to more difficult tasks

18. Classrooms observations: evidence that the teacher is covering topics in sequence

19. Teachers sequence of topics covered by quintile

20. Sequence, Pace and Corrections, Grade 3 Mathematics

21. Sequence, Pace and Corrections, Grade 6 Mathematics

22. Sequence, Pace and Corrections, Grade 9 Mathematics

23. Correcting Work Within Workbooks Classroom observations and responses from teachers and School Governing Body representatives provide evidence that, typically, teachers are assessing and correcting work within the workbooks. However, qualitative data collected in teacher interviews and focus groups indicate that while some teachers are assessing and correcting work within the workbooks, they may not be monitoring learner progress over time.

24. Classroom Observations: evidence that DBE workbooks facilitate correcting/assessing learners’ work…SGB and learner responses confirmed this

25. Schools using DBE workbooks by province The evidence suggests that there are few provincial differences in the number of teachers using DBE workbooks in class. Learners in Western Cape, KwaZulu Natal, Gauteng and Eastern Cape were more likely to have been using workbooks since the beginning of the year as compared with Limpopo and Mpumalanga.

26. National Teacher Usage of DBE workbooks by Qualification

27. National Teacher Usage of DBE workbooks by Quintile

28. Teachers opinions on the cultural and gender representivity in the DBE workbooks

29. Support needed in the use of DBE workbooks

30. Schools Not Using DBE Workbooks School Governing Board representative and teacher responses were cross checked. There were only 19 schools in the sample of 327 which were reported to have no teachers using any workbooks. Therefore, there were very few schools available to the study to investigate what is happening in those schools without workbooks. A list of books being used in these schools, and the curriculum being used is listed in the report for indicative purposes only.

31. Learner use of DBE workbooks The available evidence suggests that: The vast majority of learners take their DBE workbooks home most days Learners complete work within their workbooks for homework between 2.5 – 4 times per week. Approximately 65 – 70 per cent of learners are assisted by parents with completing work in the workbook for homework. Approximately three-quarters of the learners report being able to take some of their workbooks home Almost the same proportion of learners indicate being able to take all of their workbooks home. Almost 90 per cent of both teachers and HODs report that learners take the workbooks home

32. Learners’ responses to how often they used their DBE workbooks for doing home work in the last week

33. Learners indication of using DBE workbooks at home for revision or homework

34. National teachers storage of workbooks

35. National learner storage of workbooks

36. National teachers storage of workbooks 2

37. Parental assistance According to all stakeholders, approximately 65 – 70 per cent of parents assist learners with workbook completion. In-depth data from parent responses in focus groups indicate that while some parents help their children with homework, others struggle with the written language due to their own level of education. In these instances other family members, neighbors or the parents employers assist. However, even parents who could not assist their children felt the workbooks help them understand the curriculum and give them the opportunity to participate in their child’s education.

38. Parental assistance 2 One parent commented, ‘It helps us parents, it teaches us about what the children are learning in school’. Parents indicated that they use the workbooks to learn and teach themselves. Should the workbook program undergo an impact evaluation, any unintended outcomes on parental literacy and numeracy could be explored. Parents and learners also indicated that the workbooks reduce the amount of parental assistance required as they enable learners to work independently, or with friends.

39. Quality assessment of workbooks and textbooks

40. Quality of workbooks Do the books have the characteristics of quality textbooks and workbooks, as defined by national and international standards? A quality rubric was developed, designed to elicit the judgments of assessors about various facets of the textbooks and workbooks. Assessors were asked to assess each characteristic using the categories: ‘not at all’;’ to a minor extent’; ‘to a moderate extent’; or ‘ to a major extent’.

41. Quality of workbook: Assessment Assessors assessing the workbooks tended to agree that the workbook assessment tasks are fit for purpose. That is, each section of the book is clearly related to its learning goals, each assessment task is clearly aligned to the goals of the lesson and assessment tasks are age and grade appropriate. However, many assessors made this judgment with the caveat that this refers to the implicit learning goals within the workbooks. A number of assessors commented that the assessment tasks within the workbooks tended to be implicit, rather than explicit.

42. Quality of workbooks: Rubric

43. Quality of textbooks: Rubric

44. Quality of workbooks :Rubric Do the workbooks reflect the values of South Africa, as enshrined in the constitution?

45. Quality of textbooks: Rubric Do the workbooks and textbooks reflect the values of South Africa, as enshrined in the constitution?

46. Hypothesis Unqualified teachers use the workbooks more often than qualified teachers There is no evidence to suggest that unqualified teachers use the workbooks more often than qualified teachers. The higher the quality of workbooks, the better utilisation by teachers There is little evidence to suggest that the higher the quality of workbooks, the better utilisation by teachers. The number of quality characteristics assessed to be present ‘to a major extent’ in workbooks was used as the measure of quality.

47. Hypothesis 2 As poverty increases, so does workbook use. There is no evidence that the use of the workbooks is related to the wealth quintile of the school. The only statistically significant difference between wealth quintiles was that of quintile 5 and quintile 2, with a greater portion of quintile 5 schools taking up workbook use. This is likely to be related to workbook availability.

48. Conclusion Nearly all schools are using the workbooks. The evidence suggests that where the workbooks are available, they are quickly taken up by schools. While most teachers see the workbooks as effective tools and use them on a daily basis, there are some indications that they are not using them as part of their mainstream teaching activities.

49. Conclusion 2 Although shortages at the class level do not appear to reduce the frequency of workbook use, shortages may be limiting their full use. The workbooks and the textbooks were assessed to have the characteristics of quality workbooks. Assessors tended to observe the characteristics of quality textbooks linked to learning goals and assessment to a greater extent than in the workbooks. Barriers to full utilisation of the workbooks can be summarised as late or no workbook delivery, lack of communication with schools and parents, lack of formal assessment and monitoring, and errors or perceived errors in the workbooks.

50. Recommendations In terms of what schools can influence, the most important factor is the quality of the teacher in front of the classroom. It is recommended that professional training for teachers related to using the workbooks be undertaken. A professional training pilot program might incorporate and compare teacher perceptions of those completing training and those who had not in order to compare pedagogic differences. Provide training DVDs for teachers showcasing the ways in which the workbooks can be used creatively within classrooms possibly through teacher resource centres Develop a communication strategy to inform School Management, teachers, parents and learners about the workbook project, including the aims of the project

51. Recommendations 2 As part of the communication strategy, it is recommended that the aims and objectives of the workbook project be made explicit to schools and parents. For teachers, this might include explanations regarding the approach to language within the workbooks. Consider the addition of an information sheet for parents with each workbook. Making learning goals explicit within the workbooks. Provide templates to monitor learner progress in workbooks Ensure all workbooks undergo further editing. Provide more space within workbooks for learner responses.