Academic Literacy Levels & Implications for Academic Progression and Curriculum Responsiveness Some findings from NBTP testing 2010 Alan Cliff & Nan Yeld

What do the NBTs aim to do? – Provide additional information about performance in core, underlying areas (additional to NSC information) – The core (domain) areas are: Academic literacy one 3-hr test Quantitative literacy Mathematics one 3-hr test

Why did HESA commission the NBTP? 1.Demonstrable inefficiencies in Higher Education itself (low throughput etc.) 2.Concerns about how to interpret the NSC in the early years while the new curriculum is being phased in / modified 3.Expressed need at the time for provision of additional information (to the NSC) to assist with placement of students in appropriate curricular routes, & for curriculum development. Difficulties in identifying students’ educational needs Lack of appropriate curriculum flexibility at entry to meet these needs (& seemingly little recognition of the need for this)

Core questions for the NSC and the NBTP THE NSC In respect of each learning area: To what extent do NSC candidates meet the Curriculum Statement expectations as expressed in the Subject Assessment Guidelines? THE NBTP In respect of each domain: To what extent do students aiming to enter higher education meet the core academic literacy, quantitative literacy and mathematics competencies required by school-leavers on entry to higher education study?

Differences between what the NSC and NBTP aim to assess: the issue of ‘transfer’ of knowledge and skill With the notable exception of Mathematics and Physical Science, school-level content knowledge is seldom stated as an important requisite for higher level study (cf admissions requirements of SA’s universities) It follows, therefore, that is not only the content that is taught in the National Curriculum Statement (the school curriculum) that is essential Rather, it is the cognitive abilities and skills underlying the school curriculum content domains that are learned in one context and are believed to be transferable to another that is important The NBTs therefore focus on the application of knowledge used in one context, to another

100% Performance in domain areas suggests that academic performance will not be adversely affected. If admitted, students should be placed on regular programmes of study. AL (65%); QL (66%) & MATHS (62%) Proficient Intermediate Challenges in domain areas are identified such that it is predicted that academic progress will be affected. If admitted, students’ educational needs should be met in a way deemed appropriate by the institution (e.g. extended or augmented programmes, special skills provision). AL (42%); QL (38%) & MATHS (34%) Basic Serious learning challenges identified: it is predicted that students will have difficulties in coping with degree level study without extensive & long-term support, perhaps best provided through bridging programmes or FET colleges. Institutions registering students performing at this level would need to provide such support. 0%

NBT information INDIVIDUAL LEVEL Performance (as a %) in each domain (AL, QL and Maths) Benchmark level (Basic, Intermediate, Proficient) for each domain Description of what this means for each domain (ie what does being in the ‘Basic’ category mean a student knows and can do in Mathematics) Clear recommendations about the type and extent of support needed GROUP LEVEL At the level of a faculty, or qualification, or institution …. Give clear indication of the needs and strengths of entering cohorts, either before entry, or at registration: useful for placement into existing courses, and/or with course design or modification.

Information from the NBTs Information for placement Information for curriculum design

INFORMATION FOR PLACEMENT (how will students do & therefore where should they be placed?) Academic progress of students in a university Mathematics course NBT benchmark category NBT domain MathematicsAcademic LiteracyQuantitative Literacy Basic 0 % [n=9] 32 % (n=7) 29 % (n=19) Intermediate 51% (n=198) 51% (n=105) 55% (n=123) Proficient 97 % [n=118] 77 % (n=222) 75 % (n=192) Note: all these students obtained at least 60% on the NSC Maths examination None of the students who fell into the ‘Basic’ category in the NBT Mathematics test passed the university Mathematics course. 97% pass-rate of students who achieved at the ‘Proficient’ level. Half of the students in the ‘Intermediate’ category passed: testament perhaps to the sustained efforts of the Mathematics department who provided extra classes and workshops once the first course tests in March revealed the extent of the students’ under-preparedness.

INFORMATION FOR CURICULUM DESIGN: W hat information can the NBTs give university lecturers about what students know and can do, and therefore what courses should address? Percentage of each benchmark category that correctly answered items NBT Mathematics CriteriaOverallBasic Low Intermediate High Intermediate Proficient Algebraic manipulation: includes inequalities, simplification, factorization, changing the subject of a formula 43%22%39%52%72% Perimeter, area, volume (modelling), relationship square/circle/area 31%19%22%35%70% Applications of principles of differential calculus to rates of change and related problems involving simple, linear and non-linear functions: point of inflection 41%19%34%51%72% Characteristics and interpretations of trigonometric functions and their graphs (such as domain, range, period, amplitude), including transformations of trigonometric functions: range, shifted and compressed sin function 38%27%31%39%69% Applications of principles of differential calculus to rates of change and related problems involving simple, linear and non-linear functions: derivative at a point on a cubic curve 56%37%48%63%87% Perimeter, area, volume (modelling): cube and sphere: circle and tangent 42%32%37%46%59% The table should be read as follows: “in the area of algebraic manipulation, 72% of the students falling into the ‘Proficient’ category could correctly answer the items, whereas only 22% of the students in the ‘Basic’ category could do so”.

Percentage of each benchmark category that correctly answered items EssentialInferencingDiscourseVocabularyMetaphorCohesion Commun FunctionGenreGrammar Proficient 70%75%87%79%76% 84%73%75% Top Intermediate 51%60%67%65%58%64%68%57%58% Bottom Intermediate 40%58%56%60%48%55%59%47%45% Basic 31%35%37%40%32%42% 32%38% INFORMATION FOR CURICULUM DESIGN: W hat information can the NBTs give university lecturers about what students know and can do, and therefore what courses should address?

Test Domains Academic literacy: The extent to which entry-level students can cope with typical reading demands in the medium-of-instruction of the institution.

The construct of Academic Literacy Skill AssessedExplanation of Skill Area VocabularyStudents’ abilities to derive/work out word meanings from their context Metaphorical ExpressionStudents’ abilities to understand and work with metaphor in language. This includes their capacity to perceive language connotation, word play, ambiguity, idiomatic expressions, and so on Extrapolation, application and inferencingStudents’ capacities to draw conclusions and apply insights, either on the basis of what is stated in texts or is implied by these texts

The construct of Academic Literacy Skill AssessedExplanation of Skill Area Understanding the communicative function of sentences Students’ abilities to ‘see’ how parts of sentences / discourse define other parts; or are examples of ideas; or are supports for arguments; or attempts to persuade Understanding relations between parts of text Students’ capacities to ‘see’ the structure and organisation of discourse and argument, by paying attention – within and between paragraphs in text – to transitions in argument; superordinate and subordinate ideas; introductions and conclusions; logical development Understanding the grammatical / syntactical basis of language Students’ abilities to ‘see’ / analyse the way in which sentence structure / word, phrase order affects meaning and emphasis in language

The construct of Academic Literacy Skill AssessedExplanation of Skill Area Understanding text genreStudents’ abilities to perceive ‘audience’ in text and purpose in writing, including an ability to understand text register (formality / informality) and tone (didactic / informative / persuasive / etc.) Separating the essential from the non- essential Students’ capacities to ‘see’ main ideas and supporting detail; statements and examples; facts and opinions; propositions and their arguments; being able to classify, categorise and ‘label’ Understanding information presented visually Students’ abilities to understand graphs, tables, diagrams, pictures, maps, flow- charts

The construct of Academic Literacy Skill AssessedExplanation of Skill Area Understanding basic numerical conceptsStudents’ abilities to make numerical estimations; comparisons; calculate percentages and fractions; make chronological references and sequence events / processes; do basic computations

AL Testlet ‘A’ and ‘B’ Specifications Reproducing orientationTransformative orientation Level 1 Knowing Level 2 Applying routine procedures in familiar contexts Level 3 Applying multi-step procedures in a variety of contexts Level 4 Reasoning and reflecting VocabularyA26; B22; B26 A6; A11; B46 B48B25 Metaphorical expressionA25; A35A7; A49; A50; A51; A52; B27; B38; B49; B50; B51; B52 A37A9; A23; A30 B8; B9 Extrapolation, application and inferencing B35A14; A38; B40 A22; B39B19A19; B12; B15; B17; B36; B42; B47 A16; A21; A29; A33; A42; A47; B16; B21; B31

Implications for teaching and learning? Approximately 32% of writers in need of some form of support False positives / false negatives What do the elements of the construct imply in discipline-specific contexts? What do we expect students to be able to know and do? How does one deal with students who are strong in some areas and weak in others? Other implications...