Presentation is loading. Please wait.

Presentation is loading. Please wait.

Research into an Exemplary Programme The International Baccalaureate Diploma Programme Group 2 Languages Dr. John ISRAEL AND THE ASSESSMENT AND THE ASSESSMENT.

Similar presentations


Presentation on theme: "Research into an Exemplary Programme The International Baccalaureate Diploma Programme Group 2 Languages Dr. John ISRAEL AND THE ASSESSMENT AND THE ASSESSMENT."— Presentation transcript:

1

2 Research into an Exemplary Programme The International Baccalaureate Diploma Programme Group 2 Languages Dr. John ISRAEL AND THE ASSESSMENT AND THE ASSESSMENT OF MODERN LANGUAGE LEARNING

3 Based in an IB evaluation philosophy of : commonality of programme levels, tasks and criteria for assessment and evaluations for any and all languages differentiation by programme, level, task and criterion descriptors (A2, B, ab initio, higher, standard), with no entry criteria authenticity as the goal in language reception and production minimal pedagogical intervention in editing authentically-sourced materials assessment and evaluation by descriptive criterion referencing as can do performance, not what is unknown or incorrect?

4 Performance assessment and evaluation: Assessment understood as qualitative : matching samples of performance to descriptions, by level and discrete criterion Evaluation understood as quantitative : assigning numerical value to assessments, as indicators of overall quality some key concepts

5 Assessment criteria for Production, Languages B These are categorised by :Language Cultural Interaction Message With each criterion valued equally, at a maximum of 33,33% (oral and written)

6 LANGUAGE Written Production Command of the language is good and effective. A range of grammar and vocabulary is used accurately despite some errors in more complex constructions. Some complex sentence structures are used clearly and effectively. Spelling/calligraphy is generally correct and clear. At least the prescribed minimum number of words has been written. Command of the language is very good and may show evidence of sophistication. A wide range of grammar and vocabulary is used accurately with few errors. Complex sentence structures are used effectively and skillfully. Spelling/calligraphy is almost always correct and clear. At least the prescribed minimum number of words has been written. B Standard Level : B Higher Level : To what extent does the candidate write the language fluently and accurately? How varied and accurate are the grammar and vocabulary used by the candidate? How clear are the sentence structures? To what extent is the candidate able to use complex structures? How accurate is the spelling or calligraphy? Has the candidate written the prescribed minimum number of words? General Criteria Skip

7 LANGUAGE Oral Production General Criteria B Standard Level : B Higher Level : Command of the spoken language is good. The production of language is fluent. The use of grammar and vocabulary is generally correct, varied and idiomatic. The intonation contributes effectively to communication. Command of the spoken language is very good. The production of language is fluent and with a touch of authenticity. The use of grammar and vocabulary is varied and idiomatic, almost error free. The intonation contributes effectively and expressively to communication. To what extent does the candidate speak the language fluently and accurately? How fluent is the language spoken by the candidate? (Fluency refers to ease of speaking.) How correct and idiomatic are the grammar and vocabulary used by the candidate? To what extent does intonation contribute to communication? (Intonation refers to the sounds and rhythms of the language that are essential for effective communication. It does not refer to accent. The candidate is not expected to sound like a native speaker of the language.) Skip

8 MESSAGE Written Production The message has been communicated well. The ideas are relevant. The development of ideas is methodical and thorough. Supporting details are appropriate. The organization of ideas is clear. The message has been communicated very well. The ideas are relevant and stimulating. The development of ideas is thorough and imaginative. Supporting details are appropriate and convincing. The organization of ideas is clear and flows well. B Standard Level : B Higher Level : General Criteria To what extent does the candidate communicate the message in a developed and organized manner? How relevant are the ideas presented by the candidate? How developed are the ideas? How appropriate are the supporting details? To what extent are the ideas organized into an overall plan? Skip

9 MESSAGE Oral Production General Criteria B Standard Level : B Higher Level : The candidate handles complex ideas well. Both simple and complex ideas and opinions are generally presented clearly, coherently and effectively. Responses are generally relevant and show some imagination. The conversation flows coherently. The candidate handles complex ideas very well. Both simple and complex ideas and opinions are presented clearly, coherently and vividly. Responses are relevant and show insight and imagination. A coherent conversation is maintained throughout. To what extent is the candidate able to communicate ideas (or message) and maintain a coherent conversation? To what extent is the candidate able to convey complex ideas and opinions? How clearly, coherently and effectively are the ideas and opinions presented? How relevant and complete are the candidates responses? How coherent is the conversation? Skip

10 CULTURAL INTERACTION Written Production The text is clear and convincing. The choice of register and style is generally effective and appropriate to the task. Rhetorical devices appropriate to the type of text are generally effective and varied. Structural elements contribute to the clarity of the text. The text is convincing and expressive, with some imagination. The choice of register and style is consistently effective and appropriate to the task. Rhetorical devices appropriate to the type of text are effective, varied and imaginative. Structural elements contribute fully to the clarity of the text. B Standard Level : B Higher Level : General Criteria To what extent does the candidate select language appropriate to the audience and type of text? How convincing and expressive is the text? How effective and appropriate is the choice of register and style to the task? Where appropriate, how varied and effective are the rhetorical devices? (Rhetorical devices include all techniques used to present the message more vividly, such as metaphor, exaggeration and repetition.) To what extent do structural elements contribute to the clarity of the text? (Structural elements include cohesive devices.) Skip

11 CULTURAL INTERACTION Written Production The text is clear and convincing. The choice of register and style is generally effective and appropriate to the task. Rhetorical devices appropriate to the type of text are generally effective and varied. Structural elements contribute to the clarity of the text. The text is convincing and expressive, with some imagination. The choice of register and style is consistently effective and appropriate to the task. Rhetorical devices appropriate to the type of text are effective, varied and imaginative. Structural elements contribute fully to the clarity of the text. B Standard Level : B Higher Level : General Criteria To what extent does the candidate select language appropriate to the audience and type of text? How convincing and expressive is the text? How effective and appropriate is the choice of register and style to the task? Where appropriate, how varied and effective are the rhetorical devices? (Rhetorical devices include all techniques used to present the message more vividly, such as metaphor, exaggeration and repetition.) To what extent do structural elements contribute to the clarity of the text? (Structural elements include cohesive devices.) Skip

12 CULTURAL INTERACTION Oral Production General Criteria B Standard Level : B Higher Level : Interaction in conversation is successful. Responses in fairly complex exchanges show some sensitivity to subtlety, nuances and prompts. Contributions to the conversation are active and spontaneous. The language is generally appropriate to the subject and context. Interaction in conversation is very successful. Responses in complex exchanges show sensitivity and subtlety to nuances and prompts. Contributions to the conversation are active, spontaneous and sensitive to others. The language is consistently appropriate to the subject and context. To what extent does the candidate interact appropriately and successfully in the conversation? How sensitive and subtle is the candidates response to nuances and prompts? (Teachers should ensure that their participation in the exchange is sufficient, subtle and complex enough to enable the assessment of the candidates listening skills.) How actively and sensitively does the candidate contribute to the conversation? To what extent does the candidate speak spontaneously, or has the candidate rehearsed the conversation? How appropriate is the language to the subject and context? (Language refers to tone and register.) Skip

13 Assessment method for samples of production By multiple, repeated moderation for reliability, as: Subjective interpretation, through matching to criteria by experienced assessors, themselves moderated for consistency. Consensal, through multiple moderations by different assessors. With statistically–determined moderation factors applied to the results of each assessor. Open to appeal and repeats of the assessment process.

14 Research Hypotheses As an investigation of IB programmes, assessment and evaluation practices, with particular reference to authenticity, understood in philosophical terms as well as purely linguistic, and as categorised for example, by Van Lier (1996), with focussing on Language B programmes for: Task Design and Assessment Criterion Task Design and Assessment CriterionValidity Assessment and Evaluation Reliability Assessment and Evaluation Reliability Credibility of Final Evaluations Credibility of Final Evaluations

15 Key research questions: In what ways and to what extent do the IBs language programmes promote authentic language production in target languages? Can authenticity in communicative and interactive language productions be a yardstick for measurements, with acceptable reliability? Can validity, reliability and credibility in IB assessment and evaluation practices be enhanced, without compromise to a programme philosophy of authenticity? If so, with what implications for the teaching, learning and measurement processes?

16 How is Authenticity to be understood? As essentially subjective consciousness in communicative interaction with other as object (in bad faith), or alternate subjectivity (in good faith), and/or self (through reflection), in (linguistically), culturally and socially definable contexts for specific purposes, including self-definition. The processes involved are dialectical. One example very briefly summarised from J-P. Sartre (1946)

17 Van Liers categorisations As a triad of:Awareness (The Cartesian consciousness of Sartre) Autonomy (The Sartrean conceptualisation of self and other) Authenticity (The unity of the above)

18 A triadic representation of interrelations, A. Exposure to language (including quality of language and the receptivity of the individual). AWARENESS 1. AWARENESS B. Perception of social and linguistic interaction (i.e. the relation between the individual and exposure). AUTONOMY 2. AUTONOMY C. Processing of language (i.e. the social and cognitive transformations that lead to conscious activities of interpretation and purposeful linguistic interaction). (See Van Lier (1996), op. cit.) 3. AUTHENTICITY between awareness, autonomy and authenticity from the perspective of language use,

19 Van Liers further categorisations This triad of:AwarenessAutonomyAuthenticity may be linked to a further triad of: Curricular authenticity Pragmatic authenticity Personal authenticity

20 Research criteria For measuring communicative value in integrated performances, interactively combining comprehension and production, as: Evidence for Curricular Authenticity This is evidence for the quality of an individuals creation and use of language, after exposure to the target environment. This major criterion is further categorised as: Creator authenticity, concerning self Creator authenticity, concerning self Creator authenticity, concerning the recognition of other Creator authenticity, concerning the recognition of other Finder authenticity Finder authenticity User authenticity User authenticity developed from Van Lier

21 Curricular authentication Evidence for Curricular Authenticity, measuring levels of (linguistic) awareness and autonomy Creator authenticity for assessing linguistic realisations of self: Creator authenticity for assessing linguistic realisations of self: focussing attention on evidence for the personal and unique voice of each producer of language. Creator authenticity, assessing perceptions of other Creator authenticity, assessing perceptions of other as interlocutor, audience or reader: as interlocutor, audience or reader: focussing attention on evidence for attempts to motivate participation in communicative interchange through personal strategies or discrete tactics for retaining listeners or readers attention. Finder authenticity, or the resourcefulness of communicators in finding material Finder authenticity, or the resourcefulness of communicators in finding material for communication: focussing attention on evidence for recognisable agency in the selection and manipulation of specific objects of awareness, sourced from outside self, or meta-cognitively within self. User authenticity, or recognitions of other as listener or reader: User authenticity, or recognitions of other as listener or reader: focussing attention on evidence for linguistic interaction through respect for commonly-acquired social traditions and communicative convention, facilitating initiations and continuations of communication: there is purposive response to set stimuli and to prompts sourced in the initiatives of other.

22 For measuring communicative value in integrated performances, interactively combining comprehension and production, as: Evidence for Pragmatic Authenticity This is evidence for appropriate, individual purposes in public language production, creating links with the physical, temporal and socio-cultural contexts within which linguistic interactions take place. This major criterion is further categorised as: Authenticity of purpose Authenticity of purpose Authenticity of context Authenticity of context Authenticity of interaction Authenticity of interaction Research criteria developed from Van Lier

23 Pragmatic authentication Evidence for Pragmatic Authenticity, measuring levels of (socio-cultural) awareness and autonomy Authenticity of context: OR the willingness by partners in communication to share culturally-situated perspectives, respecting the conventions and traditions of a collaboratively-modifiable culture. respecting the conventions and traditions of a collaboratively-modifiable culture. There should be evidence of agreements, explicit or implicit, interactively to share communication and so construct extensive and extendable social relations through language. with no suggestion of self-determined, one-sided closure of communication. Authenticity of purpose: Authenticity of purpose: OR OR self-awareness and transparency in choices of expressive form and the content to be communicated. There should be evidence of intentional facilitations of changes in perspective and / or knowledge amongst interlocutors, audiences, or readers of the text created, and reflexively in self, where relevant. Authenticity, of interaction between partners in communication: Authenticity, of interaction between partners in communication: OR recognitions of power in questions of validity, balance and convincingness, determining communicative quality in social relationships between speakers and listeners, writers and readers. There should be evidence for accommodations of self to other in processes of continuous change, and recognitions within self and in self as other, of ability to guide this development.

24 For measuring communicative value in integrated performances, interactively combining comprehension and production, as: Evidence for Personal Authenticity This subsequently emerges from processing language, creating ontological, or existential status for individuals who participate in linguistic interchanges, through intrinsically- motivated, purposeful choices extending over time. This major criterion is further categorised as: Existential authenticity Existential authenticity Intrinsic authenticity Intrinsic authenticity Autotelic authenticity Autotelic authenticity Research criteria developed from Van Lier (after Csikszentmihalyi, 1990)

25 Personal authentication Evidence for Personal Authenticity, measuring levels of (cognitive) awareness and autonomy Existential authenticity : OR OR social constructions and expressions of self through (communicative) actions. Focussing attention on evidence of an awareness of the uniqueness of personal voice, or negatively, on avoiding overt plagiarism, through its absence as evidence in any given production. Intrinsic authenticity : Intrinsic authenticity : OR OR recognitions of self-determination in continuous operations of choice. Focussing attention on evidence for metacognitively-aware, active and responsible selections of style and content in communicative performances. Autotelic authenticity : Autotelic authenticity : OR experiencing and expressing flow as optimal experience, relating linguistic coherence and psychological balance to the inner mental worlds of subjects. Focussing attention on evidence for concentrated awareness of, and commitment to communication as present activity, with intentions to satisfy personally-chosen goals, and without intrusive distractions or irrelevance. ( after Csikszentmihalyi, 1990) ContinueReminder

26 Based in an IB evaluation philosophy of : commonality of programme levels, tasks and criteria for assessment and evaluations for any and all languages differentiation by programme, level, task and criterion descriptors (A2, B, ab initio, higher, standard), with no entry criteria authenticity as the goal in language reception and production minimal pedagogical intervention in editing authentically-sourced materials assessment and evaluation by descriptive criterion referencing as can do performance, not what is unknown or incorrect?

27 Application to samples Method Van Liers concepts grouped into sets of descriptors, as with the IB referent model. No leeway within each level for further, subjective adjustments, as with the IB scheme. Three simple, discrete descriptions and quantifications for the provision of evidence: little (1 point), adequate (2 points), little (1 point), adequate (2 points), significant (3 points) With Van Liers ten categories, an unweighted, maximum aggregation gives thirty points. Two complementary levels later added for extremes: * one negative, for the complete absence of evidence (0 points) * one positive, for interpretatively-incontestable displays of competence (4 points). This provides clear indicators of inauthenticity. An excess at extremes should reveal inappropriate programme and level selection by candidates, with tasks either too easy or too difficult. Compensations permit adjustments, more precisely discriminating individual performances, but increasing the maximum aggregated total to forty points.

28 Some results: oral production

29 Some general analysis of results NOTE: NOTE: This graph was derived from samples of responses to a single, common task and rubric of individual productions for external IB moderation. It allows for comparisons across raters and rating systems, across the individual assessment of each production. Assessment by criteria derived from categorisations of authentic language production in communicative interactions can produce meaningful evaluations. Internal and external moderator scores are closely correlated. Assessment by researcher using the Van Lier derived model also correlates closely, with finer distinctions between individual productions. Assessment using the Van Lier model with plussages produces a more finely- distinguished gradient closer approaching the ideal for perfect discrimination across an infinite number of productions from zero to the maximum. NOT

30 Some results : written production

31 Some more specific analysis NOTE: NOTE: This graph was derived from samples of written responses to a single, common task and rubric in one examination for external IB assessment. It allows for comparisons across raters and rating systems, AND across the individual assessment of each production. Similar general trends are apparent, as previously, despite the far smaller sample and application to reading and writing, rather than listening and speaking. Extreme cases are clearly identified, with the plussages in most cases exaggerating the effect of aberrance. ContinueDevelop

32 Some refinements to interpretations, after experiment Creator authenticity as expressions of self: Creator authenticity as expressions of self: assessment requires evidence of particularisations of identity. An individual should be revealed, for example in the recounting of autobiographical incident, personal attitudes, emotions, dilemmas, expectations for the future, amongst others, thus allowing for originality and avoiding recitation of pre-learned models, or plagiarism. This is the self identified by listeners and readers as a personal voice. Creator authenticity in perceptions of other, as interlocutor, audience or reader: Creator authenticity in perceptions of other, as interlocutor, audience or reader: assessment rates appropriate participation in communicative interchanges. There should be for example, evidence for retaining and developing listeners or readers attention through the use of appropriate content, appropriately adapted by form, maintaining appropriate levels of participation, the text being relevance to its audience. Negative evidence pointing to a lack of proficiency in this respect, would be the demonstrable need for interlocutors and readers to put in special effort to maintain appropriate levels of interaction. Finder authenticity: Finder authenticity: where evidence in the content of responses illustrates purposeful, critical selection and manipulation of sources of knowledge appropriate to the chosen task and its genre, in intellectual, cultural, or emotional terms, and so forth, as sourced outside self. It demonstrates the resourcefulness of the producer of language. User authenticity: User authenticity: where evidence is assessed for the purposeful adaptation of content to fit task and context. That is, the content found from resources is appropriately applied to the task in hand.

33 Some refinements to interpretations, after experiment Authenticity of context: Authenticity of context: or evident construction of extensive and extendable communicative interactions. Text-production indicates appropriate negotiation and agreement with initiatives from other, as a representative of the target-language culture, given tasks and situations chosen. There should be demonstrable awareness of effects on response-receivers, as markers of interaction. There will be no evidence of sustained irrelevance, evasiveness (unless appropriate), cultural ignorance or insensitiveness, or indeed attempts to close down channels of communication. A clear respect for other is indicated in a situation of give and take. Authenticity of purpose: Authenticity of purpose: or organisations of content with evidence for promotions and facilitations of changes in perspective and knowledge amongst audiences of the text created, and on occasion reflexively in self, thus granting the production a quality of convincingness. The receiver knows easily, why communication takes place, both in choice of genre as its framework and in the message that it contains. Authenticity of interaction: or the accommodations of self to other in processes of continuous change, marked through the unfolding of the text. There is assessable evidence of responsive, perhaps spontaneous recognitions within self and in other, of ability and will to guide this development.

34 Some refinements to interpretations, after experiment NOTE: The remaining criteria under Evidence for Personal Authenticity are more psychological in focus, and therefore more purely dependent upon the experientially-based, subjective and personal interpretations of overall effect in communication, developed by listeners or readers. Nevertheless: Existential Authenticity Existential Authenticity precludes overt plagiarism, whose presence would constitute a negative factor, indicating a weak or absent level of authentic language use in this respect. Intrinsic Authenticity Intrinsic Authenticity denotes the operations of selection and choice, with evidence to maintain linguistic interactions between speaker and listener, writer and reader, in a manner that involves their deepening and / or widening. Autotelic Authenticity Autotelic Authenticity reveals flow and psychological balance throughout communicative productions, experienced as levels of engagement with, and concentration on the task in hand.

35 Conclusions in relation to the IB programmes, assessments and evaluations Need better to theorise categorisations and relationships for converting descriptive assessments to quantitative evaluations. Need for an explicit rationale in ascribing 33,33% as a maximal value for each discrete, major criterion, as conceptualised in the current scheme. Need for the assessment of the desirability and feasibility of measuring interactive, communicative value as authentic, across a range of languages and proficiency levels.

36 Some considerations put forward The reduction of assessment categories by descriptor could improve assessor reliability, with assessment choices more constrained and therefore, less variability in interpretation possible. An increase in the overall number of discrete, assessment categories could isolate more significant features of effective, authentic communication, better focussing the attention of teachers and students alike on what is of value. The conceptualisation of authenticity in linguistic interaction suggests that balance across all criteria is significant in effective communication and should be taken into assessment consideration. The inclusion of extreme categories could aid in identifying the (authentic) appropriateness of candidate placement by school administrations.

37 Some key questions remaining How can we ascribe justifiable quantitative values to qualitative assessment categories and level descriptors? In the interests of transparency and improvements in understanding, how can the values of weighting and aggregation of discrete quantitative assessments be made explicitly justifiable? How can final quantitative evaluations be justifiably described, (most particularly to interested third parties)? Can the differences between discrete assessment categories be made more distinct without loss of validity, in order to facilitate use by assessors, trained, untrained, experienced or inexperienced? DevelopContinue

38 More research needed on: the reliability of the experimental model over a range of: triangulations of data across different languages at different levels, from beginner to native speaker, and all in between. assessors, from expertly trained and highly experienced to trained novice, untrained teacher, students and perhaps non-specialist native speaker, as targeted receiver of language input.

39 The relationship of criterion categories to quantitative value, as perceived by a range of interested parties, such as: Professional assessors Teachers Students Parents Institutes of Higher Education and their admissions sections More research needed on:

40 Relationship of values determined by any new model of assessment and evaluation to other systems, such as: Previous IB programmes Council of Europe values Other systems, such as: National for non-national learners (e.g. DELF/DALF, ZDaF, ZMP, etc) National for national learners of foreign language More research needed on:

41 References CSIKSZENTMIHALYI, M., (1990), Flow: the psychology of optimal experience, New York, Harper & Row. INTERNATIONAL BACCALAUREATE ORGANISATION, (1997), Guide to the Diploma Programme (in English and French versions), Geneva, International Baccalaureate Organisation. INTERNATIONAL BACCALAUREATE ORGANISATION, (2001), Paper Specific Instructions, Language B, Higher and Standard Levels, Cardiff, International Baccalaureate Curriculum and Assessment Centre. INTERNATIONAL BACCALAUREATE ORGANISATION, (2002), The Diploma Programme: Group 2 Languages (in English and French versions), Geneva, International Baccalaureate Organisation. INTERNATIONAL BACCALAUREATE ORGANISATION, (unpublished draft), Standardisers Guidelines and Checklist, Cardiff, International Baccalaureate Curriculum and Assessment Centre. ISRAEL, J., (unpublished doctoral thesis, 2003), Authenticity and the Assessment of Modern Language Learning, Open University, Milton Keynes. ISRAEL, J., (2007), Authenticity and the Assessment of Modern Language Learning for International Baccalaureate Diploma, Group 2 Languages, in The Journal of Research in International Education, London, Sage Publishers. SARTRE, J-P., (1946a), LÊtre et le Néant, Paris, Gallimard. SARTRE, J-P., (1946b), LExistentialisme est un Humanisme, Paris, Nagel. VAN LIER, L., (1996), Interaction in the Language Curriculum: Awareness, Autonomy and Authenticity, London and New York, Longman.

42


Download ppt "Research into an Exemplary Programme The International Baccalaureate Diploma Programme Group 2 Languages Dr. John ISRAEL AND THE ASSESSMENT AND THE ASSESSMENT."

Similar presentations


Ads by Google