PROGRAM EVALUATION PROJECT 2013 PRELIMINARY FINDINGS
ıNTRODUCTıON This presentation aims to share some of the preliminary findings that were reached as a result of the data collected and analyzed for Program Evaluation Project – 2013. The slides mainly have tables and graphs, and following each slide, we added some explanations to help interpret them. The rest of the findings were presented in the slides used on 18 September, 2014.
CONTEXTUAL VARıABLES This slide shows the components that are subject to evaluation at the School of Foreign Languages. The main components are the two main departments, and the EPE is viewed as a component in between the two.
INITIAL PHASE OF PROGRAM EVALUATION 2013 Needs Assessment The existing curriculum and syllabus The materials in use The assessment tools The learners The instructors (DBE, DML, Faculty) The resources
DBEMLDFaculty Ss QIns Focus Group IntInterviews Ins QSs QMaterials Analysis Ss Focus Group IntClassroom Obs Ins Interview Classroom ObsDocument Analysis Materials AnalysisSs Product AnalysisFreshman Int Ss Product AnalysisSs Interviews (?) Freshman Product Analysis
DATA SOURCES AND DATA COLLECTıON ıNSTRUMENTS This slide shows the departments from which data were collected, the sources, and the data collection instruments used. The darker color used in the boxes in the lower half of the slide indicates that the data collection procedure is still in progress.
COMPLETED JOBS ToolsDBEMLDFaculty Questionnaire 2,612 (ss) 95 (Ts) ~800 Interviews 2029 Focus Group Interviews 4511 Classroom Observations 155
ANALYSES ıN PROGRESS DBE Ss Questionnaire DBE Ins Questionnaire MLD Ins Focus Group Interviews Faculty Interviews Faculty Documents
PENDING JOBS Qualitative Data Analysis Freshman student samples MLD Student samples DBE Student samples Interviews Observations
PENDING JOBS Quantitative Data Analysis MLD Student Qs
IMPORTANCE VS ADEQUACY OF PROGRAM COVERAGE OF SKILLS & COMPONENTS (INS) Both the instructors and the students were asked to rate the skills in terms of their importance and the adequacy with which they are covered in the program. This graph compares the instructors’ responses to “importance” and “adequacy of coverage” of individual skills. Please remember that 1 indicates “strongly disagree”, 4 indicates “undecided”, and 7 indicates “strongly agree”. In this line graph, the only area where the degree of importance and the amount of coverage are seen as close to equal is “grammar”. Instructors perceive grammar as important and adequately covered in the program.
CONT. Instructors’ views indicate that they rate “reading” and “vocabulary” as the most important areas, followed by listening, writing and speaking, and grammar (in descending order). Pronunciation is rated the lowest, with a score above 5.5, indicating that it is considered “important”. The coverage of the skills and subskills (except for grammar) is seen as below the importance ratings. There is a small difference between the importance (at around 6.5) and adequacy of coverage of “listening”, “reading” and “writing” (at around 5.5) skills. A larger difference emerge in “speaking” (importance=6.25 but adequacy of coverage=3.88). We may say that instructors believe that speaking is an important skill to master but the instructional coverage is not quite adequate. Similarly, differences exist between importance and coverage in “pronunciation”, “vocabulary” as well.
The following five slides have two aims: to show and compare students’ and instructors’ views of the degree of importance and the degree of adequacy of coverage of skills and sub- skills in their group’s program. The top lines indicate “importance” and the bottom lines indicate “adequacy of coverage”. To be able to interpret the lines, it would be a good idea to have a look at lines that cross each other or move away from each other in one graph. Then, have a look at those in the other graph. Compare and contrast. Please keep in mind to check the scores (from 1 to 7).
EXAMPLE: BEGıNNER GROUP Students’s views Speaking: most important BUT least adequately covered Listening and vocab: very important BUT not so adequately covered Grammar: important BUT covered too much Instructors’ views Speaking: important BUT not adequately covered Vocab: important BUT not so adequately covered Grammar: quite important BUT covered a little bit too much
Level Taught BEGELINTUINREP Mean I have my students do pair or group work activities. 5.685.635.765.755.29 I have my students work individually. 5.235.545.334.505.29 I assign activities that involve group-work outside class. 2.092.542.331.752.57 I assign activities that involve individual work outside class. 6.095.604.905.255.29 I make use of posters, flash cards, realia, etc. 4.9188.8.131.52.71 I make use of games, contests and puzzles. 5.004.344.194.754.00 I lead a teacher-centered classroom. 3.554.00 4.86 I involve my students in role play/scenario activities and discussions. 5.275.144.575.254.57 I involve my students in whole-class debates. 4.504.914.865.504.14 I encourage peer teaching in my classroom. 4.774.463.863.504.00 I have my students discover grammar rules from texts. 5.505.124.754.505.14 I explain the grammar points myself. 5.004.744.624.755.67 I use online resources for my students to practice English outside class. 3.773.713.764.753.86 I make use of technology in class. 5.824.944.814.753.71 I have my students use their own technological devices 4.824.004.764.503.71 I regularly allocate time to revise input. 5.274.804.904.505.00 I make use of weekly office hours. 5.094.894.052.254.29 I have my students provide feedback on each other’s work. 4.9184.108.40.2063.86 I provide the feedback on student output myself. 6.005.945.815.756.29
This table shows the DBE instructors’ preferred styles in teaching. The list includes activities and tasks that they can have students do during the lesson. Each activity is rated from 1 to 7. 1 indicates “never”, 4 indicates “half of the time”, and 7 indicates “always”. Each colon on the right shows the average score of a group. It would be a good idea to examine groups one by one and to locate the highest and lowest scores in each. Then, findings commonalities and differences among groups may indicate trends in the department. Pairs of activities that seem related gives some idea about the preferred styles in teaching (e.g.. The first two activities in the list).
EXAMPLE The most frequently preferred activities among beginner group instructors seem to be “I assign activities that involve individual work outside class” and “I provide the feedback student output myself”. The least frequently preferred activities among beginner group instructors “I assign activities that involve group work outside class”, “I lead a teacher-centred classroom”, “I use online resources for my students to practice English outside class”. The common points among all groups are: the high score in the last item: “I provide the feedback on student output myself”. the low score in items 3 and 13.
“I REGULARLY ALLOCATE TıME…” This graph shows us the responses of instructors to the item “I regularly allocate time to revise input” according to groups. Since the question was related to frequency, the scale was in that nature. 1= never, 2= rarely, 3=occasionally, 4= half the time, 5= often, 6= frequently, 7= always In each colon, the color on the top (maroon) indicates 7 = always. The other colors indicate frequency in descending order. Thus, lilac indicates 4= half of the time. No one indicated 1= never, therefore, there is no color to show it. Example: In Beginner group, more that half of the teachers allocate time to revise input at 6 & 7 (frequently and always).
MOST IMPORTANT SKILLS SPWRLGV BG S/TTT EL STS/T INT STS/TT UIN S/TTT REP TST
MOST ıMPORTANT SKıLLS This table aims to display differences and commonalities between the views of the students and instructors regarding the importance of skills. One thing to remember is that this does not mean to say the groups do not find the other skills as unimportant. ALL emerged as important in the graphs; however, we wanted to see which one/s came out as the “highest”. BLUE: Only students; RED: Only teachers; MAUVE: both groups S: speaking; P: pronunciation; W: writing; R: reading; L: listening; G:grammar; V: vocabulary Example: Reading is seen as the most important skill in all groups by instructors but this skill is not seen as the most important by students.
LEAST IMPORTANT SKILLS SPWRLGV BG TS/T EL TS INT TS/T UIN TSS/T REP TTS
IMPORTANCE OF FACTORS AFFECTING LANGUAGE IMPROVEMENT This bar chart displays the students’ and instructors’ rating of the importance of factors in language improvement. The bars on the right (light colored bars) indicate students’ ratings and the bars on the left (dark colored bars) indicate instructors’ ratings. In the rating scale, 1 indicates “totally unimportant”, 4 indicates “undecided”, and 7 indicates “very important”. Example: Students do not regard “reading texts in English” as an important factor but the instructors do. Example: Students’ highest score was “studying together”. This may indicate that they actually prefer and benefit from studying with someone else, perhaps with a “buddy” or “a big sister or brother”.
DBE SS - FGI SKILL/AREA WRITING 2 drafts, more often, more variety in topics READING MTR good, more dep. related texts / extensive rd SPEAKING extra time in program LISTENING more practice / more challenge GRAMMAR less detail & terminology VOCABULARY more practice – in class Main book: redundant / tasks too simple /DBE own book ? Lessons : less book following / more student activity Homework : more HW / more guidance for self-study
DBE SS - FGI This table shows the preliminary findings obtained from the focus group interviews held with the students at DBE. The ideas that emerged in all the groups (from Beginners to UIN) were summarized here. Only the suggestions and ideas have been displayed here.
FACULTY REPRESENTATION Architecture 4 Arts and Sciences 12 Economic and Ad. Sciences 3 Education 2 Faculty of Engineering 8 Total 29
This table shows the number of instructors that have been interviewed from different faculties. The tables in slides with the title “faculty needs” provide a list of the tasks and activities that the students are required to perform in the first year first semester in their departments.
FACULTY NEEDS WRITING Assignments Response paper (1-3 pages) Report (3-10 pages) Assessment (MT & Final)Short answer (1/2 page to 1 page) Essay (2-6 pages) Multiple Choice True False Fill in the Blanks Recurrent Theme: Paraphrasing, Comparing, Analyzing, Synthesizing
FACULTY NEEDS READING Assignments Book chapters Articles Recurrent Theme: Need to compare, analyze, synthesize, read long texts
FACULTY NEEDS LISTENING Out of Class Documentaries (BBC, CNN, etc.) Films In Class Lectures Seminars Recurrent Theme: Problems in understanding different accents (international instructors and native speakers), speed
FACULTY NEEDS SPEAKING In Class Conversational skills Presentations Recurrent Theme: Lack of confidence, too grammar focused
FACULTY NEEDS GRAMMAR Expectations Simple present & past tense Adj clause, noun clause Active and passive structures Recurrent Theme: Short simple sentences
FACULTY NEEDS VOCABULARY Expectations Knowledge of an acceptable range of vocab Knowledge of collocations Recurrent Theme: Discipline-specific vocab should be taught by content course instructors
MODERN LANGUAGES DEPT SKILL/AREA ? WRITING Prg org & conceptsContent (minors) READING Low level skillsHigher level skills SPEAKING A bitNot much LISTENING A bitNo note-taking (?) GRAMMAR A bitSimple mistakes VOCABULARY A bitSimple level Points : It would be a good idea to know how much is covered at DBE so we can make adjustments in our tasks/expectations.
This slide displays the findings reached from the interviews conducted with two groups of instructors at the Department of Modern Languages. It lists the points that are seen as “done well” and those that are “not done so well” by students in the first year first semester courses.
CONCLUSION Complete the analyses Factual information from the faculties How do we share instructional goals w/ the MLD? Exchange information with all parties involved Make use of research conducted at DBE (previous studies)
WHAT’S NEXT 2014-2015 Fall & Spring Term SFL Mission SFL Curriculum DBE Goals & Objectives DBE Syllabus Materials Development Assessment Tools Development