ENHANCING TRAINING AND TESTING EFFECTIVENESS THROUGH LESSONS LEARNED Emilia Nesheva, Ministry of Defense, Bulgaria Greta Keremidchieva, Rakovski National Defense Academy, Bulgaria MINISTRY OF DEFENSE REPUBLIC OF BULGARIA

ENGLISH LANGUAGE TRAINING AND TESTING – OFFICIAL POLICY OF THE MOD BASED ON: DOCUMENTATION DOCUMENTATION TRAINING PROGRAMS TRAINING PROGRAMS QUALIFIED TEACHERS – “SOFT POWER” QUALIFIED TEACHERS – “SOFT POWER” BOOKS AND EQUIPMENT – “HARDWARE” BOOKS AND EQUIPMENT – “HARDWARE”

REGULATING DOCUMENTS 1. Order of the Minister of Defense №130/ regarding Strategy for development of the English language training system 2. Order of the Minister of Defense №760/ regarding selection of candidates, preliminary testing and conducting English courses 3. Regulation №Н-19/ regarding planning and development of language courses.

LANGUAGE TRAINING CENTERS  RAKOVSKI NATIONAL DEFENSE ACADEMY – SOFIA  NATIONAL MILITARY UNIVERSITY – VELIKO TARNOVO  LANGUAGE TRAINING CENTER – SHUMEN - part of the Individual Training and Education Programme – ITEP, NATO Tier 1 Smart Defence 1.7 Project  NAVAL MILITARY ACADEMY – VARNA  MILITARY UNITS

ENGLISH LANGUAGE TRAINING IN RAKOVSKI NATIONAL DEFENSE ACADEMY  Training of officers in the Master’s degree program  Training of military personnel and civilians in General and Specialized English courses

Target audience in Rakovski National Defense Academy   Officers in the Master’s Program Entry – now 1111; next year (2222) Exit – now 2222; next year ( )   Officers, NCO’s and civilians in English language courses NEW

Training program in Rakovski National Defense Academy   The Master’s Program starts with a preliminary block of 360 hours intensive English training prior to their first semester of military studies   Advantage: some military subjects could be studied in English NEW

General and Specialized English language training Course LengthBooks ALC Entry requirements Expected exit outcome Level I18 weeks points ALCPT Level II18 weeks points1+ 1 Level III15 weeks Level IV15 weeks Specialized courses 2-8 weeks2 2 - NEW

ASSESSMENT AND CONTROL  Preliminary Testing (ALCPT before Level ІІ)  Entry Tests  Book quizzes, Progress tests  Final Test, ALCPT, STANAG 6001

FACULTY MEMBERS  Teachers – 10 University graduates with specializations in different countries and institutions; experienced; participants in national & international conferences specializations in different countries and institutions; experienced; participants in national & international conferences  Test Team members – 5 University graduates; trained; design, pilot & administer STANAG Tests nation-wide

BOOKS & EQUIPMENT  ALC Books  Specialized textbooks – Campaign, Military English, Army, Air Force, Navy, etc.  Language Labs  Self-Access Center  Audio & multimedia programs

FINANCIAL RESOURCES  MOD program provides financial resources for the accomplishment of Force Goal Е 0356;  for improvement of infrastructure and learning technologies;  for conferences and seminars.

Enhancing Testing Effectiveness through Computer Testing Enhancing Testing Effectiveness through Computer Testing

Action Plan Ministerial Order № 857/  till 1 March, 2012 – approbation of the computer-assisted “writing” module;  till 1 July, 2012 – analysis of the results;  since exam session Autumn 2012 – optional computer-assisted “writing” module;  till 1 March, 2013 – approbation of computer-assisted modules for “reading” and “listening”;  till 1 July, 2013 – analysis of the results;  since exam session Autumn 2013 – optional computer-assisted “writing”, “reading”, and “listening” modules;  since exam session Autumn 2014 – all candidates are to be tested on computers for the modules “writing”, “reading”, and “listening”. End of pen-and-paper STANAG 6001 tests.

Approbation of the computer-assisted “writing” module In February 2012, two groups of volunteers – military and civilian personnel sat for the approbation of the writing test. The only requirement for taking part in the approbation was to have a STANAG 6001 certificate issued during the exam session “Autumn 2011”. In February 2012, two groups of volunteers – military and civilian personnel sat for the approbation of the writing test. The only requirement for taking part in the approbation was to have a STANAG 6001 certificate issued during the exam session “Autumn 2011”.

Comparability of handwritten and computer- based writing assessments – some findings in the specialized literature

Questionnaires (excerpt)  Did you find the font of the tasks friendly enough?  Did you meet any technical difficulties during the computer-assisted “writing” test?  Do you think that your speed of typing is decisive for your success at the computer-assisted “writing” test?

Recommendations from the Questionnaires 1. The font of typing to be Times New Roman 12 or Candidates to be able to go back to the previous task for revising or correcting mistakes. 3. Word count to be available. 4. Tabulation for indicating a new paragraph to be available. 5. Five minutes for a spell check to be added to the time allotted for the writing test.

Actions Taken  The commercial company that had developed the testing software was tasked to correct the faults noticed during the approbation and to add the functions recommended in the questionnaires.  The STANAG testing team developed detailed administrative instructions for the computer-assisted writing test.

Approbation Results

Analysis Special attention was paid to the candidates who attained higher or lower STANAG 6001 level at the computer-assisted writing test. Analysis of the possible reasons: Special attention was paid to the candidates who attained higher or lower STANAG 6001 level at the computer-assisted writing test. Analysis of the possible reasons: - was the rating of their pen-and-paper test subject to discussions? how many raters had assessed the papers? - additional English language studies? - different “pre-writing” habits? - rater effects in scoring handwritten and computer-based writing assessments?

LESSONS LEARNT (possible scoring biases)  Typed essays generally appear to raters shorter than identical handwritten responses.  Errors made are more evident when typed.

LESSONS LEARNT (cont.) (the importance of the Interface)  Traditional text editing tools – cut, copy, and paste – need to be designed to support less experienced computer users.  Space management – “word count” and “print preview” should be available.

Questions?

In this presentation have been used quotations by:  Breland, Lee, &Muraki, 2005  Bridgeman & Cooper, 1998;  Davis, Strain-Seymour, Lin, & Kong, 2008;  Horkay et al., 2006;  Pearson, 2010;  Russell &Plati, 2001;  Russell & Haney, 1997;  Way & Fitzpatrick, 2006.