1. BUREAU FOR INTERNATIONAL LANGUAGE COORDINATION BUREAU DE COORDINATION LINGUISTIQUE INTERNATIONALE STANAG 6001 TESTING: INTEGRATING TEST DEVELOPMENT STAGES SHAPE, MONS-BELGIUM Julie J. Dubeau, BILC Chair Welcome Address,
Good morning distinguished guests, dear delegates, dear hosts and Mike! : Welcome to the annual STANAG 6001 Testing Conference.
For those of you who don’t know me, my name is Julie Dubeau, Chair of BILC since May of this year.….
I am absolutely thrilled to be here, as this is my first time attending the STANAG Testing conference. Back in 1994, almost 20 years ago, I had the honour of attending my first BILC conference. At that time, I had been working for Defence for 4 years: I had spent two years as a French teacher, was a new tester and the FL test coordinator. My background was in languages and I had taken one 4th year University course in language testing. I worked alone, developing tests in multiple FLs with contractors, and had to learn many things the hard way – from my mistakes.
I can assure you that coming to BILC was a real eye opener -- I was NOT alone! Having the opportunity to meet with other language testing professionals working in a military environment was both reassuring and rewarding and that experience stayed with me for a number of years. As I mentioned in my opening brief in Tbilisi, at that time, I could not have imagined that I would someday be addressing the delegation as the chair of BILC… I’m particularly pleased to be here because I worked directly in testing for 10 years, language testing is/was my area of research for my M.A. In Applied Linguistics, and although I no longer test, it is a subject which is close to me. We are testing colleagues, most of us are or have been practitioners, and here we are.
As an organisation, we have come a long way. This conference is a very unique forum… This is the only testing conference dedicated to testing with the STANAG 6001 framework. This is our discourse community, and our shared testing situation: We test military personnel in a high stakes environment, and we have a mutual understanding of the rewards and challenges that our unique setting implies... One of those challenges is integrating all aspects of the various test development stages….and on that note, I will begin my brief opening address.
Next slide

2. Outline What is BILC? Membership, Secretariat, Mission
Programme of work:
Events
Cooperative Language Training Assessments
Assistance to Testing Programmes
NATO Language Context
STANAG 6001 Conference Theme
This morning, my presentation will be very brief< I will begin by saying a few words about our BILC organisation; then I will talk a little about our NATO language context before address this year’s theme.
Peggy will provide you with more detailed information on the main programme of work and standardizing efforts.
Next slide

3. BILC Secretariat Chair
Steering Committee
BILC
Secretariat
Chair
J. Dubeau
GRB ( )
DEU ( )
USA ( )
CAN ( ?)
Senior Advisor
Dr. R. Clifford
Secretary
J. Vasilj-Begovic
BILC was established in 1966 as an advisory body to NATO. Partner countries routinely participate. All nations are welcome to attend and/or observe. The BILC Chair presides over the Steering Committee which meets at the spring conference. Voting members are the heads of NATO delegations.
Next slide Canadian Secretariat may be extended beyond We are awaiting official confirmation.
Associate
Secretaries
Language
Testing Programmes
P. Garza
Language
Training Assessments
K. Wert

4. BILC Mission & Vision
Mission: To promote and foster interoperability among NATO and PfP nations by furthering standardization of language training and testing. To support the Alliance's operations through the exchange of knowledge and best practices, IAW established procedures and agreements.
Vision: To achieve levels of excellence where progress made by one is shared by all.
As our mission and vision statements capture, our primary mission is to promote interoperability through standardization of training and testing practices. Our goal is to foster the exchange of knowledge and best practices.
The BILC vision is to share progress made with member nations, partner nations and any nation who wishes to advance in the field of language training for military purposes.
How is this done? Through the secretariat, there are a variety of initiatives that aim to foster these exchanges and promote standardization, and as I mentioned a moment ago, Peggy will provide you with more detailed information on those important programmes.
Just want to mention that BILC has assisted numerous NATO and NATO Partner countries in the past decade, first under US leadership and now in collaboration with Canada: Azerbaijan, BiH, Bulgaria, Czech Republic, Georgia, Jordan, Kazakhstan, Moldova, Netherlands, Romania, Russian Federation, Serbia, Spain, FYROM, Ukraine.
There are two main BILC events, and a third, equally important – this STANAG testing conference, which as of next year, will be known as a STANAG WORKSHOP that will not only be limited to testing but will address STANAG-related training issues as well. It will remain a practical workshop focusing on specific STANAG-related challenges and topics.
Next slide

5. Two Main Events BILC Conference in Spring
Last hosted in Tbilisi by Georgia
“NATOspeak: English in Multinational Settings”
2014 Conference in Bruges, Belgium: 4-9 May
Theme TBD
BILC Professional Seminar in Fall
Last hosted in Bled by Slovenia
“Back to Basics: Recipes for Instructional Success ”
2013 Seminar hosted by Sweden, in Stockholm: Oct
“ Diligent Design: Building Blocks to Maximize Learning”
The Conference in the Spring and The Professional Seminar in the fall each offer opportunities for language teachers, testers, managers and military language stakeholders to learn and contribute to the collective effort.
Our next Professional Seminar will be held in Stockholm: October. The theme “ Diligent Design: Building Blocks to Maximize Learning” and some of the following sub themes: Iinstructional design techniques, Innovative test designs and formats, Designing training plans and lesson plans, Blended learning models, Measuring instructional success, etc.
The Spring Conference will be in Bruges, Belgium, 4-9 May, We are currently working on defining the theme which will be announced in Stockholm.
Next slide

6. Military / NATO Language Context: Complicated and Complex!
Each nation is responsible for own training / testing programme
No common tests - only a common standard - STANAG 6001 Nations certify their military and give members SLPs based on own STANAG– based tests.
SLPs of job descriptions may not reflect actual language requirements
‘Targets’ may be set too high, but national policies sometimes set unrealistic goals as well
We first must acknowledge that in the language training and testing world, the military testing environment is somewhat of an anomaly.
Each nation is responsible for its training / testing programme. There is no overarching language training/testing authority – all is done nationally – and in good faith. BILC is an advisory body only - we certainly suggest, but we do not prescribe.
As you know, there are no common tests - only a common standard - and BILC is the custodian of that standard, the STANAG Still, this is atypical in the language testing world whereby a central authority normally controls language testing and accuracy of results. But, we all certify our own military and give members Standardized Language Profiles (SLPs) based on our own STANAG– based tests.
Almost a decade ago, BILC accepted a NTG tasking to standardize through Benchmark testing and a BILC WG created a Benchmark Advisory Test – (The BAT) to be used by nations as an external measure against which nations could calibrate their national STANAG tests. Over a dozen nations have taken advantage of this tool to get an unbiased view of their SLPs. This is still today a useful tool at nation’s disposal, but its use is voluntary… And unless nations voluntarily take the BAT, there is no way to know if those test instruments are providing results that are valid and reliable.
Another piece of the NATO language puzzle is the fact that SLPs of job descriptions may or may not reflect actual language requirements. BILC has argued for a number of years that it may be the case that not all positions in all HQs require level 3 skills across the board, and that it might be useful for nations to know which ones do. A rigorous Language Needs Analysis (LNAs) of positions would be informative. This is still something I am exploring, and thanks to our IMS colleague Phil Turner, I will be exploring this further this week at NATO HQ.
Furthermore, Nations must reach Eng language “Targets” which may or may not be realistic goals, for example of aiming to train all officers to level 3 is an incredible challenge for many nations. In some nations, we know it is extremely difficult to find qualified teachers with this high level of proficiency.
As a dear BILC colleague occasionally quips: “Having all military personnel trained to level 3 is not a policy, it’s a fantasy”. In some nations, the onus is on the military member not only to reach this level, but to maintain it. It is a challenge.
Some counties first language is very different from English, and as this next graphic shows, there is great linguistic diversity among NATO’s partners, all of whom are expected to understand and speak International English for Operational and Staffing purpose.
Next slide

7. Global Partnerships
"English is the de facto Lingua Franca , the common language, but, as we discussed in Tbilisi,.
We know all nations have their own robust training system, well, let’s be honest, some are more robust than others, but overall huge investments of money and many hours are spend on language training. That said, despite our best efforts, we are continuously made aware that there are gaps in interoperability due to language shortcomings.
Next slide

8. Language is #1 shortfall
English skills lacking - Need for proficiency is increasing
Native speaker not proficient in communicating in multinational environment
Documents & needed correspondence too lengthy, too complex
Inadequate harmonization of requirements
& procedures
Content/intensity of training falling short
Despite the hard work, language is still viewed as a critical factor in the interoperability shortfall: Responses to a 2010 survey from ISAF and KFOR revealed that deficiencies in English language skills were perceived to be detrimental to mission. 8

9. INTEGRATING POLICY & REQUIREMENTS
What is needed – When? for whom?
Testing Proficiency and Performance
Capturing Capability through testing
Accountability through measurement
High stakes
Access to promotions & postings
In all of our military language training systems, we have or should have over seeing policies which define the requirements and outline how to get to the end goals. An organisational language policy for mil and civ personnel provides the underpinning of a framework which can be trickled down to testing plans and directives, indicating who has access to testing, when, and why.
It would outline how the tests would be delivered, what may be tested, and results captured for future use. Testing provides information on how well a training system is performing, or on the success of the training.
And we know that our testing context carries high stakes for personnel. Results are often used for promotional purposes, for posting allocations and even for monetary incentives.
Next slide 9

10. INTEGRATING TEST DEVELOPMENT STAGES
Communications with stakeholders
Washback
7 Stages of Development:
Practicality Issues
Challenges
These STANAG testing conferences have focused on specific skills tested, and on standardisation. This year’s theme is providing opportunity to focus on integrating the various stages of test development into a coherent whole in order to end up with a valid, reliable product which will provide information to the various stakeholders.
Communication is paramount and candidates and users of this information such be informed of the testing process, format and results. In some institutes the testing specifications are shared, and in others they are kept confidential.
We are well aware of the effect of testing on training, positive and negative and this washback effect cannot be ignored.
From making the statement of the testing ‘problem’, to writing specifications for the test, Writing and moderating items, Trialing the items informally on native speakers, Trialing the test, Analysing the results of the trial and make any necessary changes, Calibrating scales, validating the items and the test, writing handbooks for test takers, test users and staff and training staff (interviewers, raters, etc.), all phases and stages must be coherently planned. There are problems to solve, and no test project will ever be perfect.
I very much look forward to the presentations and discussions this week, which will allow us to see how the whole process comes together.
Now for brief updates
Next slide

11. Updates www.NATOBILC.org STANAG for Non-Specialists
LNA – Planning, on-going
WG on Level 4 - Final report
WG on Translation & Terminology, on-going
After many obstacles and delays I am pleased to announce that we have purchased the domain name NATOBILC.org and that we are now hosted on a commercial server.
The STANAG for Non-Specialists document has been through a number of drafts, but it is now a final document that is available to all for distribution. It will also be available on the BILC website. It is intended for HR staff and other non language professionals to use as a quick reference. We will pilot its use when we assist IMS with the language needs analysis of HQ posts.
Next, The WG on Level 4 has produced a Report for nations who want to know more about the complexities tied to testing at this very high level of proficiency. An executive summary is available and the full document has been posted on the website.
Last but not least, the WG on Translation and Terminology has completed Phase 1 of their review of terms and are moving on to a next document.
That’s it for my briefing. NEXT SLIDE

12. Contact us: www.NATOBILC.org Julie.Dubeau@forces.gc.ca
Before I turn the floor over to the next presenter this morning, I’d like to take this opportunity to thank Mike Adubato for hosting this event at SHAPE. For many of us, it’s the first and maybe only chance to get this close to the action. I’d also like to express my gratitude to my colleague and friend, BILC secretary Jana Vasilj-Begovic, who works so hard to organize and plan the content and our days’ work during the conference. Due to her dedicated interest in this conference, we will have a very interesting few days ahead. Last but certainly not least, I’d like to thank BILC associate secretary, Peggy Garza, who not only works closely with Jana on these conferences but whose commitment to NATO language standardizing efforts are the real, concrete success story behind standardizing. Ladies, thank you for making these workshops possible.
BILC is a special and unique organization, and the BILC family members, among other things, like to work hard and to play hard. Let’s enjoy the workshop and look forward to also sharing some good times together later on.
Thank you for your attention!!