1. Evaluation at the Defense Language Institute
Dr. Thomas S. Parry
Directorate of Evaluation and Standardization
Defense Language Institute
Bureau for International Language Coordination
Budapest, Hungary
May, 2006

2. Why Evaluation?
Evaluation can make you rethink what you have been doing!

3. What is Evaluation?
The process of finding the value or amount of something—to determine it’s significance or quality.
-- Webster’s New World Dictionary

4. DLI Facts Average daily resident attendance: 3,079
Languages taught: 22
Number of students taught annually through all Continuing Education programs: 5,000
Highest enrollment languages:
Arabic
Chinese
Korean
Persian-Farsi
Russian
Spanish
Serbian-Croatian

5. Typical DLI Student Day
0515: Physical Training
0630: Breakfast
0730: Formation with Service Unit
0755: Class start
1530: Class end
1600: Military Training
Cumulative:
6 hours of class instruction
45 minute special assistance or enhancement training
3 hours of homework plus weekend assignments
2-3 hours of study hall (M-Th)
Developing & sustaining basic warrior-linguist skills, common task training, and physical readiness
The team approach enables DLI to maintain a classroom ratio of 1 instructor to 10 students, which is being modified to a 1 to 6 ratio across the board within the next 5 years.
Our small group instruction affords several benefits:
More time for instructors to focus on student proficiency
More time for students to speak in every classroom hour – vital to language development
Better student/faculty interaction, closer to immersion experience

6. Student Load by Difficulty
Resident Courses at Presidio of Monterey
DLIFLC
Basic Course
Language FY06 Student Load Faculty
Category IV Languages (63 weeks)
Arabic
Korean
Chinese
Japanese
2591 (71%)
Category III Languages (47 weeks)
Dari
Pashto
Persian Farsi
Russian
Uzbek, Hindi, Urdu, & 6 other lang
824 (23%)
Category II Languages (34 weeks)
German
27 (1%)
Category I Languages 26 weeks)
Spanish, French,
Italian, Portuguese
196 (5%)
Totals
94% of students taking CAT III & IV Languages
Resident Student Population
USA
USAF
USN
USMC
TOTAL
2001
1160
647
437
240
2484
2002
1560
754
549
205
3068
2003
1383
848
414
218
2863
2004
1469
1044
449
229
3191
2005
1426
1304
373
355
3458
2006
1297
1685
405
367
3754
2007
1479
1600
362
3846
2008
1609
1360
535
388
3892
2009
1597
1340
519
394
3850
2010
1594
1277
533
379
3783
Latest TRAP Numbers:
FY06*: 130 Students gained (3.7% increase over program)
FY07: 372 (10.6% increase over program)
*There is another TRAP this week; DLI could gain another 100 students

7. DLI Evaluation Goals Ensure student and teacher accountability
Provide for continuous academic program improvement
Faculty and staff development
Curriculum development and implementation
Gauge our return on investment
Ensure quality of life and support services are conducive to learning

8. Teachers Evaluate Students
Students are held to high levels of accountability for their learning

9. Teachers Evaluate Students
Teachers evaluate student performance in the classroom through:
Participation in classroom activities
Teacher-developed chapter/unit tests
Periodic counseling
Diagnostic assessments
Learning style assessment
Performance tests
Oral Proficiency Interviews (OPI)
Defense Language Proficiency Tests (DLPT)
Listening
Reading

10. Students Evaluate
Teachers
Program of Study
Quality of Life

11. Evaluation at DLI is Bi-directional
Teachers Evaluate Students
Students Evaluate Teachers

12. Teacher Evaluation Teacher evaluation at the DLI includes:
Complete student evaluation of the teacher
Scheduled and unscheduled classroom observations by the supervisor
Periodic reviews of performance objectives with the supervisor
The combination of these three teacher evaluation elements provides an accurate assessment of the quality and effectiveness of foreign language instruction

13. Teacher Evaluation
Each teacher is evaluated on classroom performance by all the students he/she teaches
A personal performance profile is created for each teacher by tracking his/her performance throughout each of their courses
Teacher evaluations by students have implications for merit pay and rank advancement

14. Interim Student Questionnaire
ISQ
Interim Student Questionnaire
Administered via computer to all students enrolled in 47- and 63-week language courses 60% of the way through the program
Takes roughly minutes to complete
Survey sections include:
Program Evaluation (PE)
Teacher Evaluation (TE)
Quality of Life (QL)

15. End-of Course Student Questionnaire
ESQ
End-of Course Student Questionnaire
Administered via computer at the end of language courses to all graduating students
Takes roughly minutes to complete
Survey sections include:
Program Evaluation (PE)
Teacher Evaluation (TE)
Quality of Life (QL)

16. Analysis and Reporting
Evaluation feedback from ISQ/ESQ delivered within two working days from survey administration
Deans
Chairs
Team leaders
Individual teachers
Special monthly reports provided to service commanders and individual units via separate questionnaires
Annual summaries for each instructor sent to Provost, Vice Chancellors and school Deans

17. ISQ/ESQ Response Scale
Items are rated from 1-4 with an option for no opinion:
0 No opinion
1 Strongly disagree
2 Disagree
3 Agree
4 Strongly agree

18. Teacher Evaluation (TE)
Sample Criteria
Class preparation and quality of instruction
Ability to teach language and related concepts effectively
Adjustment of teaching strategies
Ability to provide a variety of instructional activities

19. Teacher Evaluation (TE)
Sample Criteria
Active use of foreign language in classroom
Ability to provide challenging instruction
Counseling and assistance offered
Feedback given to students

20. Team and Individual Instructor TE Analysis

21. Teacher Improvement

22. Program Evaluation (PE)
Sample Criteria
Testing and grading
Team teaching
Homework
Curriculum and supporting materials
Overall evaluations

23. Quality of Life (QL) Sample Criteria
Barracks, family housing and child care
Quality of food, mess hall and related services
Medical and dental care
Chaplain and community services
Military support
Sport and recreation opportunities
Transportation and travel office services

24. Quality of Life (QL) Other Areas of Concern Equal Opportunity Military
Prejudice
Discrimination
Sexual harassment
Military
Training
Duties
General quality of life

25. PE and QL Analysis

26. Immediate turnaround; sent directly to Command Group for action
Serious Issues
Depression
Suicide threats
Sexual assaults and harassment
Fraternization and other misconduct
Immediate turnaround; sent directly to Command Group for action

27. Evaluation of Continuing Education Programs
ESQ questionnaires distributed world-wide to training sites and include:
Video tele-training sites
Mobile training team deployments
Language Teaching Detachments
Command Language Programs
Special, customized programs
Questionnaires completed and returned to DLI via on-site program managers
Analysis and feedback provided to instructor’s school and training site within two weeks

28. Conclusion DLI evaluation practices:
Are well established—honed through years of practice
Are rigorous and thorough
Bring about faculty and program improvement
Establish baselines for student/teacher accountability
Give us cause to rethink what we are doing and consider how we might do things better

29. Evaluation at the Defense Language Institute
Questions?
Discussion?