1. Test Writing: Moving Away from Publisher Material
Welcome back from lunch, I am Bruce Butterfras and I have the most enviable spot on the schedule today, right when our parasympathetic nervous systems are working overtime to make us all sleepy. Hopefully I can be entertaining enough to keep us both awake and maybe help us learn something of value.
This probably sounds like a confused title, come on how is “Test Writing” related to moving away from publisher material? Bottom line is that I had two ideas that I was passionate about presenting and couldn’t decide on just one, so I combined them. Hopefully it will be successful, we will see.
Bruce Butterfras, MS Ed., L.P.
Assistant Professor – Dept. of Emergency Health Sciences
University of Texas Health Science Center at San Antonio

2. Objectives
Explain the importance of an EMS curriculum unique to your program
Identify the steps involved in creating good exams
Create and evaluate exams for your courses
These are the objectives of the presentation. We will start out looking at why it is important to make your EMS curriculum unique, that is not just like every other EMS program or course out there. There are advantages to keeping EMS curricula the same, but if you want your students to have a good experience and learn to be good medical care providers, you need to think a little out of the box. One of the ways you can make your curriculum or just your individual course unique is to write your own exams. Good exams evaluate student knowledge fairly but they can also identify shortcomings in the student’s instruction. Knowing steps to follow to create exams can help you do a better job. Finally the last thing I want you to be able to do after this presentation is to make and evaluate your own exams for your courses. I know this sounds like an ambitious undertaking and it probably is, but what the heck, let’s dig in.

3. Why Customize My Course?
EMT/Paramedic National Standard Curriculum
From 1971 to 2000
Contained specific knowledge and skills objectives
EMS Education Agenda for the Future (2000)
National EMS Education Standards
“The EMS NSC, with their detailed declarative material, limit instructor flexibility and the ability to adapt to local needs and resources”
First I have to start with a little history. I have to warn you history was always one of my favorite subjects, but I realize it isn’t for everyone.
What I want to look at very briefly is the birth and early years of EMS Education. The first EMS courses were taught by the Medical Direction physicians.
If you want to know what that was like, search YouTube for the pilot episode of a show you might have heard of called ‘Emergency’.
Based on the success of early programs such as the one in LA County, the US Department of Transportation decided we needed a National Standard Curriculum for EMS.
The first EMT National Standard Curriculum was published in 1971 and the other levels soon followed.
These Standard Curricula contained specific knowledge and skills objectives, everything you need to know to be a functioning EMT / Paramedic.
At around the turn of the century, The EMS Education Agenda for the Future was published and was quickly followed by the National EMS Education Standards.
This quote from the ‘new’ Education Standards indicates the need for local educators to teach a curriculum geared towards their community needs.

4. Publisher Resources Most EMS textbooks come with instructor resources
These often include chapter objectives, lesson plans, PowerPoint presentations and test banks to be used to construct exams
When selecting a textbook these additional resources should be a strong consideration
Disclaimer: I have helped create Publisher material including PowerPoint presentations and test questions
In the early days of EMS education, all EMS textbooks included every objective from the National Standard Curriculum and so they were often very similar.
As competition began between textbook publishers, one of their selling points became ‘Instructor Resources’ often including detailed lesson plans and decks of slides (which were the predecessor of PowerPoint but without the ability to be edited and not requiring a computer). As time passed the sets of slides became replaced by PowerPoint presentations and Lesson Plans were included on floppy discs. Another ‘help’ provided to instructors were test banks or test generators which could be used to create exams at the touch of a few keys. Obviously publisher materials have improved tremendously over the years but there are still some significant shortcomings you need to consider.
Oh and by the way, I have to include the disclaimer that I have done some freelance work for a book publisher including writing and editing chapters, creating PowerPoint presentations and writing questions.

5. Publisher Objectives
Publishers create objectives based on the National EMS Education Standards
The National EMS Education Standards are intentionally broad, so specific objectives vary among the publishers
Publishers’ objectives are by necessity generic in order to be applicable to all areas of the country
Often specific to their textbook as well
Just as the old textbooks relied on the National Standard Curriculum for their objectives the newer ones develop their objectives based on the National EMS Education Standards.
The difference is that the new standards don’t specify objectives so the publishers or actually their authors and editors must create their own objectives.
Since the new EMS Education Standards are fairly broad, the objectives have begun to be a little different among the EMS Textbooks. The thing that hasn’t changed is that textbook objectives need to be broadly applicable. So if you are on the coast of Texas and need to emphasize heat and ocean emergencies, your textbook will probably have a little information on that included along side the information on hypothermia treatment and avalanche rescue. Not saying that Texas EMTs and Paramedics don’t need to know a little about dealing with cold weather, after all we do have at least a couple of weeks of it here in Texas every year, but we really have a little bit different focus from the medics in the mountains of Colorado.

6. Publisher Lesson Plans
Publisher provided Lesson Plans are also relatively generic in order to work for programs around the country
These lesson plans usually focus on presentation of material (that is found in the textbook) to students in a standard (ie. lecture oriented) educational model
Just like objectives, Publishers sometimes include Lesson Plans in the Instructor Resources, that come with their textbooks. Just like you probably need to tweak the objectives to fit your program or course, you also need to look at changes you can make to these Lesson Plans. The Publisher provided lesson plans are very generic by necessity in order to meet the needs of programs around the country. These lesson plans also generally focus on a standard or traditional educational model, that is the one where you lecture to your students on material they can read for themselves in the textbook. Hopefully you know that there are many other models of teaching that have been proven to be more successful for teaching, but using these will require you to create your own Lesson Plans (or significantly modify the ones provided from the publisher).

7. Publisher Presentations
Publisher provided PowerPoint presentations highlight the material presented in the textbook
This can allow students to follow along in the textbook as the lecture progresses
Publisher provided PowerPoint presentations can also be:
Reorganized to fit your way of teaching the subject
Added to with additional materials from your own research on the topic
Publisher provided PowerPoint presentations generally restate the same material that is found in the textbook.
If you want your students to read the textbook don’t read it to them from the slides.
It is OK if you don’t want to start from scratch and create your own presentations, to modify those provided by the publisher.
Do you prefer to present suctioning before oxygen administration, but the textbook has it the other way around, reorganize the slides.
Have you heard the latest information about using ultrasound technology in the prehospital setting, but that information isn’t on the slides, add it.

8. Modifying Presentations
Publisher provided PowerPoint presentations can also be:
Deleted from if you feel there is material included that doesn’t need to be discussed
Spiced up to make them more appealing to students
Do your students already have a good grasp on applying a traction splint from your skills sessions, then you can probably delete those slides from the presentation.
And of course if you need to bring their attention back, you can spice up the presentations.
We all know what can happen if we dim the lights and read word for word from the slides.
Some students will follow along and others will succumb to the fact that they stayed up too late last night. Especially right after lunch!

9. Publisher Test Banks
Publisher provided test banks serve primarily to determine if your students remember what they read in the textbook
The questions almost always refer to a particular page in the textbook
You need to test your students on what you taught them and not just what is in the book
Finally Publisher provided test banks can be very helpful. If you are in a hurry and need a test quickly, many of these will allow you to randomly select a number of questions and create an exam with the click of a few keys. Will it be a good exam? Will it tell you what your students really know? Possibly not.
If the purpose of the exam is to let you find out what your students know, presumably based on what you have taught them, then you need to put a little more effort in building your exams.

10. So What Do I Do?
To make your class more effective, you need to modify publisher materials or create your own
Go through the objectives and modify them to meet your needs. Add, delete, or edit objectives to fit your particular course
Create your own presentations or edit those provided to make them fit your course
Use other teaching techniques besides lecture!
OK so the publisher materials are not perfect, what should you do to make your class more effective?
Start by not reinventing the wheel. Use what the publisher provided, but make changes to it so that it fits your style and the needs of your class.
Publisher objectives are usually very good, but you may not feel the need to cover every one or you may need to add some. Do it.
Create your own presentations or at the very least modify the ones presented to make them fit your teaching style.
And of course, consider using other teaching techniques besides lecture. Flipping the classroom may be too radical for you, but an active classroom provides more learning opportunities than a passive one.

11. Can I write a good exam myself?
Finally create your own exams with questions you create to test what your students have learned
Start by asking what type of exam will this be
A short quiz to see if the students read their book
A longer exam covering several chapters worth of material
A final exam testing mastery of the entire course
The final step in making your course unique is writing your own exams. You can do it. Especially if you have made other changes we have talked about.
Start by asking yourself what you want to test and why.
Exams are given for many purposes so the first thing you need to decide is what is the purpose of this exam.
The purpose of the exam will determine how you proceed. Short quizzes over a reading assignment are usually pretty simple while comprehensive final exams are more complex.

12. Where do I start? Start by making a Blueprint for the exam
Identify what information will be tested
Select an exam length appropriate for the information and the purpose of the exam
Determine how many questions will be included from each section
Blueprints are a good place to start. What is a blueprint? It is an outline of what you will be testing.
Blueprints help you stay focused, and create exams that accurately test what you taught or intended to teach.
Example: Exam #1 will cover 4 chapters with anywhere from 10 to 30 objective each.
An appropriate length to see if your students have retained the information from these chapters might be 50 or 100 questions.
Select a percentage of questions to be selected from each chapter based on how many objectives are in each chapter.
Follow your blueprint as you create your exam.

13. Am I testing what I should?
Exams should test if the students can meet the course objectives
So each test question should be tied to an individual objective
Think about the objective as you write each question and make sure the question you write actually tests the objective
Now that you have a blueprint, you need to consider specific objectives you want to test (in each section).
It may not be necessary to have an exam question for every single objective or you may have multiple questions on a single objective.
But if your course goal is for the students to meet the objectives, then your exam should be based on the objectives.
If you can’t tie a question to an objective, then get rid of it or add another objective.

14. What question format is best?
There are many types of test questions
You need to choose the one (or ones) that best fit your situation (and/or your students)
Essay Questions
What are the advantages and disadvantages of Essay questions?
Now comes the interesting part. What types of questions do I want to use to best test the understanding of my students.
Here is an example of an essay question:
Essay questions are the best (possibly the gold standard) for measuring what students know (and are able to articulate).
They are often avoided because they are difficult to grade (primarily due to the subjectivity of grading these types of exams)
I learned early to look for key words or phrases in essays to prove they understood the material
If you use essay questions you need a rubric to follow to help you grade them but that is for another presentation

15. What about other types of questions?
Short Answer / Fill in the Blanks
Name one advantage of a short answer question.
Fill in the blank exams are good for testing ______ of presented materials.
True / False questions
Is this statement true or false:
True / false questions are a favorite of students because they are always clearly written and give students a 50/50 chance of getting the answer right.
Short answer questions are shorter than essay but still difficult to grade. You should have a list of acceptable answers prepared ahead.
Fill in the blank questions can be easier to grade but also generally best at testing recall – Fill in the blank grading still has some subjectivity especially if the student doesn’t use the exact correct word but one that can mean essentially the same. Here ‘recall’ would be keyed as correct, but ‘memory’ might also be considered correct.
True-False questions can be confusing and may result in students guessing even if they really know the material

16. Can I use Matching questions?
1. Matching questions make this easy:
2. Matching questions are poor at:
3. Matching questions are good for:
4. Matching questions are difficult in this respect:
a. Creating
b. Evaluating understanding
c. Grading
d. Testing recall
Matching questions as I discovered when making this slide are difficult to write well. I’m sure with practice I could get better, but this was a pain!
Matching questions make grading easy
They are poor at evaluation of understanding
They are good for testing recall of information
But they are difficult to create – trust me, this one was a pain

17. What about Multiple Choice?
Multiple Choice Questions
How many of you primarily or exclusively use multiple choice exams? – Why?
Multiple choice questions are:
A. The best choice for testing recall of information
B. Easy to write as a test of advanced understanding
C. Capable of evaluating higher levels of thinking
D. A poor choice for evaluating general knowledge
Finally we come to Multiple Choice. Why do you think we use MC questions so much
NR exam is written as MC and MC questions are much more objective to grade – can avoid conflicts
Testing recall can be done with matching or fill in the blanks just as well maybe better
I would not say that it is ‘easy’ to write questions that test advanced understanding – but it certainly is not impossible
MC question definitely ARE capable of evaluating higher thinking skills
And of course they are a good for measuring knowledge and understanding
Note how answer C is the best / most correct choice!

18. Writing Multiple Choice Questions
Since this is probably the most popular type of question used for EMS exams, let’s look at how to construct a good MC question
Multiple choice questions have three parts
A ‘Stem’
A ‘Correct answer’
Several ‘Distractors’ (incorrect answers)
If you haven’t figured it out yet, I am going to focus on writing Multiple Choice questions.
They can be a good format for testing student understanding of material but they can also be just a test of recall.
To be good they have to have all three parts written well.
The stem is the basic question or statement to which each of the answer choices relate.
The correct answer and the distractors are the options students must choose from to demonstrate their knowledge.

19. Multiple Choice Questions
Start with a Stem tied to one of the objectives being tested
Try to create a stem that requires more than just recall (memorization of facts)
Scenarios work well for these types of questions
Include enough information to choose the correct answer and possibly additional irrelevant information
Writing a good stem is critical to having a good question.
Make sure it is tied to an objective and read it out loud to make sure it makes sense.
Short scenarios work well as stems for EMS exam questions.
If you use a scenario (and these are not the only types of stems you could or should use) make sure you give the students enough information to figure out the correct answer.
I often include irrelevant material just to see if students can pick out the relevant information.
This also sometimes makes it easier to come up with distractors.

20. Multiple Choice Questions
Write the Correct Answer
Tie it to the objective and make sure that there is only one correct answer
Avoid ‘All of the Above’ and ‘None of the Above’
You can use ‘Both A and B’ type answers if you have enough choices (usually more than 4)
Consider writing a ‘feedback’ statement to explain why this answer is correct
Make sure there is only one correct answer or your students will point the other correct answers to you!
Try to avoid ‘all of the above’ and ‘none of the above’ since they dilute the options for most students.
You can use options such as “A and B” but in order to do this right, you need to have more than 4 or 5 answer choices.
Again it is better to just include 4 or 5 good options.
Also consider a feedback statement for the correct answer.
These are handy for use in computerized exams and they also help you remember later why you chose that answer as correct especially when students question it.
You might go so far as to include page numbers or presentation names and slide numbers.

21. Multiple Choice Questions
Now write the Distractors
Make each distractor plausible
Try to use answers that might be given by students who do not have a good grasp of the material
Make sure there is something in the distractor that makes it incorrect
Consider writing a ‘feedback statement’ telling why this answer is not correct
One of the most important parts of writing MC questions is good distractors.
How many of you have taken a MC exam where you can easily throw out 2 of the choices.
That is just a modified true/false question.
The best distractors are ones that a student might choose who is unclear – they sound good, but are not correct – for example in a scenario you might include an answer which would have similar signs and symptoms but which could not be the correct answer because of a particular bit of information in the stem. Think cardiac tamponade vs. tension pneumothorax.
Again consider including a feedback statement to explain why the answer is not correct.
Feedback can help you answer the question ‘Why did I get this wrong?’ question from students.

22. Multiple Choice Questions
Distractors (continued)
Make sure both distractors and correct answer sound correct (read each question out loud with each answer option)
Make all answer choices almost the same length
Shortest or longest answer choices often stand out
If necessary reword the stem to make the answers fit
Avoid ‘NOT’, ‘EXCEPT’, etc. in stems if possible
Once you have the correct answer and all of the distractors, read the question (stem) with each one of the choices to make sure they make grammatical sense.
At this point you might need to change the wording of your stem to make the distractors fit grammatically.
Try to keep all of the answer choices close to the same length. Often on exams an answer which is significantly shorter or longer is correct.
Try to avoid questions with wording like ‘all of the following except’ or ‘which of the following is not…’ because these can trip up even good students.

23. Multiple Choice Questions
Now consider alphabetizing the answer choices
Prevents the tendency to write 2 distractors, the right answer and then another distractor so the correct answer is most likely “C”
Include enough questions to test the objectives covered in the material
Somewhere within at least one exam there should be a question for each course objective
Once you have polished your answer choices, consider alphabetizing the choices. This helps offset the tendency to have 2 incorrect answers followed by the correct answer making the default correct answer “C”. If your choices are numbers then put them in numerical order.
Make sure to include enough questions to test the objectives. Theoretically you should have a question on an exam to cover every knowledge objective in your course.
In actual practice this may not necessarily happen, but if not make sure that you covered the objective somewhere within the course and consider documenting that information.

24. Building the Exam Use a variety of easy and difficult questions
Some recall, some higher level thinking
This will allow you to identify students at different levels of understanding
It will also allow you to anticipate the grades
Aim for a mean grade of 85% and half should be above and half below
If they score below a 70% they need remediation
When you are writing questions, remember that some questions need to be easy to boost the student’s confidence and keep their grades at a reasonable level.
Most students are comfortable with a system where is an A, is a B, is a C and usually anything lower than a 70 is not passing for EMS programs.
If you give an exam where all or most of the grades are extremely low, that means a) the students didn’t put forth effort or b) you didn’t teach them the material or c) you wrote a poor exam
Your exam should be related to your course and test what you covered. A test that asks about things not covered in the class is unfair to students.
Most schools grading systems would allow you to use a Bell curve model with an 85 as the mean and a high of 100 and a low of 70
If you structure your exam this way, anyone who scores below 70 needs remediation.

25. Testing the Exam When you finish your exam, test it out
Give the exam to your fellow program instructors
See what grades they make & listen to their feedback
Don’t expect their grades to be perfect – they didn’t sit through your class
Consider also giving the exam to a few recent graduates and listen to their feedback
When you finish this process, you will have your ‘perfect’ exam. Now it’s time to watch it be destroyed.
Give the exam to your fellow EMS Instructors and get their feedback. Don’t be surprised if they score poorly, they didn’t take your course.
See what problems they had and look at their grades. Don’t expect your students to make an 85 to 100 if the best your colleagues can do is a 65.
Another option (especially if you don’t have a lot of other instructors) is to give the exam to recent graduates of your program. Invite them in to take the exam and give you feedback. It will be good for them and it will be very helpful to you.

26. Giving the Exam Finally you should be ready to give the exam
Consider giving computerized exams if possible
Allow about one minute per question
Listen to the student’s feedback afterwards
If there is another way to interpret your question, they will find it and point it out to you!
Make adjustments to the questions based on student feedback
Keep statistics on each question
Finally the time has come to give the exam.
Consider giving your exams on computer if you can. This helps prepare students somewhat for what they will see with NR testing.
If you have to set a time limit, or anticipate how long the exam might take, a good estimate is 1 minute per (multiple choice) question.
After the exam consider reviewing the exam with the students. If there is a different way to interpret a question, trust me, they will find it.
When appropriate make adjustments to the questions for that exam, otherwise consider making changes to the question before the next time you use it.
If possible keep statistics on your questions. This can help if you need to make another exam in the future.

27. Evaluating the Exam
Each time before you teach the class review your exams and make adjustments
Keep your own ‘test bank’ of questions in case you need to create a ‘make-up’ exam
It will also allow you to use different exams each time you teach the class
Always be open to suggestions for improvement
Each time you are ready to teach a class, make sure you go over the syllabus and update it, then go over the exams.
Create new exams each time you teach a class once you have developed a test bank of your own questions.
You can also use this test bank for make-up exams if necessary.
Be open to suggestions for improvement and always look for ways to make your classes better

28. Conclusion
Publisher provided materials are a great starting point, but don’t rely on them exclusively for teaching your class
Multiple choice questions can make great exams so practice making them whenever you can
Constantly reevaluate your courses and strive to make them better. The life your student may eventually save may be yours!
In conclusion, publisher materials are great, but should not be relied on exclusively.
Multiple choice questions are fun and challenging to write, but it is a skill you need to develop.
Always be open to suggestions for improvement from you students, your colleagues, and your boss.

29. Selected References
National EMS Education Agenda for the future:
Cason, D. McKenna K. Foundations of Education: An EMS Approach 2nd edition Clifton Park, New York; Delmar Cengage Learning (2013) Print
My Contact Information:
Office phone
Here are a couple of sources I use and my contact number.
I would be happy to answer any questions you might have now or later via or phone. Thanks.