logo Setting Exams

Notes on the awesome discussion about setting exams, November 25th, 2008.  
OMIGOD - BrownBag Meetings are, like, so totally worth coming to!

Attendees: 5 professors, 3 STLFs

Brett Gilley facilitated this discussion by raising several questions. Discussion revolved around answers to those questions. These questions are numbered, and points raised are listed beneath each question. (By the way, one participant mentioned being there "... because a student said 'you should go'".)

Now, here are issues suggested by Brett, and discussion points raised in response during our 1-hour discussion.

  1. "What frustrates you about setting exams"?
    1. Time to write good Qs
    2. Bombproofing Qs
    3. Constrataints in large classes
    4. Continually having to generate new questions
    5. Writing higher level Qs
    6. Balance between good Qs yet easy to mark
  2. What's the exam for?
    1. Gives information about what students have learned, to be used to update the next version of course
    2. Simply grading (no feedback to students). This is "evaluative assessment" (as distinct from "formative assessment" which includes providing feedback to students).
  3. What types of questions have you used?
    1. Short answer
    2. Multiple parts to aid marking, and to help give "part" marks. But these can be a pain to mark if errors propagate.
    3. Use or make figures or charts. These can be easier to mark
    4. "Fill in the blanks" with tables to structure the question. These too can be easier to mark. They are similar to "fill in the blanks" type questions.
    5. short essays - half page (up to 3 or 4). Long to mark
    6. equations - how much to give - what's the cut off. But no definitions of parameters.
    7. MLB: Bring 3x5 cards in. Errors were on one!
  4. Any unique ideas?
    1. Consider reproducing some of the types of questions used in online assessment systems such as Vista. These include ...
      1. Calculated
      2. Fill in the Blank (one or more blank spaces within the question) - regular expression marking.
      3. Jumbled Sentence or ranking questions
      4. Matching
      5. Multiple Choice (with (almost) as many options as you like)
      6. Short Answer (words or a sentence or two) - regular expression marking.
      7. True/False
      8. Combination (Students are first presented with a list of answers, and then they are asked to choose the correct combination of these answers is correct).
      9. Paragraph (i.e. essays)
    2. Use of a pretest at the beginning of the term as part of the final exam.
    3. Oral (student's must be used to this kind of interaction with a professor)
    4. Use of slides
    5. Teams in exams, and various permutations. Again, students need to be familiar with working in this way. Design of good questions is the challenging part.
    6. Essays - see "One important perspective" above.
    7. Take home exams (Similar to online exams?) One participant did a huge number of these at all levels. They were usually opportunities for significant number crunching.
    8. Giving choice on exams (eg: "answer 7 of the 10 questions ...") (Some pros and cons to this approach include harder to mark, time taken to make a choice, and others).
    9. Posing exam questions. One way for students to practice this in the course involves using index cards, and specific Bloom's learning level as a target. The exercise could even insist on use of specific verbs.
    10. Have students pose questions based on experiences during class. Then allow these questions to become essay questions in a "personalized" exam.
    11. Ask students to make up a question (and answer it) about something they did not find on the exam. Again, it is probably lgood to allow them to practice doing this.
    12. Essays - do they get practice at this form of communicating understanding? When is use of point form acceptible? (See one perspective above.)
    13. Ask students what they think they migh deserve on the test.
  5. Is there anything you do to help you mark:
    1. Isolate for the whole period.
    2. Self-bribary (beer / rum / chocolate after completing 3-4 questions for all students ... etc.)
    3. Mark one question at a time.
    4. Do it at a big table - spread them all out in order of ability.
    5. Coordinate TAs carefully. Also TAs do computations and low level questions while instructor does "explain" type questions.
    6. Read a sampling before tackling the whole bunch.
    7. Have an "exam testing" session with a bunch of TAs and/or colleagues. This helps weed out less well-prepared questions.
  6. One further question was "How soon before I can reuse a question in subsequent terms?"
    1. Do we worry about "stolen" exams?
    2. Some instructors end up keeping a "bank" of preferred questions and then pick / choose for specific exams.
    3. Questions do inevitably get out into the big wide world, but in the end, it will only be a few that try to "cheat" by studying past exams exclusively. Most participants agreed that even questions that are re-used often tend to generate a consisten pattern of answering ability. Therefore the "leaked questions" issue is probably (though not certainly) a minor concern.
    4. Don't forget that it is very hard and time consuming generating new questions every time there is an exam. There is in the end a limited number of ways to assess certain concepts, and if there is a good question that seems to "work", it is probably good to use it more than once.

One important perspective to always keep in mind is: always be very clear in your own mind about what exactly you are tryiug to assess. For example, if you expect excellent communication and writing skills to be demonstrated in an essay you set that is all about explaining a concept or working through a problem, then students must know that "communication" is one of the learning goals of the course. This point is particularly important when there are significant number of ESL studnets (English as a Second Language). The exam is about assessing capabilities or "degree of mastery". If the student is prevented from demonstrating their ability by the medium (writing, drawing, whatever), then you have done them a disservice. That is NOT to say communication skills are UNimportant. It simply means that students need to know what kinds of skills are being tested.

An implication of this important perspective is that "transfer" is a special type of skill. Transfer refers to the ability to apply new knowledge / skills in a new setting. This is a totally ligitimate aim for a course. But it should never be tested by surprise. If you ask students to carry out a significant "transfer" tasks in an exam setting, they should have had chances to practice that skill during the course.

Finally, another implication concerns the speed at which students must work. Expecting students to demonstrate their abilities under pressure of time limits is totally acceptible, but it is DIFFERENT from asking them to demonstrate their skills without time constraints. Therefore, if you plan to give an exam that is only just doable within the 2.5 or 3.0 hour time limit, students should have been given opportunities to practice working under pressure.

Please sent questions, comments, alternative perspectives, to any of the STLFs.