How to avoid ambiguity in setting questions for your students

My favourite Christmas gift this year was from my Father-in-Law. Knowing my professional interest in assessment, he bought me the ‘F in Exams’ desk calendar. For those not familiar with the concept, this gives a daily example of an amusing candidate response to a test question. For example, the question that asked candidates, ‘Where was the Declaration of Independence signed?’ Answer: ‘At the bottom’.

I shared these with colleagues and we all had a chuckle. But the more of them I read, the less funny they became. After all, the Declaration of Independence was signed at the bottom. Even if they were being facetious, they weren’t wrong. Should they have received a mark? Most of the answers we were laughing at had been prompted by questions that were ambiguously written and, if real, represented young people who had been failed by the test they took, not just young people who had failed a test.

In my previous role as a test developer, I spent seemingly endless hours poring over exam scripts from trials. As a specialist in developing reading comprehension tests, I have found that candidates often don’t respond in the ways we expect them to. They sometimes give far more imaginative, but no less valid, responses than the writer of the test could have anticipated. Here are some of things I have learned.

• The more eyes the better. What is clear to me may not be clear to you. The more people you can show something to, the more chance you have of universal comprehension. An example might be the question, ‘Why have ellipses been used in this sentence?’ I wanted answers around creating a sense of awe, disbelief and mounting excitement. I got technical answers, such as ‘to create a pause’. I should have asked ‘What is the effect on the reader of the ellipses in this sentence?’ It wasn’t until I read candidate responses that I realised my mistake.
• Knowing what is wrong can be as important as knowing what is right. There will always be spaces between answers which we know are correct, and answers which we know are wrong: this is the ‘zone of uncertainty’. But we should endeavour to reduce this to the smallest possible space. Let’s use an example from a well-known fairy story.

Question:  How does Snow White get awakened from her sleep?

Answer: Her True Love’s Kiss.

So are the following answers right, or wrong?

a) She is kissed (no True Love)

b) She gets kissed by the Handsome Prince (is this enough for True Love?)

c) Her True Love wakes her (no kiss)

Are we sure these are incorrect?  If we are, then listing some common incorrect answers might help markers not to be tempted by answers that are not quite accurate enough.  If we think these are correct, we might adjust the answer given in the mark scheme to ‘A Kiss’ and/or ‘Her True Love / The Prince’.

• Don’t make assumptions about the test taker. I made an assumption just now by assuming familiarity with the fairy story of Snow White, but I shouldn’t have. There’s no reason why everyone in the world should know the story, and even less reason why everyone should know the Disney version. In the Grimm Brothers’ version, the Prince is in the process of having the comatose beauty carried off back to his palace by servants when one stumbles and the resulting movement dislodges the bit of poisoned apple stuck in her throat. What if I only knew that version, and was confronted by the Disney tale as an unseen text in an exam setting? Would I be unnecessarily distracted? And if I am already familiar with the story, will I read less carefully, assuming I already know all the answers? Making assumptions about the range of somebody’s experience can have unintended side effects for the test taker.

I think about the number of times I say, ‘that’s not what I meant’, when I’ve been misunderstood, misinterpreted, taken out of context… but how much of that is the fault of the receiver, and how much of the blame lies with me? Could I have been clearer? Could I have communicated what I meant more effectively than I did? Perhaps I need to practice what I preach in my wider life as well as in my work!

But let’s end on an example of a candidate who has reflected carefully on a question and used it to acknowledge their own shortcomings.

Question: What is a vacuum?

Answer: Something my mum says I should do more often.