The Reeve Guildmaster's Art: Making Good Reeve's Tests
by Matthias Fleewinter
"Having a Good Eye": Test Elegance
While it may not seem that important at face value, a good reeve's test should be aesthetically pleasing and easy to browse. Any experienced publisher will tell you that using very small type or using an odd font or a font that is sans serif can interfere with the comprehension of the text. Most people don't notice such details consciously but having to "fight the text" can at times make reading and comprehension more difficult or more tedious. Aesthetically-pleasing tests don't have to have illuminated characters or incorporate pretty pictures unrelated to any of the questions, but a pleasant and legible text font more than does the job.
Additionally, the reeve's test should be written using good (or at least decent informal) grammar and in a style that will not confuse or mislead the average literate test-taker. If the test writer has very poor writing composition skills, it's his responsibility to find someone who can edit his test and proofread it for legibility. The average literate test-taker should not have to fight both his own ability to recall information and his inability to understand what each sentence means. I am not saying that a reeve's test necessarily has to be written in perfect English (or Spanish, German, Croatian, or whatever), nor does the punctuation have to be perfect, but the candidate should be able to guess exactly what information a given question is asking for, even if he can't answer the question. If he can't understand the very sentence structure or guess what a certain misspelled or missing word is supposed to be, it is an unfair test.
A good test should also have a structure which changes pace occasionally. Everyone can attest to having to trudge through an endless series of multiple-choice or true-false questions, losing their place, succumbing to periods of "zoning out", so to speak. A good reeve's test should change its tempo once in a while so that the test-taker can properly maintain their concentration. The best way to do this is to vary the type of questions being offered, changing up the format and thereby refreshing the mind of the reader when he has to deal with the next series of questions written in a different format.
There are several common types of questions which most individuals will recognize immediately. There are advantages and disadvantages to each:
1) True/False. This is the most simple type of question, and sometimes the most insidious because they are easiest to use as "trick questions". However, their chief disadvantage is that a test-taker guessing randomly will guess the right answer 50% of the time. A test that is 50% True/False questions and 50% everything else can open the door to candidates passing the test by randomly guessing on the True/False section, and getting a perfect score on everything else.
2) Multiple-Choice. This is probably the second-simplest question type, since there are a finite number of answers to choose from. Multiple-choice questions make it more difficult to improve one's score by random guessing, but an intelligent candidate can often use logic to "outthink" the test and find the right answer if they weren't otherwise certain of it to begin with, by reverse-engineering the question. Because of this, it is always a good idea to include a "none of the above" answer for each multiple-choice question to prevent the test-taker from finding the answer by "out-thinking" the questions. The main advantage of this type of question is how quickly the test-giver can grade the test, with a secondary one of not having to decipher bad penmanship very much.
3) Matching: This is a variation on the multiple-choice question type, the difference being that each group of questions uses the same set of possible answers. It is by no means required that every choice per group be used, nor should a given choice be necessarily used only once. A group of matching-type questions with a one-to-one ratio of possible answers is fairly easy to reverse-engineer, where each question has a single corresponding answer and there are no extra answers. In such cases, a test-taker can skip a question and return to it, and be able to give the right answer simply because it was among the ones left over. The key to making it hard for the test-taker to "outthink" the test in this fashion is to include several unused answers similar to some of the right ones, and to include some questions that will reuse one or more of the possible answers. That way, no one-to-one ratio exists.
4) Short Answer: This type of question is good because it gives nothing away to the reeve candidate about the answer to the question, other than what is written in the question itself. There is no list of potential answers that can be used to "out-think" the test and discover the right answer. Probably the biggest disadvantage of this question type (and not a big one) is that the test-giver may have to deal with illegible writing, or (if the questions are written onto the test itself) include space for the answer to be written. Short-answer questions are fairly quick to grade, as well. This is usually the best type of question to use (although one should not write a test with only short-answer questions since that would become monotonous).
5) Fill-in-the-Blank: This is basically a variation of the short-answer type, but requires multiple answers. As a side note, it isn't a good idea to use fill-in-the-blank questions to quote the rulebook directly since there are some passages which Amtgarders can recite by heart because they've read them often enough to have memorized them. A reeve's test is not meant to test a candidate's ability to recall information by rote, but to test his comprehension of the rules. To be sure, some memorization is required, but so is understanding how the rules mesh together and how to apply them to real-world situations.
6) Listing: Listing questions are quite useful, as much as short-answer questions, since no clues or hints need be given away for free. Listing questions usually require the candidate to be familiar with several different parts of the rulebook in order to give all the answers required. For example, answering the question "Which classes can use javelins?" requires that the test-taker be able to answer 'yes' or 'no' when he asks himself about each class in turn. Extrapolating from what he remembers about the allowed weapons of each class, he can then draw up a list of which ones are allowed to use them. Next to essay questions, these tend to be the most tedious.
7) Essay: This question type, while popular with college professors, does not really belong on a reeve's test. This type of question requires a lot of effort on the part of both the candidate and the test-giver, one to answer it and the other to grade it. On a test written to certify players as referees for a game of all things, essay questions go a little overboard, except perhaps in extraordinary circumstances such as a reeve's test given to a person who needs to qualify for a kingdom-level office. Yet even on kingdom-level reeve's tests, they should be used conservatively and possibly only for extra credit.
A good reeve's test that varies its pace might have (in order) ten true/false questions, five multiple-choice questions, a number of listing questions totalling fifteen answers, ten short-answer questions, a group of ten matching questions, and ten more true/false questions. Sixty answers in all, a nice and round number making a passing grade (75%) exactly forty-five items to be answered correctly.