Guide to Writing Short Answer and Select-Response Items

 

 

Based on suggestions from your textbook and Haladyna (1999). Developing and validating multiple choice test items. Mahwah, NJ: LEA Associates and Measurement and Assessment in Teaching, Linn and Gronlund

 

 

Keep the test blueprint/spec  in front of you as write items.

 

 

Short Answer Items

 

Word the item so that the required answer is both brief and specific:

Poor: An animal that eats the flesh of other animals is _____________. 

Better:  An animal that eats the flesh of other animals is classified as  _____________. (carnivorous)

 

 

Do not use quotes taken from the book in short answer items:

Poor: Chlorine is a  _____________.

Better:  Chlorine belongs to a group of elements that combine with metals for form salts.  It is therefore called a   _____________. (halogen)

 

 

Use only one blank per item:

Poor: President _____________ emancipated the _______ in _____ .

 

 

Use direct questions:

Poor: John Glenn made his first orbital flight around the earth in  _____________. 

Better:  In what year did John Glenn make his first orbital flight around the earth? ________.  (1962)

 

 

Binary Choice (T/F)

 

 

Avoid items that use the words usually, sometimes, never, always, all, none:

Poor:   T  F     The president of the US is usually elected to that office.  

 

Avoid trivial statements. 

Poor:   T  F     The US declared war on Japan on December 7, 1941.  

 

 

Avoid long, complex sentences.  These items tend to test reading skills rather than content.

 

Avoid using opinion not attributed to some source.

Poor:     T  F     Adequate medical care can best be provided through socialized medicine

Better:   T  F     The American Medical Association favors socialized medicine as the best way to provide adequate medical care.  

 

 

Matching Items

 

 

Use only homogeneous material in a single matching exercise:

 

Include an unequal number of responses and premises, and instruct the student that responses may be used once, more than once, or not at all.  

 

Keep the items to be matched brief, and place the shorter responses on the right.   

 

e.g., 

 

___  1.  Boston Tea Party A.  1765-1769
___  2.  Battle of Lexington B.  1775-1779
___  3.  Repeal of the Stamp Act  
___  4.  First Continental Congress  

 

Indicate in the directions the basis for matching the responses and premises.  

 

 

Multiple Choice Items

 

 

The "stem" of the item should be meaningful by itself and should present a definite problem:

Poor: 

1.  South America

  1. is a flat, arid continent

  2. imports coffee from the US

  3. has a population less then Australia

  4. was settled mainly by colonists from Spain.

 

Better:  Most of South America was settled by colonists from 

 

  1. England

  2. France

  3. Australia

  4. Spain.* 

 

The "stem" of the item should include as much of the item as possible and should be free from irrelevant material:

Good: 

1.  Spanish colonists settled most of South America in search of

  1. adventure

  2. wealth

  3. lower taxes

  4. religious freedom

 

Avoid negatively stated stems

 

Poor: 

1.  Which of the following states is NOT located north of the Mason-Dixon line

  1. Maine

  2. New York

  3. Penn

  4. Virginia *

 

All of the alternatives should be grammatically consistent with the stem of the item.

 

Good: 

1.  An electric transformer can be used to

  1. store electricity *

  2. increase the voltage of alternating current

  3. convert electrical energy into mechanical energy

  4. change alternating current to direct current

 

Items used to measure understanding should contain moderate novelty.

 

 

All of the distracters should be plausible.

 

Poor: 

1.  Which explorer first landed at the North Pole?

  1. Roberty Peary *

  2. Robert Peebrain

  3. Your teacher

  4. Marco Polo

 

The relative length of the alternatives should not provide a clue to the answer

 

"All of the above" and "None of the above" should be used sparingly  .

 

 

 

More Suggestions

 

Keep the specific content of items independent from one another.

 

The following questions come from the story "Stones from Ybarra."

 

Who was Lupe's best friend?

Kate

Dolores

Roxie*

 

Who was always quarreling with Lupe?

Kate

*Sara

Roxie

 

It is easy to see that Roxie can't be correct in both items.

 

 

Avoid overly specific and overly general knowledge.

Who wrote the Icon of Seville?

Lorca

Ibanez

Rodriguez

 

Avoid opinion-based items.

 

Unqualified open item:

What is the best comedy film ever made?

Abbott and Costello go to Mars

Young Frankenstein

A Day at the Races

 

Qualified opinion item:

According to American Film Institution, what is the greatest American film?

It Happened One Night

Citizen Cane

Gone With the Wind

Star Wars

 

Avoid Trick Items

Is there a fourth of July in England?

Some months have 31 days. How many have 28?

How many animals of each gender did Moses bring on his ship?

A butcher in the meat market is 6 feet tall. What does he weigh?

 

Avoid complex multiple-choice formats.

Which of the following words describes you?

1. Kind

2. Considerate

3. Easy on the eyes

1 only

2 only

2 and 3

1,2,3

 

Keep your tests as pure a measure of achievement as possible. .

Format the items vertically instead of horizontally.

Below is a vertically formatted item:

You draw a card from a deck of 52 cards. What is the chance you will draw a card with an odd number on it?

36/52

32/52

About one half

Here is an item that is formatted horizontally:

You draw a card from a deck of 52 cards. What is the chance you will draw a card with an odd number on it?

A. 36/52 B. 1/2 C. 1/3 D. Not given

 

 

Edit and proof items.

An example in the question format is:

Which retirement investment is most recommended by investment counselors for young investors?

U.S. Bonds

Certificate of Deposits

C.* Fast growth stock funds

An example in the partial-sentence (completion) format is:

Regarding aging people, in other modern societies,

Programs for the elderly are generally a low priority item.

Programs for the elderly are more advanced when compared to the United States.

Programs for the elderly are about the same as in the United States.

The elderly have been shown to have different needs, based on cultural differences.

 

 

Don’t use these "fill in the blank" type items.

The __________ format is the best way to format a MC item.

completion

question

fill-in-the-blank

 

 

Ensure that the directions in the stem are very clear.

Unclear directions in the stem:

Regarding gravitation:

Clearer directions in the stem:

Which of the following best shows the concept of gravitation?

 

 

Avoid Window Dressing (Excessive Verbiage).

 

Poor:

High temperatures and heavy rainfall characterize a humid climate. People in this kind of climate usually complain of heavy perspiration, even moderately warm days seem uncomfortable. Which climate is described?

Savanna

*Tropical rainforest

Tundra

 

Better:

Which term below describes a climate with high temperatures and heavy rainfall?

Savanna

Tropical rainforest

Tundra

 

 

Word the stem positively; Avoid negative phrasing when possible

Which is NOT an advantage of the MC format over the CR format for measuring knowledge?

Better sampling of content

Greater reliability of test scores

Ease of construction

Which are benefits of MC testing? Mark A if true or B if false.

Greater potential for content sampling (A)

Greater potential for high reliability (A)

More appropriate for measuring aspects of performance (B)

Objectively scored (A)

Able to measure higher level thinking (A)

Subject to guessing (A)

Free of bias (B)

Use as many distractors as possible but three seems to be a natural limit.

Make sure that only one of these choices is the right answer.

Vary the location of the right answer according tot he number of options.

 

 

Place options in logical or numerical order.

 

Wrong:

What is the cost of an item that normally sells for $9.99 that is discounted 25%?

$5.00

*$7.50

$2.50

$6.66

 

Right:

What is the cost of an item that normally sells for $9.99 that is discounted 25%?

$2.50

$5.00

$6.66

*$7.50

 

Keep choices independent; Choices should be mutually exclusive.

Wrong:

What age range represents the physical "peak" of life?

11-15

15-19

19-25

25-32

 

 

Avoid using the choices None of the Above, All of the Above.  

 

Phrase choices positively.

 

Avoid various clues to the right answer, such as 

Clang associations

Who were known as the Magnificent Seven?

A touring softball team.

*A group of seven cowboys.

A rock and roll group.

 

Grammatical inconsistencies

The most effective strategy in playing winning tennis for a beginner is:

hit with more pace.

*to keep the ball in play.

volley at the net as often as possible.

 

Blatantly absurd, ridiculous options.

Who is best known for contributions to electronics?

Comedian Jay Leno

*Robert Sveum

Actor Bruce Willis

Make all distractors plausible.

Use common errors of students.

Avoid the use of humor.