Short Answer Open-Ended versus Multiple-Choice Questions: A Comparison of Objectivity

INTRODUCTION

Over the last few decades in India, there has been substantial re-evaluation in the undergraduate medical curriculum, especially on the teaching and assessment methodology. Assessment is a means of measuring knowledge and competence, has a pivotal role in stimulating learning as well as providing feedback to students and teachers(1). Written assessment is an integral part of medical education. Subjective assessment in theory is gradually giving way to objectivity; longer essay-type questions (LAQs) are being replaced by Short Answer Questions (SAQs) and Multiple Choice Questions (MCQs) (2). There have been much debate with respect to the type of written assessment to be administered to the students in order to test higher order cognition(3).

An assessment has to be reliable and valid, free from bias and manipulations (1,2). Reliability is the degree to which an assessment tool produces stable and consistent results. Validity refers to how well a test measures what it is purported to measure (4). Open- ended format in the form of long-essay type, mini-essay type and short answer type questions is still preferred format for summative assessment. It is well documented that open ended questions have greater validity and they test higher order cognition level of knowledge (3, 5,6). On the other hand, closed-ended or multiple choice format is shown to be reliable and efficient, objective, unbiased and make linguistic skills redundant. However MCQs can provide the unprepared students an opportunity to score if they guess right. One of the disadvantages of open ended questions is its low objectivity and reliability (7, 8).

Physiology as a discipline, creates a framework for understanding the normal functioning of the human body. It is a concept based subject and therefore has a scope for testing higher order cognitive skills with conceptual questions which, though open ended, have a precise answer. We designed our study with the hypothesis that open ended SAQs, no matter how carefully framed, cannot be as objective as MCQs.

Methods:

This Cross-sectional study was conducted on 1st year MBBS students (n=99) of All India Institute of Medical Sciences, Jodhpur in the department of Physiology, in 2014. A class test from 'Blood and Immune System' was declared about 2 weeks prior to the commencement of the study.

The following format of the test was decided. The students were not told about the type/format of the questions (Fig.1).

Close

Fig. 1:

Algorithm of topic and questions in Section A and B.

Export to PPT

Both the sections comprised same questions but in different formats; MCQs and SAQs. Nevertheless, the questions were prepared by two different examiners. Marks for all questions in both sections were equal. Section B paper was distributed after collecting section A answer sheets. All the answers of section B were evaluated separately by two different examiners to minimize the level of subjectivity. A model answer sheet for both sections was prepared and provided to both the examiners. One of the examiners was the one who had taught the topic to these students during their routine teaching sessions.

There was no negative marking in either section of the question paper.

Table 1 shows the questions in both the formats.

Section A consisted of Assertion-Reason (Type-E) multiple choice questions with following 5 options.

1. Both A and R are true and R is the correct explanation of A.

2. Both A and R are true but R is NOT the correct explanation of A.

3. A is true but R is false.

4. A is false but R is true.

5. Both A and R are false.

Section B composed same concept based questions in SAQ format (explain why?)

Question on identical subtopics in both the formats :

Statistical analysis was done using statistical packages; SPSS version 21 and Graph Pad Prism version 6. Data on students' performance is expressed in percent Mean ± SD. Unpaired t-test was applied to check inter-examiner bias. The mean and standard deviation, median and interquartile range of marks was calculated and scores in the 2 methods were compared using Wilcoxon Signed Rank test. Regression analysis was performed to see the association of the two scores. Spearman's correlation coefficient (r) was calculated to see the correlation between students' performance on short answer questions and the MCQs. Bland Altman analysis of differences and averages between SAQ and MCQ scores was done (9). Mean bias and limits of agreement (mean bias ± 1.96 times SD) were computed.

Close

Fig. 2 :

Scatter plot of the marks obtained in MCQ versus SAQ. Also shown is the linear best-fit line (Spearman r = 0.99, p<0.0001).

Export to PPT

Results:

Mean marks awarded by examiner-1 and examiner-2 in all 12 questions in section B (open ended questions) were 14.44 ± 2.61 and 14.69 ± 2.82, respectively, the difference was not statistically significant. Mean of the marks awarded by the two examiners was taken as the final score of each student in section B. Table 2 depicts the percent marks (mean & median) scored by the students in section A and B. The difference in the scores was non-significant (p=0.14). A significant correlation (r=0.99, p<0.0001) was found in SAQ and MCQ scores (Fig.2, Table 2).

There was no significant difference in the marks scored by the students in the 2 sections. Significant correlation was observed in the 2 scores.

Figure 3 shows the Bland Altman Plot for the differences and averages between marks scored in SAQ and MCQ, it is showing considerable agreement between the two methods of assessment (non-significant mean difference of 0. 058), regression test showed non-significant results suggesting no proportion of bias (t=-0.48, p=0.63).

Close

Fig.3:

Bland Altman Plot for the differences and averages between marks scored in SAQ and MCQ depicting considerable agreement between the two methods of assessment

Export to PPT

Discussion:

Our study was designed with the hypothesis that small answer type open ended questions cannot be as objective as MCQs. In our study, we assessed the students by both open ended and closed ended questions on the same sub-topics. Contrary to our hypothesis, it was found that there was not much difference in the average scores obtained by the students, giving the impression that both the format are comparable in efficacy. Significant positive correlation was found between students' performance on MCQs and SAQs, emphasizing the fact that students who are bright, performed equally good in both the format. These findings were comparable to the findings by Dagogo J Pepple et al (10) who compared MCQs scoring with long essay questions. Similarly positive correlation between student performance on MCQ and short essay questions was observed by Mujeeb et al in Pharmacology. However, the correlation was not seen in the scores of the students who either failed or scored a distinction in the subject (11). We didn't correlate the marks in the two format on the basis of students' level of performance.

The Bland Altman plot for the measurement of agreement investigates any possible relationship between measurement error and true value. Bland Altman/Tukey mean-difference plot of the marks obtained by the two methods are showing agreement between the two methods.

A written assessment method should be reliable, valid, cost effective and acceptable (1). There always have been a debate on type of questions most suited for a reliable assessment. Open ended questions are more pliable, requiring creativity, spontaneity—but they have lower reliability (1,3). Answering open ended questions is much more time consuming than answering multiple choice questions, hence they are less suitable for broad sampling. They are also expensive to produce and to score. Since the students have to frame the answers spontaneously, open ended questions are believed to be suitable to test the ability to solve medical problems(4,5,11). 'Subjected to examiners bias or lack of inter-examiner reliability' is probably the most recognisable disadvantage of such questions(7,8). But in our study we have shown that if meticulously framed, open ended questions can be as objective as MCQs, free from other pitfalls of open ended questions as well.

Multiple choice questions are well known, and there is extensive experience worldwide in constructing them. Their main advantage is the high reliability per hour of testing—mainly because they are quick to answer—so a broad domain can be covered, free from examiners bias (1,3). However a common prevailing misconception about MCQs is that they are not suitable for testing the ability to solve medical problems (12). The reason behind this assumption is that all a student has to do in a multiple choice question is recognise the correct answer and the belief that MCQs test just the factual recall (13).

This was disproved by Palmer EJ and Devitt (14) who showed that the percentage of questions testing factual recall is same in MCQs as that of modified essay questions (MEQs) (14). Recently Moeen-uz-Zafar & Badr Aljarallah concluded in their study that a well- constructed MCQ is superior to MEQ in testing higher cognitive skills (15). If constructed well, multiple choice questions can test much more than simple facts as shown in the MCQs framed by us (assertion-reasoning type). This finding was supported by Hift who debated that MCQ format is better than open ended question format and suggested phasing out of open ended question format in summative assessment (16). Even if corrected for random guessing component, MCQs may overestimate some group of students and underestimate others (17). This was disproved in our study where the score in the two methods were comparable.

Research has repeatedly shown significant differences in the scores of the students with variation in the question's format (9, 16-18). However, others have proved that the question's format is of limited importance and that it is the content of the question that determines almost totally what the question tests (19,20). Every format has its advantages and disadvantages and a combination of different format is thus essential in assessing the students' performance (21). We too agree that though a well-designed testing format does not affect the performance of the students but assessment programme should include different type of questions appropriate for the content being tested. At different stage during the course covering the subject curriculum.

Conclusion:

It is a well-known fact that MCQs are the most objective method of assessment testing higher order cognitive skills and several topics can be simultaneously covered through these questions. However, cheating and guessing component are the common drawbacks associated with this format. In this preliminary study, our hypothesis that open ended short answer type questions cannot be as objective as multiple choice type questions was proven wrong. The results suggest that meticulously-framed open-ended short answer type questions can be as objective as multiple choice questions (MCQs). Further they have short precise answers, so multiple topics can be covered and chances of guessing the answers are negligible.