Biomedical Research

Abstract

Introduction: Clinical reasoning skills play a major role in the ability of physicians to make diagnosis and take an appropriate intervention for treatment of patients. The aim of this research is the comparison between the key features examinations and common multiple choice examinations for assessing the individuals’ clinical decision makings ability in obstetrics and gynaecology department.

Methods: In this study, 59 midwifery students were participated. After each multiple choice examination, the Key features examination was held. The correlation between these two examinations was analysed. The reliability of the Key features examination was measured by the Cronbach’s alpha values. By means of the Whitney and Sabers method, difficulty index of questions was determined; and correlation of every question with the total examination was investigated by using SPSS software. Satisfaction of clinical reasoning assessment in top students was compared with weak students by using LIKERT scale.

Results: The mean score in the key points’ examination was 10.19 out of 20 and in multiple choice examination was 8.88, there was a significant difference between these two exams’ outcomes (P=0.005). The reliability of the key points’ examination was 0.75. The correlation every question with the total examination was 0.29-0.82, and the difficulty index was 0.43-0.77. The discrimination index was 0.31-0.63. The mean correlation between the key feature and routine multiple choice questions was 0.49 (P=0.001).

Conclusion: The difficulty index, the discrimination index and the Cronbach’s alpha values suggested that clinical reasoning cannot be evaluated by the multiple choice examinations; therefore the use of the novel evaluation tools in the clinical field such as key features examination was suggested.

Keywords

Clinical reasoning, Multiple choice questions, Key point examination, Midwifery, Students.

Introduction

Clinical reasoning is a crucial component of physician ability to distinguish the problem, and it plays a major role in the ability of physicians to make a diagnosis and exact treatment decisions [1]. Despite various usages of clinical reasoning, it is agreed that clinical reasoning in nursing practice have a critical importance [2]. It has been cleared that graduate nurses may lack the clinical reasoning skills to take a safe and effective intervention for the patient’s health [3,4]. By considering such this concern, in undergraduate nursing programs, the development of students' clinical reasoning should be considered essential. There some literature about the development of models, demonstrating the clinical reasoning process [5-7], but there isn’t enough publication about nursing students’ clinical reasoning during their clinical placements. The process of decision making includes problem recognition and values analysis (choosing from a set of alternatives depending on their probability) [8]. Experienced practitioners possess “scripts” which is an elaborated networks of knowledge fitted to the regular tasks they perform [9]. The, based on cognitive psychology script theory [9], may provide a more proper way to build a theory-based tool to assess decision making skills such as those in the intra-operative setting. In the other words, script concordance approach provides us to measure the concordance between the examinees’ answers and a panel of experts’ judgments [10]. Studies have confirmed the reliability, validity, and linearly relation of script concordance approach to experience [11-14].

Schmidt et al. concluded that with increasing experience, practitioners move through three kinds of mental representations: basic mechanisms of disease, illness scripts, and a bank of cases which are derived from experience [15]. Clinical medicine is affected by illness problems that clinicians solve them with a variety of solutions, depending on their experience [16]. Barrows et al. assumed that expert practitioners have a bank of “illness scripts” consisting of a story-like narration of a typical case of the condition that can be used when confronted with a new case [17]. In addition, it has been assumed that experienced practitioners and clinicians have mental probability matrices that they use in conjunction with illness scripts to perform the right and appropriate diagnosis [18].

Unfortunately, medical sciences education system of Iran, lacks a proper examination module; there is vast emphasize on memorization, the ability of reasoning and problem solving, has been forgotten [19]. The proper problem solving ability is essential for clinician. Clinical reasoning is a thinking process which guides the physician to take wise full and purposeful steps toward the recognition and treatment of the problems [20]. Incorrect problem solution ability of clinicians would result in treatment failures, and in some cases it would lead to the patient's death [1]. Although, few studies have done on reliability and validity of clinical reasoning, the results illustrated an improved estimate of reliability and validity, especially confirming the usefulness of clinical reasoning as an indicator of the accuracy of the diagnostic reasoning [21].

Researches in other disciplines, except medical education such as educational psychology and cognitive psychology, would provide theoretical frameworks and evidence based learning strategies that could be combined with suggestions from research on medical expertise to design practical instructional approaches [22,23]. In order to overcome the difficulties, Page and Bordage [24] suggested that, in any clinical case, there would be a few" unique, essential" elements in decision making which, alone or in combination, would be the critical steps in the successful resolution of the clinical problems. These elements are labelled as key features which are critical step in the resolution of a problem [8]. This concept does lead to the creation of a new method of testing the ability of clinical decision-making skills, which named candidates’ responses concerning only the critical steps in the resolution of each problem-the problem’s key features. Testing only critical steps, would enable the practitioner candidates to be able to distinguish a much larger number of clinical problems [25]. This method was shown to have a potential reliability of 0.8 in 4 hours of testing [25].

The aim of this research is the comparison between the key features examinations and common multiple choice examinations for assessing the individuals’ clinical decision makings ability in obstetrics and gynecology department of Hamadan University of Medical Sciences.

Materials and Methods

Study design and setting

The study was conducted as a quasi-experiment at Hamadan University of Medical sciences, investigating the consistency level between the key features examination of clinical reasoning and multiple choice examinations.

Study sampling

Target population of the present study consisted of 100 senior midwifery students who participate in final apprenticeship course at Hamadan University of Medical sciences. Only data of 59 students were analysed and 41 students (10 of the student didn’t will to fill the form and 31 didn’t fully completed the form) were excluded from statistical analysis. The omitted participants didn’t answer the questions fully or they were absent. We gathered the statistical population using simple sampling method.

Key feature scoring

Due to the importance of the measurement of clinical reasoning, we decided to use the key features examination and the routine multiple choice examination as the most important examinations in this field for midwifery students. The difficulty index and the discrimination index, reliability and correlation between key features examination and routine multiple choice examination were calculated [25].

Development of key feature test

The test are designed by an expert committee with members of Gynaecologist of Hamadan medical school based on the targets of gynaecologic subjects focused on the output and efficiency of assistants Key feature Questions are evaluated and approved by reference panel, which consists of 10 gynaecologic professors from some Iranian medical schools; the key questions are also determined by reference panel. The papers are corrected by the mentioned committee members.

Data collection

With assistance of the gynaecology wing, the appropriate key features examination for comprehensive examination was provided. After each one-month courses, the comprehensive final exam of gynaecology was held in the format of multiple choices. We tried key features examinations for different courses (4 courses with one-month duration). Collected data was used to score the examinations; two methods were used: (I) Cumulative method, where the sum of scores would be reported as a total score of the examination. In this method, different questions based on the number of correct options and their weight, have different scores; and (II) Mean method, where the mean score of exams was considered as a total score. In this method, the weight of different questions would be same. In each examination, uses of different scoring methods for each question would be possible. Clinical reasoning skill was assessed before and after the interventions by means of key features examinations.

Data analysis

The descriptive tests (t-test) were used to analyse the key features examination. Mean, standard deviation, mode and analytic tests have been done. The correlation of the key features examination and multiple choice examinations were measured. By means of Cronbach’s alpha method the reliability of key features examinations were measured [26]. The difficulty index was determined by means of Whitney and Saber's method [27]; the correlation of each question with the total examination was investigated as a discrimination index.

Results

Results demonstrate that mean score of clinical reasoning tests of the participants in this study was 10.19 ± 3.69, while the multiple choice examinations’ mean score was 8.88 ± 3.05; based on the double T-test, there is a significant difference between these two exams’ outcomes (P=0.005). Clinical reasoning test showed more success compared to the multiple choice examinations. The mean, minimum and maximum of mode each score, standard deviations and variances are shown in the Table 1.

Table 1. Cross multiple choice test and key features results of participants.

Difficulty index for each question in clinical reasoning test was measured by means of Whitney and Sabers method (Table 2). Difficulty index for the first and second questions was acceptable. Difficulty index for multiple choice questions are also presented in Table 2. The level of clinical test satisfaction from the strong students' view (the top 10% of the participants) was compared to those from weak students' view (the lowest 10% of the participants). The level of students' satisfactory of clinical tests compared to multiple choice examinations was evaluated. The mean satisfaction was 2.78 ± 0.056. Moreover, based on the T-test evaluation, there was no significant difference in satisfaction values of strong and weak students (P-value of 0.163). The strong students' satisfaction mean value was 3.05 ± 0.13; while weak students showed similar mean value of 2.81 ± 0.53. The results of clinical reasoning test and participants' mean were compared with multiple choice examinations’ results in Table 3. Table 4 demonstrates each key feature question item discrimination which is calculated by Whitney and Sabers. In this study, Correlations between the total examination score and score for each item were calculated with Pearson correlation coefficient in Key Feature test (Table 5). The correlation between Key feature test results, multiple choice tests were measured. The Pearson correlation coefficient between the two tests was 0.49 (P=0.001). The reliability of both tests (Key Feature, MCQ) was measured by Cronbach's alpha, which was 0.75 for Key Feature test and 0.155 for MCQ test.

Table 2. The difficulty index of clinical reasoning and multiple choice examination tests using the method of whitney and sabers.

Table 3. The relationship of the results of clinical reasoning test and multiple choice tests with participants' mean.

Table 4. Discrimination item of key feature test questions.

Table 5. Correlation key feature tests.

Discussion

All medical science educational systems have the responsibility to shift the medical learners from the memorizing the materials to reasoning and creative problem solving. In this study the key points examination was used for students of obstetrics and gynaecology wing; this examination has been administered simultaneously with the final exam in the format of multiple choice. The results indicated that, the uses of the proportional and cumulative scoring method would increase the validity of the examination, but it seems that depending on the examination situations, questions types, and options, we can use different kind of these two scoring methods. In the key features examination method the proportional scoring method and mean scoring had the best results. In this study we used both methods. Results of the present study with acceptable reliability, difficulty index and the content validity showed that the key features examination would be acceptable for the assessment of the clinical reasoning skills. The reliability of the key features examination in the study is between 0.6 and 0.8. The comparison of this reliability with other studies indicated the accepted reliability of these examinations.

In the Martin Fisher's study, the reliability of the key points’ examinations was 0.65 which is similar to our obtained results [12]. In the present study the difficulty index of questions was between 0.43 and 0.77; the correlation coefficient of each question with the total examination for all questions were between 0.29 and 0.82, which can distinguish weak students from the strong students. In Martin Fisher’s study most students were satisfied with the patient-based examinations and consider them appropriate and optimal for education which is consistent with the results of the present study [12], and also discrimination index of each question, which was between 0.06 and 0.6 [13]. These results of Martin Fisher’s and Amini’s studies [28] were in good accordance with our output results. Key features examination can be used in a variety of testing situations. While the reliability of the format was good, in high stakes testing this format would be presented as part of assessment approaches. In any clinical case, there are a few essential elements in decision making which, alone or in combination, are considered as critical steps in the successful and efficient resolution of the clinical problems. The key features examination would lead to a new test of clinical decision-making skills, which elicited candidates’ responses concerning only the critical steps in the resolution of each problem.

Conclusion

It seems that the administration of clinical reasoning examination in this country, like others’ medical education systems, as a valid examination would be must. Key features examination with pre-training seminars would show better achievements in education than other existing examinations. It demonstrated test score reliability which is acceptable for making decisions about individual candidates’ clinical decision-making ability.

Acknowledgments

The present study is derived from a MS thesis of Medical Education student by funding of Shiraz University of Medical sciences. It has the agreement of Medical Ethics Committee of Shiraz University of Medical Sciences and is confirmed by the code 7290 in Vice Chancellor Department for Research of Shiraz University of Medical sciences. Authors would like to thank all participants for their effort and expenditure to the study.

References

1. Norman G. Research in clinical reasoning: past history and current trends. Med Educ 2005; 39: 418-427.
2. Simmons B. Clinical reasoning: concept analysis. J Adv Nurs 2010; 66: 1151-1158.
3. del Bueno D. A crisis in critical thinking. Nurs Educ Perspect 2005; 26: 278-282.
4. Killam LA, Luhanga F, Bakker D. Characteristics of unsafe undergraduate nursing students in clinical practice: an integrative literature review. J Nurs Educ 2011; 50:437-46.
5. Tanner CA. Thinking like a nurse: a research-based model of clinical judgment in nursing. J Nurs Educ 2006; 45: 204-211.
6. Lasater K. Clinical judgment development: using simulation to create an assessment rubric. J Nurs Educ 2007; 46: 496-503.
7. Levett-Jones T, Hoffman K, Dempsey J, Jeong SY-S, Noble D, Norton CA, Roche J, Hickey N. The ‘five rights’ of clinical reasoning: An educational model to enhance nursing students’ ability to identify and manage clinically ‘at risk’patients. Nurs edu tod 2010; 30:515-520.
8. Roth BM, Mullen JD. Decision making: Its logic and practice: Rowman & Littlefield, 2002.
9. Charlin B, Tardif J, Boshuizen HP. Scripts and medical diagnostic knowledge: theory and applications for clinical reasoning instruction and research. Acad Med 2000; 75: 182-190.
10. Charlin B, van der Vleuten C. Standardized assessment of reasoning in contexts of uncertainty: the script concordance approach. Eval Health Prof 2004; 27: 304-319.
11. Brailovsky C, Charlin B, Beausoleil S, Cote S, Van der Vleuten C. Measurement of clinical reflective capacity early in training as a predictor of clinical reasoning performance at the end of residency: an experimental study on the script concordance test. Med Educ 2001; 35: 430-436.
12. Charlin B. Script questionnaires: their use for assessment of diagnostic knowledge in radiology. Med Teach 1998; 20: 567-571.
13. Charlin B, Brailovsky C, Leduc C, Blouin D. The Diagnosis Script Questionnaire: A New Tool to Assess a Specific Dimension of Clinical Competence. Advances in health sciences education: theory and practice. 1998; 3: 51-58.
14. Charlin B, Roy L, Brailovsky C, Goulet F, van der Vleuten C. The Script Concordance test: a tool to assess the reflective clinician. Teach Learn Med 2000; 12: 189-195.
15. Schmidt H, Norman G, Boshuizen H. A cognitive perspective on medical expertise: theory and implication. Acad med 1990; 65: 611-621.
16. Grant J, Marsden P. Primary knowledge, medical education and consultant expertise. Med Educ 1988; 22: 173-179.
17. Barrows HS, Feltovich PJ. The clinical reasoning process. Med Educ 1987; 21: 86-91.
18. Papa F, Shores J, Meyer S. Effects of pattern matching, pattern discrimination, and experience in the development of diagnostic expertise. Acad Med 1990; 65: S21- S22.
19. Amini M, Moghadami M, Kojuri J, Abbasi H, Abadi AAD, Molaee NA, Pishbin E, Javadzade HR, Kasmaee VM, Vakili H. An innovative method to assess clinical reasoning skills: Clinical reasoning tests in the second national medical science Olympiad in Iran. BMC research notes. 2011; 4: 418.
20. Higgs J. Clinical reasoning in the health professions: Elsevier Health Sciences; 2008.
21. Groves M, Scott I, Alexander H. Assessing clinical reasoning: a method to monitor its development in a PBL curriculum. Med Teach 2002; 24: 507-515.
22. Dunlosky J, Rawson KA, Marsh EJ, Nathan MJ, Willingham DT. Improving students’ learning with effective learning techniques promising directions from cognitive and educational psychology. Psychological Science in the Public Interest. 2013; 14: 54-58.
23. Mayer RE. Applying the science of learning to medical education. Med Educ 2010; 44: 543-549.
24. Page G, Bordage G. The Medical Council of Canada's key features project: a more valid written examination of clinical decision-making skills. Acad med 1995; 70: 104-110.
25. Farmer EA, Page G. A practical guide to assessing clinical decision-making skills using the key features approach. Med Educ. 2005; 39: 1188-1194.
26. Carmines EG, Zeller RA. Reliability and validity assessment: Sage publications; 1979.
27. Whitney D, Sabers D. Improving essay examinations III: Use of item analysis. Iowa City: University of Iowa, University Evaluation and Examination Service (Technical Bulletin No 11). 1970.
28. Amini M, Moghadam M, Lotfi F, Kazempour R, Abdolfathi E. Comparison between key feature exam and multiple choice questions in internal medicine department of Shiraz University of medical sciences. J Hormoz Univ of Med Sci 2013; 17: 265-272.