|Year : 2017 | Volume
| Issue : 2 | Page : 83-88
Training and validation of standardized patients for assessing communication and counseling skills of pharmacy students: A pilot study
Mahboobeh Khabaz Mafinejad1, Mansoor Rastegarpanah2, Fereshteh Moosavi2, Mandana Shirazi3
1 Education Development Center, Health Professions Education Research Center, Tehran University of Medical Sciences, Tehran, Iran
2 Clinical Pharmacy Department, Tehran University of Medical Sciences, Tehran, Iran
3 Department of Medical Education, Education Development Center, Tehran University of Medical Sciences, Tehran, Iran; Department of Clinical Science and Education, Soder Hospital, Karolinska Institute, Stockholm, Sweden
|Date of Web Publication||31-May-2017|
Clinical Pharmacy Department, Tehran University of Medical Sciences, Tehran
Source of Support: None, Conflict of Interest: None
Objective: The objective of this study is to describe the process of training valid simulated patients (SPs) for assessing communication and counseling skills of pharmacy students' performance. Methods: This is a cross-sectional and correlational study. Psychometric properties of checklist and SPs' portrayals and their filling of the checklist regarding assessing pharmacy students were assessed. Five SPs who were working in the simulated patient's pool were volunteered to take part in the project, which one of the SPs failed. Three scenarios, along with corresponding checklists, were developed based on the usual medications of different diseases consisting of asthma, respiratory infections, and osteoporosis. The SPs' role-play performance was video-recorded and rated independently by two experts according to an observational rating scale to assess validity. The role-play was repeated after 1 week with the same scenario and the same doctor, to assess test-retest reliability. The inter-rater agreement between SPs and experts was determined by calculating the intraclass correlation coefficient and kappa coefficient. Findings: The four eligible SPs were all women, with an average age of 37 years. The correlation between mean scores of raters and mean scores of SPs was 0.91 and 0.85, respectively. The Pearson's correlation between mean scores of raters with SPs was 0.75. The checklists' reliability, Cronbach's alpha, was calculated to be 0.72. The measured weighted Cohen's kappa for the ratings of by each SP, and the gold standard was between 0.53 and 0.57, indicating a moderate agreement. The inter-rater reliability kappa coefficient between raters was 0.75 (P = 0.01).
Conclusion: The authors have demonstrated the technique of using standardized patients to evaluate communication and counseling skills of pharmacy students. The findings indicated that trained SPs can be used as an effective tool to assess pharmacy students' communication and counseling skills.
Keywords: Assessment, communication skills, counseling, pharmacy education, standardized patient
|How to cite this article:|
Mafinejad MK, Rastegarpanah M, Moosavi F, Shirazi M. Training and validation of standardized patients for assessing communication and counseling skills of pharmacy students: A pilot study. J Res Pharm Pract 2017;6:83-8
|How to cite this URL:|
Mafinejad MK, Rastegarpanah M, Moosavi F, Shirazi M. Training and validation of standardized patients for assessing communication and counseling skills of pharmacy students: A pilot study. J Res Pharm Pract [serial online] 2017 [cited 2017 Oct 22];6:83-8. Available from: http://www.jrpp.net/text.asp?2017/6/2/83/207359
| Introduction|| |
There is a universal consensus that pharmacists should be able to provide counseling to the patients, therefore, teaching such skills should be incorporated in pharmacy curriculum.[1-4] The definition of communication skills as an important proficiency in pharmacy is the ability to communicate with patients, which includes; building trust, using effective verbal and nonverbal communication, considering patient beliefs and attitudes, providing relevant information with effective questioning skills, and giving therapeutic instructions and counseling., Pharmacists with good communication and counseling skills can easily interact with patients, develop a patient-centered counseling and educate them about how to use the drugs. In addition, effective communicating and counseling by pharmacists are the key to improve the use of medications by patients and to promote adherence to medication.,
Although these skills can be taught in the curriculum,, they are not easily assessed by traditional methods. Effective methods for assessing communication and counseling skills include direct observation, simulated patients (SPs), objective structured video exam, surveys of patients' experience, interviews, etc.,
The use of SPs as a method for education and assessment of students' competencies was introduced in the 1960s by Barrows; its use has increased in different domains of medical education. SP is a person who has been trained to play her/his role in a standardized manner as an actual patient.
Using SPs in communication skills and counseling in pharmacy still seems to be a newborn. However, studies have shown that SP is used for training and assessment purposes in recent year,, but seldom used for evaluating the quality control of their performance as a valid assessment method.
According to Monaghan et al., using well-trained SPs, it would be possible to provide safe situations which they encounter with the actual patient, as well as feedback and reducing the variability of assessment among different students. In addition, research on SPs has demonstrated that when SPs are used under standardized conditions, they will perform consistently and equivalent for all students.
Hence, quality control of SPs is inevitable, through expanding SPs application in educational intervention in the field of pharmacy. In the current study, we focused on the techniques and outcomes of training and validation of SPs in pharmacy communication and counseling, as a part of a research project that assessed pharmacy students' performance at Tehran University of Medical Sciences (TUMS). The aim of the current study was to assess psychometric characteristics of standardized patients' assessments of pharmacy students' performance at TUMS.
| Methods|| |
This is a cross-sectional and correlational study to describe the process of creating valid SPs for assessing communication and counseling skills of pharmacy students' performance. The study was approved by the Medical Education Research Center at TUMS (No. 133-16592). The following steps have been accomplished: (1) development of scenarios, (2) development of checklist and assessing its psychometric properties, (3) selection of SPs, (4) training of SPs, (5) video recording, (6) validation of SPs, and (7) analyzing the data.
In the present study, three scenarios and general template were developed based on the usual medications of different diseases consisting of asthma, respiratory infections, and osteoporosis. The content and face validity of the written scenarios were determined by consensus of the expert panel, including faculty members of the Medical Education and Clinical Pharmacy Departments at TUMS.
An extensive literature review was conducted to collect potentially relevant items for the checklist.[20-23] The content validity of the checklist was determined by three clinical pharmacist experts' and two medical educationists. These items were categorized under the following headings: communicating and building a relationship, interviewing and collecting information, and counseling and delivering information. Twenty-one items were identified, and each item was to be answered on a Likert scale (0–2) where a score of 0 pointed out a poor performance and a score of 2 as an excellent performance. The correlation coefficient between SPs scores with experts' scores was calculated to assess criterion validity. Reliability of the checklist was calculated by Cronbach's alpha, where each SP reviewed the recorded video of other SPs (3n) performance and filled out the checklist. The reason of using data of filling checklist based on the recorded video of the other SPs performance was to have sufficient sample size as a pilot study which assesses the internal consistency of the checklist.
Five SPs who were working in the simulated patient's pool at TUMS were volunteered to participate in the project, one of the SPs failed. Four females were selected. Their age range was between 22 and 53 years. They have been trained to represent patient performances realistically and to evaluate pharmacists' communication and counseling skills. Overall, communication and counseling skills of 12 pharmacy students in their 4th year were evaluated by SPs to determine the accuracy of SPs' ability in filling out the checklists.
The recruited SPs were trained by a pharmacist and a medical educationalist in three small group sessions, each session lasting 2 h. The training process was consisted: accurate portrayal of the patient, appropriate reactions to questions, precise observation and recall of the pharmacy student's performance, accurate completion of the checklist, and sufficient feedback to the student.
During each session, SPs played their roles regarding their encounter with pharmacy students under the supervision of the trainers, and they rate SPs' portrait by the use of the checklists. Then, each SP was received appropriate feedback by the trainers who assessed SPs' portrayal.
In addition, SPs learned how to fill in the checklists. All the sessions were video-recorded. According to the available literature, the content (scenario, checklist, and rating scale) and process (SPs' portrayal, reliability of their ability to fill out checklists, reliability of checklist) are the components of an SP instructions which should be considered when its validation was documented.
The validity of SP' portrayal means how well acts as a real patient or portrays correctly. The SPs' portrayals were rated by two experts. One week after the training course, the ability of the SPs in verbal and nonverbal communication were evaluated by using the previously validated observational 9-item rating scale. The content validity of the rating scale was determined by two experts from the Medical Education and Clinical Pharmacy Departments. The SPs' performance had to have better than 90% accuracy rate regarding fidelity of SPs' role portrayal.
The encounter of each SP and pharmacy student was video-recorded. The performance of every four students was evaluated based on a similar scenario. In all, 12 encounters of SP and pharmacy student were recorded. Each encounter was rated individually by four SPs (48 ratings). Each encounter interval was around 10–15 min. Total time for test-retest recording estimated around 5 h.
For evaluating the inter-rater reliability of SPs' ability to fill out the checklist, each SP played her role with three pharmacy students in the pharmacy skills laboratory. After each encounter, SPs rated the pharmacy students' performance. In later session, recorded videos of SPs' portraits were watched and rated by one medical educationist and one clinical pharmacist individually. The correlation coefficient between each SP who completed checklists and two experts' assessments as a gold standard was evaluated.
To determine the reliability of SPs' filling out the checklist under the same conditions, the test-retest approach was used. Initial filling out the checklists was done by SPs in the first encounter. Ten days later, SPs rated the pharmacy students' performance for a second session. Ten days interval was considered in this study based on the results of studies which have shown there were no statistically significant differences in the test-retest reliability for 2 days or 2 weeks. Furthermore, the second session with the same student was held considering previous scenario. The inter-rater reliability was evaluated by determining the correlation between scores.
The inter-rater reliability was assessed by intraclass correlation coefficient and kappa coefficient. The intra-rater reliability of the SPs' performance to complete the checklist was assessed by means of a Student's t-test analysis. Cronbach's alpha was calculated to assess internal consistency of checklist. The Pearson's correlation was computed to determine criterion validity.
| Results|| |
The mean age of the SPs was 37 years old and all of them were women. The pharmacy students' age range was 21–25 years. They were enrolled in their third or fourth year and 60% were women. The correlation between mean scores of raters and mean scores of SPs was 0.91 and 0.85, respectively. The Pearson correlation between mean scores of raters with SPs was 0.75. The checklists' reliability, Cronbach's alpha, was calculated to be 0.72.
The results of inter-rater reliability using intra-class coefficient are presented in [Table 1]. The measured weighted Cohen's Kappa for the ratings of by each SP and the gold standard was between 0.53 and 0.57, indicating a moderate agreement [Table 2]. The inter-rater reliability kappa coefficient between raters was 0.75 (P = 0.01). When initial scores were compared to scores of retest session, there were no significant differences [Table 3]. The results of the check list showing the performance of pharmacy students' communication skills and practice evaluated by simulated patient is presented in [Table 4].
|Table 1: The correlation coefficient between the mean scores of simulated patients and rater calculated by intraclass coefficient|
Click here to view
|Table 2: The correlation coefficient between the ability of simulated patients and the rater calculated by kappa|
Click here to view
|Table 3: Intra-rater reliability of the checklist completed by the simulated patients in test-retest approach|
Click here to view
|Table 4: Performance of pharmacy students' communication skills evaluated by simulated patients|
Click here to view
| Discussion|| |
Prior studies have noted that the interaction between pharmacist and patient is remarkably dependent on the pharmacist' ability of effective communication and counseling skills. Therefore, improving pharmacist' communication and counseling skills are considered by most organizations such as the Accreditation Council for Pharmacy Education. As mentioned in the literature review, along with educating the communication and counseling skills in the pharmacy curriculum, there is a need to underpin communication and counseling assessment procedures. The use of SP as a rigorous and robust method of measuring pharmacists' competencies has received much attention in recent years.
The quality of analysis should be assessed based on a valid interface between SPs and the pharmacy students. Hence, to achieve a suitable SP-based assessment, it is necessary to validate SP's performance. The validation of assessment depends on the raters' accuracy; for this reason, the process and content of SPs' training must be suitable to guarantee the quality of their performance.
The purpose of this investigation was to describe the procedures and results of training and validation of SPs in pharmacy communication and counseling skills at TUMS.
While previous studies showed that less systematic attention has been paid to validation process of SPs, we have done several steps for it. In the present study, the SP's standardization process is to evaluate the consistency of the SPs' performances, including the validity and reliability of their performance, and the checklist.
To check the validity, we obtained consensus among five experts about the key aspects that should be included in assessment tool. This issue was also emphasized by Swanson and van der Vleuten that standardized patient checklist errors can affect the student scores. In this study, we reviewed, compiled, and standardized the checklist which was devoted much time and effort by the experts. In addition, our results indicated that the checklist had high internal consistency and reliability.
In this study, the inter-rater reliability of the SPs' ability was tested by correlating between the SPs' completing checklists and experts' judgments which considered as a gold standard and was confirmed by a high association between them. In addition, our results showed a high correlation between the mean scores of examiners and SPs. This finding is in agreement with Zanten's findings which showed a strong correlation between SPs' and raters' scores. In a similar study of general practitioners regarding the management of depression, Shirazi et al. demonstrated the strong coefficients., In contrast, McLaughlin et al. suggested that SP ratings are acceptable to students, but there was less agreement between SP-based assessment scores and experts' scores., They concluded that experts, not SPs, are the only qualified persons who could judge the students' performance. Nonetheless, the methods of SPs' training, their standardization process and doing the research in a testing environment can affect the results of this study.
Our findings indicated that the use of SP instead of experts is a good way for assessing communication skills of pharmacy. However, the generalizability of the results may be questionable because of using the small sample of students. Nevertheless, based on our findings, holding the training sessions for SPs, delivering feedback from their trainers, and developing a guideline for the assessment would be useful to increase the inter-rater reliability between SPs ratings.
The reliability of SPs' ability to fill out the checklist under the same experimental conditions was determined by a paired t-test analysis. Our results demonstrated that there was no statistically significant difference between the scores of test-retest scores.
In line with our findings, another study has also presented well to very good SPs' accuracy in completing checklist. To increase the reliability of the SPs' ability to fill out the checklist in this study, we trained SPs before they started to rate the pharmacy students and developed a guideline for the assessment.
Training and validation of SPs are fairly time-consuming, however, it is necessary to follow these steps to ensure the quality of the assessments. In addition, the number of SPs in our study was limited which might have affected the results, however, having a restricted budget was the important reason. In addition, all the SPs were women, however, we do not believe that the results will be changed dramatically if we recruited male SP.
Policy makers should note that when using SPs to assess communication and counseling skills, additional support to teach and validate their performance may be necessary. More studies may show the effects of using SPs in evaluating pharmacy students' at large scale. The long-term effects of training in SPs' performance in clinical settings are needed to augment this and other practice implications.
This study outlines how to train and validate the standardized patients to assess communication skills and counseling in pharmacy. The findings of this study indicated that a trained SP can act as a consistent rater for assessing pharmacists' performances. It is necessary that policy makers consider SP as an important method for assessment of pharmacists in high stake exams to improve patient safety.
| Authors' Contribution|| |
Mahboobeh Khabaz Mafinejad participated in the study design, conducted the analysis, and drafted and revised the manuscripts. Mansoor Rastegarpanah as a supervisor participated in the study design, supervised the data gathering process, and revised the manuscripts. Fereshteh Moosavi developed the study design and conducted the data gathering. Mandana Shirazi assessed the quality assurance of SPs and revised the analysis and manuscript.
The authors thank all pharmacy students who have participated in this study and SPs in the simulated patients' pool of Tehran University of Medical Sciences for their support in implementing the project.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Beardsley RS. Communication skills development in colleges of pharmacy. Am J Pharm Educ 2001;65:307.
Wallman A, Vaudan C, Sporrong SK. Communications training in pharmacy education, 1995-2010. Am J Pharm Educ 2013;77:36.
Accreditation Council for Pharmacy Education. Accreditation Standards and Guidelines for the Professional Program in Pharmacy Leading to the Doctor of Pharmacy Degree. Chicago, Illinois: ACPE; 2016.
Medina MS, Plaza CM, Stowe CD, Robinson ET, DeLander G, Beck DE, et al
. Center for the Advancement of Pharmacy Education 2013 educational outcomes. Am J Pharm Educ 2013;77:162.
Kimberlin CL. Communicating with patients: Skills assessment in US colleges of pharmacy. Am J Pharm Educ 2006;70:67.
Beardsley RS, Kimberlin CL, Tindall OWN. Communication Skills in Pharmacy Practice: A Practical Guide for Students and Practitioners. Philadelphia: Wolters Kluwer, Lippincott Williams and Wilkins; 2012.
Hasan S. A tool to teach communication skills to pharmacy students. Am J Pharm Educ 2008;72:67.
Ali F, Laurin MY, Larivière C, Tremblay D, Cloutier D. The effect of pharmacist intervention and patient education on lipid-lowering medication compliance and plasma cholesterol levels. Can J Clin Pharmacol 2003;10:101-6.
Penn C, Watermeyer J, Evans M. Why don't patients take their drugs? The role of communication, context and culture in patient adherence and the work of the pharmacist in HIV/AIDS. Patient Educ Couns 2011;83:310-8.
Kurtz SM, Silverman DJ, Draper J, van Dalen J, Platt FW. Teaching and Learning Communication Skills in Medicine. Oxford: Radcliffe Publication; 2005.
Yedidia MJ, Gillespie CC, Kachur E, Schwartz MD, Ockene J, Chepaitis AE, et al.
Effect of communications training on medical student performance. JAMA 2003;290:1157-65.
Humphris GM, Kaney S. The objective structured video exam for assessment of communication skills. Med Educ 2000;34:939-45.
Duffy FD, Gordon GH, Whelan G, Cole-Kelly K, Frankel R, Buffone N, et al.
Assessing competence in communication and interpersonal skills: The Kalamazoo II report. Acad Med 2004;79:495-507.
Barrows HS. An overview of the uses of standardized patients for teaching and evaluating clinical skills. AAMC. Acad Med 1993;68:443-51.
Cleland JA, Abe K, Rethans JJ. The use of simulated patients in medical education: AMEE Guide No 42. Med Teach 2009;31:477-86.
Davies ML, Schonder KS, Meyer SM, Hall DL. Changes in student performance and confidence with a standardized patient and standardized colleague interprofessional activity. Am J Pharm Educ 2015;79:69.
Watson MC, Skelton JR, Bond CM, Croft P, Wiskin CM, Grimshaw JM, et al.
Simulated patients in the community pharmacy setting. Using simulated patients to measure practice in the community pharmacy setting. Pharm World Sci 2004;26:32-7.
Monaghan MS, Jones RM, Schneider EF, Richardson RF, Grady AR, McCormack JR, et al
. Using standardized patients to teach physical assessment skills to pharmacists. Am J Pharm Educ 1997;61:266-70.
Gomez J, Prieto L, Pujol R, Arbizu T, Vilar L, Pi F, et al
. Clinical skills assessment with standardized patients. Med Educ 1997;31:94-8.
Mackellar A, Ashcroft DM, Bell D, James DH, Marriott J. Identifying criteria for the assessment of pharmacy students' communication skills with patients. Am J Pharm Educ 2007;71:50.
Mesquita AR, Lyra DP Jr., Brito GC, Balisa-Rocha BJ, Aguiar PM, de Almeida Neto AC. Developing communication skills in pharmacy: A systematic review of the use of simulated patient methods. Patient Educ Couns 2010;78:143-8.
Pharmacists ASoH-S. ASHP guidelines on pharmacist-conducted patient education and counseling. Am J Health Syst Pharm 1997;54:431-4.
Hargie OD, Morrow NC, Woodman C. Pharmacists' evaluation of key communication skills in practice. Patient Educ Couns 2000;39:61-70.
Wallace P. Coaching Standardized Patients: For Use in the Assessment of Clinical Competence. New York: Springer Publishing; 2006.
Shirazi M, Sadeghi M, Emami A, Kashani AS, Parikh S, Alaeddini F, et al.
Training and validation of standardized patients for unannounced assessment of physicians' management of depression. Acad Psychiatry 2011;35:382-7.
Marx RG, Menezes A, Horovitz L, Jones EC, Warren RF. A comparison of two time intervals for test-retest reliability of health status instruments. J Clin Epidemiol 2003;56:730-5.
Keen AJ, Klein S, Alexander DA. Assessing the communication skills of doctors in training: Reliability and sources of error. Adv Health Sci Educ Theory Pract 2003;8:5-16.
Shah B, Chewning B. Conceptualizing and measuring pharmacist-patient communication: A review of published studies. Res Social Adm Pharm 2006;2:153-85.
Watson M, Norris P, Granas A. A systematic review of the use of simulated patients and pharmacy practice research. Int J Pharm Pract 2006;14:83-93.
Rethans JJ, Gorter S, Bokken L, Morrison L. Unannounced standardised patients in real practice: A systematic literature review. Med Educ 2007;41:537-49.
Swanson DB, van der Vleuten CP. Assessment of clinical skills with standardized patients: State of the art revisited. Teach Learn Med 2013;25 Suppl 1:S17-25.
van Zanten M, Boulet JR, McKinley D. Using standardized patients to assess the interpersonal skills of physicians: Six years' experience with a high-stakes certification examination. Health Commun 2007;22:195-205.
Shirazi M, Labaf A, Monjazebi F, Jalili M, Mirzazadeh M, Ponzer S, et al.
Assessing medical students' communication skills by the use of standardized patients: Emphasizing standardized patients' quality assurance. Acad Psychiatry 2014;38:354-60.
McLaughlin K, Gregor L, Jones A, Coderre S. Can standardized patients replace physicians as OSCE examiners? BMC Med Educ 2006;6:12.
Parkes J, Sinclair N, McCarty T. Appropriate expertise and training for standardized patient assessment examiners. Acad Psychiatry 2009;33:285-8.
Boulet JR, McKinley DW, Norcini JJ, Whelan GP. Assessing the comparability of standardized patient and physician evaluations of clinical skills. Adv Health Sci Educ Theory Pract 2002;7:85-97.
Vu NV, Marcy MM, Colliver JA, Verhulst SJ, Travis TA, Barrows HS. Standardized (simulated) patients' accuracy in recording clinical performance check-list items. Med Educ 1992;26:99-104.
[Table 1], [Table 2], [Table 3], [Table 4]