A key performance indicators redefinition initiative at a school of pharmacy using a modified Delphi consensus technique

Main Article Content

Teresa M. Salgado https://orcid.org/0000-0003-2708-7145
Taylor N. Reynolds https://orcid.org/0000-0003-3330-7781
Laura M. Frankart https://orcid.org/0000-0002-7337-0323
David A. Holdford https://orcid.org/0000-0001-7661-9326
Joseph T. DiPiro https://orcid.org/0000-0002-5442-0129
VCU School of Pharmacy KPIs Redefinition Taskforce


Faculty, Pharmacy, Schools, Pharmacy, Academic Performance, Education, Pharmacy, Program Evaluation, Quality Improvement, Delphi Technique, United States


Objective: The Outcomes and Assessment Committee at the Virginia Commonwealth University School of Pharmacy was tasked with refining the school’s key performance indicators (KPIs) to improve programmatic assessment by focusing on the most important measures.

Methods: Initially, 56 KPIs were tracked, nine of which were university mandated, divided into 10 modules: admissions, community outreach, continuing education, diversity, faculty experience and success, fundraising, graduate program, research and scholarship, staff experience and success, and student experience and success. Using a three-round Delphi consensus technique, KPIs were reviewed by faculty and staff. Each participant responded whether they considered each KPI to be essential or not essential for school quality assessment and improvement. Consensus for the first, second, and third rounds was defined as ≥90%, ≥80%, and ≥75% agreement, respectively.

Results: Of 109 faculty and staff invited, 49 participated in the first round, 51 in the second, and 42 in the third. At the end of the third round, accumulated consensus was achieved for 35 out of 88 (39.8%) KPIs that were considered essential and 3 out of 88 (3.4%) that were considered non-essential. Consensus percentage per module was: 15.4% (2/13) admissions, 28.6% (2/7) community outreach, 33.3% (3/9) continuing education, 27.3% (3/11) diversity, 62.5% (5/8) faculty experience and success, 55.6% (5/9) fundraising, 40% (4/10) graduate program, 33.3% (3/9) research and scholarship, 57.1% (4/7) staff experience and success, and 66.7% (4/6) student experience and success.

Conclusions: Ultimately, 35 KPIs achieved consensus as essential to measure achievement of benchmarks for the school, which totals 44 KPIs, including nine university mandated KPIs. The process facilitated faculty and staff involvement in KPI selection and achieved improved focus for programmatic assessment.


Download data is not yet available.
Abstract 583 | pdf Downloads 228 online appendix Downloads 0


1. Armstrong M. Handbook of performance management. 4th ed. London: Kogan Page; 2009.
2. Lewis DR, Hendel DD, Kallsen L. Performance indicators as a foundation of institutional autonomy - Implications for higher education institutions in Europe. Tertiary Educ Manag. 2007;13:203-226. https://doi.org/10.1080/13583880701502158
3. Borden V, Banta T. Using performance indicators to guide strategic decision making. New directions for institutional research, 82. San Francisco, CA: Jossey-Bass; 1994.
4. Burke J. Performance funding indicators: Concerns, values, and models for two- and four-year colleges and universities. Albany, NY: Rockefeller Institute; 1997.
5. Ruppert SS. Charting higher education accountability: A sourcebook of state-level performance indicators. Denver, CO: Education Commission of the States; 1994.
6. Higher Education Statistics Agency. About the uk performance indicators. Available at: https://www.hesa.ac.uk/data-and-analysis/performance-indicators/about (accessed May 30, 2020).
7. Breakwell GM, Tytherleigh MY. University leaders and university performance in the United Kingdom: Is it “who” leads, or “where” they lead that matters most? Higher Educ. 2010;60:491-506. https://doi.org/10.1007/s10734-010-9311-0
8. Tertiary Education Commission. Educational performance indicators for teos. Available at: https://www.tec.govt.nz/funding/funding-and-performance/performance/teo/epi-reports/ (accessed May 30, 2020).
9. Weingarten HP, Hicks M. Performance of the Ontario (Canada) higher-education system: Measuring only what matters. In: Curaj A, Deca L, Pricopie R, ed. European higher education area - The impact of past and future policies. Cham: Springer; 2018. ISBN: 978-3-319-77407-7
10. Fleming S, Apps N, Harbon P, Baldock C. Assessing space utilisation relative to key performance indicators – how well, not how much, space is used. J Higher Educ Policy Manag. 2012;34(5):503-515. https://doi.org/10.1080/1360080X.2012.715999
11. Watt C, Lancaster C, Gilbert J, Higerd T. Performance funding and quality enhancement at three research universities in the United States. Tertiary Educ Manag. 2004;10:61-72. https://doi.org/10.1023/B:TEAM.0000012243.95698.3d
12. Hall PD, Dipiro JT, Rowen RC, McNair D. A continuous quality improvement program to focus a college of pharmacy on programmatic advancement. Am J Pharm Educ. 2013;77(6):117. https://doi.org/10.5688/ajpe776117
13. Grabenstein JD. Trends in the Numbers of US Colleges of Pharmacy and Their Graduates, 1900 to 2014. Am J Pharm Educ. 2016;80(2):25. https://doi.org/10.5688/ajpe80225
14. Pharmacy College Application Service and American Association of Colleges of Pharmacy. 2018-2019 PharmCAS applicant data report. Available at: http://connect.aacp.org/HigherLogic/System/DownloadDocumentFile.ashx?DocumentFileKey=6fd0bf77-20d6-8bd0-ea8f-802291427396 (accessed May 30, 2020).
15. Pavuluri N, Aparasu RR, Boje KMK, Danielson J, El-Ibiary SY, Iyer AKV, Ochs LA, Robinson JD, Shah S, Williams CC, Moore TM, Wargo KA. Consideration of Aggressive and Strategic Approaches to Address Declining Enrollment in US Pharmacy Schools. Am J Pharm Educ. 2019;83(6):6959. https://doi.org/10.5688/ajpe6959
16. Studer Q. Build the foundation (passion and purpose). In: Struder Q, ed. Results that last - Hardwiring behaviors that will take your company to the top. Hoboken, NJ: John Wiley & Sons; 2008.
17. Benson H, Lucas C, Williams KA. Establishing consensus for general practice pharmacist education: A Delphi study. Curr Pharm Teach Learn. 2020;12(1):8-13. https://doi.org/10.1016/j.cptl.2019.10.010
18. Traynor AP, Borgelt L, Rodriguez TE, Ross LA, Schwinghammer TL. Use of a modified Delphi process to define the leadership characteristics expected of pharmacy faculty members. Am J Pharm Educ. 2019;83(7):7060. https://doi.org/10.5688/ajpe7060
19. Covvey JR, Ryan M. Use of a modified Delphi process to determine course objectives for a model global health course in a pharmacy curriculum. Am J Pharm Educ. 2018;82(8):6358. https://doi.org/10.5688/ajpe6358
20. Desselle S, Rosenthal M, Holmes ER, Andrews B, Lui J, Raja L. Components of a measure to describe organizational culture in academic pharmacy. Am J Pharm Educ. 2017;81(10):6022. https://doi.org/10.5688/ajpe6022
21. Janke KK, Kelley KA, Sweet BV, Kuba SE. A modified Delphi process to define competencies for assessment leads supporting a doctor of pharmacy program. Am J Pharm Educ. 2016;80(10):167. https://doi.org/10.5688/ajpe8010167
22. Janke KK, Traynor AP, Boyle CJ. Competencies for student leadership development in doctor of pharmacy curricula to assist curriculum committees and leadership instructors. Am J Pharm Educ. 2013;77(10):222. https://doi.org/10.5688/ajpe7710222
23. Larson S, Davis LE, Stevens AM, El-Ibiary S, Grice G, Pogge E, Raney E, Storjohann T. Development of a tool to assess and advance the effectiveness of preceptors: The habits of preceptors rubric. Am J Health Syst Pharm. 2019;76(21):1762-1769. https://doi.org/10.1093/ajhp/zxz183
24. Mattingly TJ 2nd, Abdelwadoud M, Mullins CD, Eddington ND. Pharmapreneur - Defining a framework for entrepreneurship in pharmacy education. Am J Pharm Educ. 2019;83(10):7548. https://doi.org/10.5688/ajpe7548
25. Boutin K, Nevers W, Gorman SK, Slavik RS, Martinusen DJ, Lo C. Development of intervention-related quality indicators for renal clinical pharmacists using a modified Delphi approach. Int J Pharm Pract. 2019;27(5):436-442. https://doi.org/10.1111/ijpp.12484
26. Fernandes O, Gorman SK, Slavik RS, Semchuk WM, Shalansky S, Bussières JF, Doucette D, Bannerman H, Lo J, Shukla S, Chan WW, Benninger N, MacKinnon NJ, Bell CM, Slobodan J, Lyder C, Zed PJ, Toombs K. Development of clinical pharmacy key performance indicators for hospital pharmacists using a modified Delphi approach. Ann Pharmacother. 2015;49(6):656-669. https://doi.org/10.1177/1060028015577445
27. Krzyżaniak N, Pawłowska I, Bajorek B. Quality pharmacy services and key performance indicators in Polish NICUs: a Delphi approach. Int J Clin Pharm. 2018;40(3):533-542. https://doi.org/10.1007/s11096-018-0623-y
28. Lima TM, Aguiar PM, Storpirtis S. Development and validation of key performance indicators for medication management services provided for outpatients. Res Social Adm Pharm. 2019;15(9):1080-1087. https://doi.org/10.1016/j.sapharm.2018.09.010
29. Mellett C, O'Donovan A, Hayes C. The development of outcome key performance indicators for systemic anti-cancer therapy using a modified Delphi method. Eur J Cancer Care (Engl). 2020;29(4):e13240. https://doi.org/10.1111/ecc.13240
30. Murphy A, Wakai A, Walsh C, Cummins F, O'Sullivan R. Development of key performance indicators for prehospital emergency care. Emerg Med J. 2016;33(4):286-292. https://doi.org/10.1136/emermed-2015-204793
31. Linstone HA, Turoff M. The Delphi method: Techniques and applications. Boston, MA: Addison-Wesley; 2002.
32. Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41(4):376-382. https://doi.org/10.1046/j.1365-2648.2003.02537.x
33. Burke JC, Minassians HP. The new accountability: From regulation to results. New Directions for Institutional Research 2002;116(special):115-127. https://doi.org/10.1002/ir.57
34. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008-1015.
35. de Villiers MR, de Villiers PJ, Kent AP. The Delphi technique in health sciences education research. Med Teach. 2005;27(7):639-643. https://doi.org/10.1080/13611260500069947
36. Stewart D, Gibson-Smith K, MacLure K, Mair A, Alonso A, Codina C, Cittadini A, Fernandez-Llimos F, Fleming G, Gennimata D, Gillespie U, Harrison C, Junius-Walker U, Kardas P, Kempen T, Kinnear M, Lewek P, Malva J, McIntosh J, Scullin C, Wiese B. A modified Delphi study to determine the level of consensus across the European Union on the structures, processes and desired outcomes of the management of polypharmacy in older people. PLoS One. 2017;12(11):e0188348. https://doi.org/10.1371/journal.pone.0188348
37. Salgado TM, Fedrigon A, Riccio Omichinski D, Meade MA, Farris KB. Identifying medication management smartphone app features suitable for young adults with developmental disabilities: Delphi consensus study. JMIR Mhealth Uhealth. 2018;6(5):e129. https://doi.org/10.2196/mhealth.9527
38. Janke KK, Bechtol RA, James S, Lepp G, Moote R, Clapp P. Determining Indicators of high-quality application activities for team-based learning. Am J Pharm Educ. 2019;83(9):7109. https://doi.org/10.5688/ajpe7109
39. Arguello B, Salgado TM, Laekeman G, Fernandez-Llimos F. Development of a tool to assess the completeness of drug information sources for health care professionals: A Delphi study. Regul Toxicol Pharmacol. 2017;90:87-94. https://doi.org/10.1016/j.yrtph.2017.08.013
40. Arif M, Smiley FM. Baldrige theory into practice: A working model. Int J Educ Manag. 2004;18(5):324-328. https://doi.org/10.1108/09513540410543475
41. Burke JC, Minassians HP. Measuring down and up: The missing link. New Directions for Institutional Research 2002;116(special):97-114. https://doi.org/10.1002/ir.62

Most read articles by the same author(s)