American Journal of Management Science and Engineering

| Peer-Reviewed |

Selection of Reliable and Valid Surgeon Performance Measures

Received: 26 October 2020    Accepted: 03 November 2020    Published: 11 November 2020
Views:       Downloads:

Share This Article

Abstract

Objective- To identify measures of surgeon performance that are valid, reliable, and capable of classifying the risk of surgeon performance. Data Sources- A surgical quality improvement program, dataset unique to selected hospitals and surgeons containing abstracted surgical case records. Study Design- Six criteria were employed to assess the validity of 24 candidate measures of surgeon performance: 1) the presence of a surgeon random intercept; 2) a surgeon signal that is greater than zero; 3) surgeon majority control; 4) reliability of the surgeon random intercept of at least 0.7; 5) the capacity to identify both low- and high-risk surgeons and 6) the presence of a learning/improvement effect. Data collection/Extraction methods- Surgical case review nurses abstracted cases for each surgeon using a structured sampling and abstraction methodology. Principal findings- Comparing outcomes requires risk adjustment and the use of the "true score" approach but is limited by case volume constraints and a confounding factor, i.e., the hospital, if used to judge surgeons' performance. Assessing surgeon performance requires a measure of the surgeon's effects on the consequences (postoperative occurrences) of surgical procedures, i.e., the surgeon-specific random intercept, which is a product of a multilevel risk adjustment model. Conclusion- Morbidities and mortality lack the characteristics necessary to be used as measures of surgeon performance. However, the process (task-time) measures LOS and OT both have high event rates, high reliability, and are capable of classifying surgeon risk.

DOI 10.11648/j.ajmse.20200505.12
Published in American Journal of Management Science and Engineering (Volume 5, Issue 5, September 2020)
Page(s) 62-69
Creative Commons

This is an Open Access article, distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium or format, provided the original work is properly cited.

Copyright

Copyright © The Author(s), 2024. Published by Science Publishing Group

Keywords

Multilevel Mixed-Effects Modeling, Risk Adjustment for Clinical Outcomes, Reliability, Validity

References
[1] Yi, S. G., N. P. Wray, S. L. Jones, B. L. Bass, J. Nishioka, S. Brann, and C. M. Ashton. 2013. "Surgeon-Specific Performance Reports in General Surgery: An Observational Study of Initial Implementation and Adoption." Journal of the American College of Surgeons 217 (4): 636-647.E631.
[2] Allen, M., and O. Pierce. 2015. "Making the Cut: Why Choosing the Right Surgeon Matters Even More than You Know." ProPublica, July 13, 2015.
[3] Birkmeyer, N. J., and J. D. Birkmeyer. 2006. "Strategies for Improving Surgical Quality--Should Payers Reward Excellence or Effort?" New England Journal of Medicine 354 (8): 864-870.
[4] Cooper, W. O., O. Guillamondegui, O. J. Hines, C. S. Hultman, R. R. Kelz, P. Shen, D. A. Spain, J. F. Sweeney, I. N. Moore, J. Hopkins, I. R. Horowitz, R. M. Howerton, J. W. Meredith, N. O. Spell, P. Sullivan, H. J. Domenico, J. W. Pichert, T. F. Catron, L. E. Webb, R. R. Dmochowski, J. Karrass, and G. B. Hickson. 2017. "Use of Unsolicited Patient Observations to Identify Surgeons with Increased Risk for Postoperative Complications." JAMA Surgery 152 (6): 522-529.
[5] Stephens, M. R., W. G. Lewis, A. E. Brewster, I. Lord, G. R. Blackshaw, I. Hodzovic, G. V. Thomas, S. A. Roberts, T. D. Crosby, C. Gent, M. C. Allison, and K. Shute. 2006. "Multidisciplinary Team Management is Associated with Improved Outcomes after Surgery for Esophageal Cancer." Diseases of the Esophagus 19 (3): 164-171.
[6] Nurok, M., N. Sadovnikoff, and B. Gewertz. 2016. "Contemporary Multidisciplinary Care--Who Is the Captain of the Ship, and Does It Matter?" JAMA Surgery 151 (4): 309-310.
[7] Statement on Physician-Led Team-Based Surgical Care. 2016. "American College of Surgeons." [accessed on August 11, 2017]. Available at: https://www.facs.org/about-acs/statements/86-team-based.
[8] Hall, B. L., K. M. Huffman, B. H. Hamilton, J. L. Paruch, L. Zhou, K. E. Richards, M. E. Cohen, and C. Y. Ko. 2015. "Profiling Individual Surgeon Performance Using Information from a High-Quality Clinical Registry: Opportunities and Limitations." Journal of the American College of Surgeons 221 (5): 901-913.
[9] Shih, T., A. I. Cole, P. M. Al-Attar, A. Chakrabarti, H. A. Fardous, P. F. Helvie, M. T. Kemp, C. Lee, E. Shtull-Leber, D. A. Campbell, M. J. Englesbe. 2015. "Reliability of Surgeon-Specific Reporting of Complications after Colectomy." Annals of surgery 261 (5): 920-925.
[10] Iezzoni, L. I. 2003. Risk Adjustment for Measuring Health Care Outcomes, Third Edition. Chicago, Illinois: Health Administration Press.
[11] Adams, J. L. 2009. The Reliability of Provider Profiling: A Tutorial. Santa Monica: RAND Corporation.
[12] Adams, J. L., A. Mehrotra, J. W. Thomas, and E. A. McGlynn. 2010. "Physician Cost Profiling — Reliability and Risk of Misclassification." New England Journal of Medicine 362 (11): 1014-1021.
[13] Rabe-Hesketh, S., and A. Skrondal. 2012. Multilevel and Longitudinal Modeling Using Stata, Volumes I and II, Third Edition. College Station, TX: Stata Press. Stata Corp LP.
[14] Guillamondegui, O. D., O. L. Gunter, L. Hines, B. J. Martin, W. Gibson, P. C. Clarke, W. T. Cecil, and J. B. Cofer. 2012. "Using the National Surgical Quality Improvement Program and the Tennessee Surgical Quality Collaborative to Improve Surgical Outcomes." Journal of the American College of Surgeons 214 (4): 709-714.
[15] Rotondo, M. F., C. Cribari, and R. S. Smith. 2014. Resources for Optimal Care of the Injured Patient. Chicago, Illinois.: Committee on Trauma, American College of Surgeons.
[16] Stone, A. B., M. C. Grant, C. Pio Roda, D. Hobson, T. Pawlik, C. L. Wu, and E. C. Wick. 2016. "Implementation Costs of an Enhanced Recovery after Surgery Program in the United States: A Financial Model and Sensitivity Analysis Based on Experiences at a Quaternary Academic Medical Center." Journal of the American College of Surgeons 222 (3): 219-225.
[17] Hopper, A. N., M. H. Jamison, and W. G. Lewis. 2007. "Learning Curves in Surgical Practice." Postgraduate Medical Journal 83 (986): 777-779.
[18] Jaffer, A., B. Bednarz, B. Challacombe, and S. Sriprasad. 2009. "The assessment of surgical competency in the UK." International Journal of Surgery 7 (1): 12-15.
[19] Khan, N., H. Abboudi, M. S. Khan, P. Dasgupta, and K. Ahmed. 2014. "Measuring the surgical 'learning curve': methods, variables and competency." BJU International 113 (3): 504-508.
Author Information
  • Independent Consultant, Knoxville, Tennessee, the United States

Cite This Article
  • APA Style

    William Thomas Cecil. (2020). Selection of Reliable and Valid Surgeon Performance Measures. American Journal of Management Science and Engineering, 5(5), 62-69. https://doi.org/10.11648/j.ajmse.20200505.12

    Copy | Download

    ACS Style

    William Thomas Cecil. Selection of Reliable and Valid Surgeon Performance Measures. Am. J. Manag. Sci. Eng. 2020, 5(5), 62-69. doi: 10.11648/j.ajmse.20200505.12

    Copy | Download

    AMA Style

    William Thomas Cecil. Selection of Reliable and Valid Surgeon Performance Measures. Am J Manag Sci Eng. 2020;5(5):62-69. doi: 10.11648/j.ajmse.20200505.12

    Copy | Download

  • @article{10.11648/j.ajmse.20200505.12,
      author = {William Thomas Cecil},
      title = {Selection of Reliable and Valid Surgeon Performance Measures},
      journal = {American Journal of Management Science and Engineering},
      volume = {5},
      number = {5},
      pages = {62-69},
      doi = {10.11648/j.ajmse.20200505.12},
      url = {https://doi.org/10.11648/j.ajmse.20200505.12},
      eprint = {https://download.sciencepg.com/pdf/10.11648.j.ajmse.20200505.12},
      abstract = {Objective- To identify measures of surgeon performance that are valid, reliable, and capable of classifying the risk of surgeon performance. Data Sources- A surgical quality improvement program, dataset unique to selected hospitals and surgeons containing abstracted surgical case records. Study Design- Six criteria were employed to assess the validity of 24 candidate measures of surgeon performance: 1) the presence of a surgeon random intercept; 2) a surgeon signal that is greater than zero; 3) surgeon majority control; 4) reliability of the surgeon random intercept of at least 0.7; 5) the capacity to identify both low- and high-risk surgeons and 6) the presence of a learning/improvement effect. Data collection/Extraction methods- Surgical case review nurses abstracted cases for each surgeon using a structured sampling and abstraction methodology. Principal findings- Comparing outcomes requires risk adjustment and the use of the "true score" approach but is limited by case volume constraints and a confounding factor, i.e., the hospital, if used to judge surgeons' performance. Assessing surgeon performance requires a measure of the surgeon's effects on the consequences (postoperative occurrences) of surgical procedures, i.e., the surgeon-specific random intercept, which is a product of a multilevel risk adjustment model. Conclusion- Morbidities and mortality lack the characteristics necessary to be used as measures of surgeon performance. However, the process (task-time) measures LOS and OT both have high event rates, high reliability, and are capable of classifying surgeon risk.},
     year = {2020}
    }
    

    Copy | Download

  • TY  - JOUR
    T1  - Selection of Reliable and Valid Surgeon Performance Measures
    AU  - William Thomas Cecil
    Y1  - 2020/11/11
    PY  - 2020
    N1  - https://doi.org/10.11648/j.ajmse.20200505.12
    DO  - 10.11648/j.ajmse.20200505.12
    T2  - American Journal of Management Science and Engineering
    JF  - American Journal of Management Science and Engineering
    JO  - American Journal of Management Science and Engineering
    SP  - 62
    EP  - 69
    PB  - Science Publishing Group
    SN  - 2575-1379
    UR  - https://doi.org/10.11648/j.ajmse.20200505.12
    AB  - Objective- To identify measures of surgeon performance that are valid, reliable, and capable of classifying the risk of surgeon performance. Data Sources- A surgical quality improvement program, dataset unique to selected hospitals and surgeons containing abstracted surgical case records. Study Design- Six criteria were employed to assess the validity of 24 candidate measures of surgeon performance: 1) the presence of a surgeon random intercept; 2) a surgeon signal that is greater than zero; 3) surgeon majority control; 4) reliability of the surgeon random intercept of at least 0.7; 5) the capacity to identify both low- and high-risk surgeons and 6) the presence of a learning/improvement effect. Data collection/Extraction methods- Surgical case review nurses abstracted cases for each surgeon using a structured sampling and abstraction methodology. Principal findings- Comparing outcomes requires risk adjustment and the use of the "true score" approach but is limited by case volume constraints and a confounding factor, i.e., the hospital, if used to judge surgeons' performance. Assessing surgeon performance requires a measure of the surgeon's effects on the consequences (postoperative occurrences) of surgical procedures, i.e., the surgeon-specific random intercept, which is a product of a multilevel risk adjustment model. Conclusion- Morbidities and mortality lack the characteristics necessary to be used as measures of surgeon performance. However, the process (task-time) measures LOS and OT both have high event rates, high reliability, and are capable of classifying surgeon risk.
    VL  - 5
    IS  - 5
    ER  - 

    Copy | Download

  • Sections