Highlights
Publicly available Skilled Nursing Facility (“SNF”) quality measures from The Centers for Medicare & Medicaid Services ("CMS") are susceptible to self-reporting bias which hampers their accuracy.
Comparison of self-reported SNF data with subsequent hospital claims data for SNF patients indicates that the self-reported data is often under-reported and unrelated to hospital data.
Pressure ulcers are under-reported by the SNF self-reported measure when compared with the hospital-based measure.
Self-reported measures for pressure ulcers, UTIs and falls are uncorrelated with hospital-based measures.
Introduction
Nursing home quality is important for patients and CMS makes quality measures publicly available online at its Nursing Home Compare website to assist patients and their family members to evaluate and select SNFs. Because of the widespread use of these measures, nursing homes may have an incentive to underreport certain measures. Nursing home quality has come under increased scrutiny recently due to the large number of COVID-19 related deaths in nursing homes and the CMS quality measures have been used for Senate testimony and academic research. However, concerns over the accuracy of these quality measures are widespread and critiques come from government reports, media articles and academic research dating from at least 2001.
Here we present further evidence of the inaccuracy of three nursing home quality measures. Using hospital claims from SNF patients we calculated measures for pressure ulcers, UTIs and falls and compared them with the CMS measures. The CMS measures on pressure ulcers, UTIs and falls are largely self-reported and are based on the Minimum Data Set, which is filled out by nursing home staff.
Results
Pressure ulcer under-reporting
The rate of self-reported pressure ulcers at a given SNF is substantially lower than the hospital data of patients admitted from the SNF would suggest. For each SNF we calculated the ulcer rate self-reported by the SNF (i.e., % of admissions with new or worsened pressure ulcers) and divided it by the ulcer rate derived from patients who were re-admitted to the hospital from the SNF (i.e., % of admissions that were re-hospitalized with a pressure ulcer excluding patients with a pressure ulcer prior to their SNF stay). The median ratio of the self-reported and hospital-based pressure ulcer rates was 0.48, indicating that over half of SNFs under-reported by at least a factor of two (Figure 1). This is actually a conservative measure of under-reporting; the hospital data only included patients that were re-admitted to the hospital in the numerator and any SNF patients with pressure ulcers that were not re-admitted to the hospital weren’t counted.
Figure 1. Under-reporting of Pressure Ulcers.
Nursing homes were classified based on the ratio of the self-reported measure divided by the hospital-based measure. Nursing homes with a ratio of 1 were included in the ‘Somewhat Over-reported’ category.
Lack of correlation
Further evidence against the accuracy of the self-reported measures is the lack of correlation between the self-reported and hospital-based measures. If a low self-reported rate was evidence that a nursing home was providing excellent care for a health outcome, one would also expect the nursing home to have a low rate of hospitalizations with that outcome. Our analysis suggests this is not the case. SNFs with low self-report rates may have high hospital-based rates and vice-versa. For example, a SNF home we analyzed had a low self-reported UTI rate of less than 1 in 1,000 which is in the first percentile of the self-reported rates. However, 5.8% of this SNF’s admissions were re-hospitalized with a UTI, which is in the 86th percentile for the hospital-based measure.
Aggregating our results, we found low correlations between self-reported and hospital-based measures for pressure ulcers, UTIs and falls at .02, .04 and .09 respectively, indicating that the self-reported measures were inconsistent with hospital-based diagnoses. Sanghavi, Pan & Caudry performed a similar analysis for falls comparing hospital-based and minimum data set measures and also found a low correlation and under-reporting for falls.
More on methods
Further details regarding the differences between the self-report measures and the hospital-based measures are noted. In order to reduce the possibility that the adverse health outcome occurred in the hospital, the outcome was only counted towards the hospital-based rate if the hospital diagnosis was marked “Present on Admission.” We note that one hypothesis for pressure ulcer under-reporting is that the SNF staff simply had not noticed their patients’ pressure ulcers. If that hypothesis were true it would still be a matter of concern even if the under-reporting was not systematic or intentional. The self-reported measures for UTIs and falls also cover a different type of patient (‘long-stay’) than the hospital data (‘short-stay’). If the SNF were good at fall prevention for long-stay patients one would still expect the SNF to be good at fall-prevention for short-stay patients but, as noted above, no correlation was observed.
Public access
We have made the quality metrics calculated in our research publicly available on this website for every Medicare-certified skilled nursing facility in the U.S. In addition, we are making available additional metrics for you to find and evaluate nursing homes, including staffing levels, potential incidents of abuse and neglect and prior risk of excessive treatment. A detailed explanation of our metrics is found in our methodology page.
References
2. https://www.kff.org/health-costs/issue-brief/state-data-and-policy-actions-to-address-coronavirus/
5. Abrams, H. R., Loomer, L., Gandhi, A., & Grabowski, D. C. (2020). Characteristics of US Nursing Homes with COVID‐19 Cases. Journal of the American Geriatrics Society.
6. Department of Health and Human Services Inspector General. Nursing home patient assessment: Quality of care. 2001. http://oig.hhs.gov/oei/repor ts/oei-02-99-00040.pdf
7. Department of Health and Human Services Inspector General. Inappropriate payments to skilled nursing facilities cost Medicare more than a billion dollars in 2009. 2012. http://oig.hhs.gov/oei/reports/oei-02-09-00200.pdf
8. Kaiser Family Foundation. Implementation of Affordable Care Act Provisions To Improve Nursing Home Transparency, Care Quality, and Abuse. Prevention - Issue Paper. 2013.
9. U S Government Accountability Office. GAO-02-279, Nursing Homes: Federal Efforts to Monitor Resident Assessment Data Should Complement State Activities. 2002.
10. U S Government Accountability Office. GAO-03-187, Nursing Homes: Public Reporting of Quality Indicators Has Merit, but National Implementation Is Premature. 2002.
11. U S Government Accountability Office. GAO-06-117, Nursing Homes: despite increased oversight, challenges remain in ensuring high-quality care and resident safety. December 2005.
12. Survey and Certification Group, Centers for Medicare and Medicaid Services. MDS/Staffing focused surveys update. https://www.cms.gov/Medicare/Provider-Enrollment-and-Certification/SurveyCertificationGenInfo/Downloads/Survey-and-Cert-Letter-15-25.pdf Published February 13, 2015.
13. Editorial Board. Is that really a five-star nursing home? New York Times. February 25, 2015.
14. Thomas K. Medicare star ratings allow nursing homes to game the system. New York Times. 2014.
15. Neuman MD, Wirtalla C, Werner RM. Association between skilled nursing facility quality indicators and hospital readmissions. JAMA. 2014;312(15):1542-1551.
16. Han X, Yaraghi N, Gopal R. Winning at all costs: Analysis of Inflation in nursing homes’ rating system. Prod Oper Manag. 2017;27(2):215-233.
17. Sanghavi, P., Pan, S., & Caudry, D. (2020). Assessment of nursing home reporting of major injury falls for quality measurement on nursing home compare. Health Services Research, 55(2), 201-210.