Short Takes On News & Events

N.Y. Hospital Group Turns The Tables On Hospital Graders

By Jordan Rau

November 4th, 2013, 10:00 AM

Judge not, that ye be not judged. Irked by the growing number of report cards assessing the quality of hospitals, a New York state hospital association has taken this biblical admonition to heart by putting out a report card grading the quality of hospital graders. Five of the 10 report cards that were evaluated were given low marks.

Photo by Phil Jern via Flickr

HANYS’ Report On Report Cards,” published Monday by the Healthcare Association of New York State, used nine criteria to assess report cards issued by government or private organizations. These principles included whether the report card used measures grounded in scientific evidence, whether the data were recent and whether hospitals had a chance to review the findings before publication to pinpoint errors.

“We had been hearing more and more from members their general frustration of all the different report cards,” said Kathleen Ciccone, director of HANYS’ Quality Institute. “It’s so time consuming for them to be able to respond to the reports, to be able to see what’s useful about them. They’re really looking for some guidance on how to use the information.”

The scoring ranged from three stars, for the report cards that met all or nearly all of the criteria, to half a star for the report cards that met few or none. U.S. News & World Report did the worst, earning half a star for its hospital rankings, which incorporate surveys of the opinions of doctors around the country.  “A subjective perception of hospital reputation is not a scientifically proven measure to evaluate hospitals’ processes of care,” HANYS wrote.

Four other report cards received only one star. HANYS dinged the Leapfrog Hospital Safety Score, which gives out grades of A to F,  for relying “heavily on unvalidated survey data” that it collects directly from hospitals.

Consumer Reports’ Hospital Safety Ratings also received a sole star. HANYS faulted the nonprofit for rolling up data that came from different sources and spanned different time frames. Some of the Consumer Reports’ data use insurance billing records, which HANYS considers inferior to data taken from medical charts. Seven of the 10 report cards, including Medicare’s own Hospital Compare website, were faulted for using billing records, even though there are no other publicly available measurements in  many areas of hospital quality such as mortality.

Healthgrades’ America’s Best Hospitals and Truven Health Analytics Top 100 Hospitals each received one star. HANYS said Healthgrades does not detail its rating method with enough specificity for others to replicate the calculations. Healthgrades and Truven violated another HANYS’ criteria because they include measures that are either not used by the federal government or not endorsed by the National Quality Forum, a nonprofit that seeks industry consensus on which quality measures are reliable.

HANYS gave two stars to the Niagara Health Quality Coalition New York State Hospital Card, which puts out a patient safety health card for state hospitals.

HANYS awarded three stars to The Joint Commission Quality Check, which evaluates how hospitals do in meeting benchmarks on a variety of measures. Medicare’s Hospital Compare and two New York State Department of Health ratings also garnered three stars. “In general, government and accrediting organizations are more successful than other report card organizations in meeting HANYS’ criteria for evaluating hospital performance,” the group wrote in its report card.

By at least one criterion, the HANYS report on report card falls short of its own standards: HANYS did not give the ratings groups an opportunity to preview the report before publication.

10 Responses to “N.Y. Hospital Group Turns The Tables On Hospital Graders”

  1. lars aanning says:

    I’m not surprised! Most of the so-called accreditation organizations depend on self-reporting of relatively simple and yes/no self-developed benchmarks. More pertinent criteria such as outcomes, efficiency, and medical/surgical errors reporting are largely ignored by the lack of transparency in corporate health care. Only the VA seems immune from these embarrassing data reporting constraints. More work is needed to make the accrediting business more credible and useful.

  2. Ted says:

    N.Y. hospitals should be thankful they don’t have Tom Corbett as their governor. If they had Corbett, they wouldn’t have Medicaid Expansion. Hospitals in Pennsylvania (PA) have been screwed. They never expected any of the governors would be so insane as to refuse Medicaid Expansion and the billions of federal tax dollars that would be returned to their state. PA hospitals expected more insured people coming through their emergency room doors to help relieve the ever increasing cost of uncompensated care. That’s why hospitals accepted the rules pertaining to them under the Affordable Care Act (ACA). Medicaid Expansion gets more people insured. However, tea party governors hate anything that has President Obama’s name on it. That includes Medicaid Expansion under the ACA. N.Y. hospitals should be thankful and grateful that they have a sane governor and a sane legislature that embraced Medicaid Expansion and they should stop complaining about the hospital graders. After all, they could have Tom Corbett, a true Neanderthal when it comes to getting healthcare for more of his constituents.

  3. Gorgonzola says:

    TED’s comment might be believable and compelling if it were not couched as a gross ad hominem smear job. Someone who communicates like this obviously has an ax to grind. Don’t believe him. Ignore him for being such a……”true Neanderthal” [sorry!]

  4. Michael Cooper says:

    While such a systematic evaluation of rating systems is helpful, it’s difficult to see it as objective since it was not done by a researcher independent of the interests represented by the association. It is indeed noteworthy that the association did not exercise the option to have the individual reporting organizations review the findings and offer a rebuttal. The association is therefore asking the same thing of us as many of the reporting organizations, namely to accept its findings as is. All of the current measurements schemes, even those generally accepted by industry insiders, have their flaws and limitations. An important question to ask of each of the HANYS criteria is whether each is reasonable given the current state of measurement science and the availability/feasibility of the idealized data/state that the criteria point to. Are these measurement organizations sloppily cutting corners or are they using the best data/methods that are reasonably available? Have the measurement organizations offered their own disclaimers to consumer audiences? What seems a little disingenuous is that hospital and health system execs do pay attention to these types of reports and use them internally, sometimes setting up their own systems of penalties and rewards. What happens when emergency room docs feel pressured to prescribe opioid painkillers to patients they suspect may be malingering because they are afraid of getting “dinged” with patient complaints and lower survey scores?

  5. Gregory says:

    Smear job? NOT EVEN CLOSE! Anybody who follows Pennsylvania politics knows Tom Corbett’s re-election campaign is in deep trouble according to recent polls. Word is, his own party might soon be asking him to step aside in 2014. His rejection of the traditional Medicaid Expansion, the same Medicaid Expansion that has been offered to all of the other states, hasn’t helped his chances in 2014. His failure to privatize liquor sales was a massive political failure. Even the GOP controlled state senate voted in favor of the Medicaid Expansion and Corbett simply ignored them. That certainly didn’t help his chances in 2014. How much more can Corbett do to lower his public image in the polls? There’s no doubt that hospitals would like to see him step aside in 2014 because they were relying on a governor to support the Medicaid Expansion to help get more Pennsylvanians insured and to help relieve their uncompensated care problem. In my opinion, if you don’t live in Pennsylvania and follow the politics, you don’t deserve to criticize others that do.

  6. Some of my children’s teachers are tough graders. If I asked my kids to grade those teachers, they wouldn’t exactly offer an objective assessment. Nor should it be news that a hospital association doesn’t give all As to its toughest critics.

    What is surprising is how this HANYS report downgrades rating services that have the nerve to use hospital reported data. They reserve their highest level of scorn for the use of billing data, meaning the information hospitals report in order to get paid. It seems to me the real story here is that a hospital association thinks its members are not telling the truth to the public they serve.

  7. Elaine Herda says:

    It’s about time someone began blowing the whistle on these rating services! The consumers who use them to make healthcare decisions should be aware of the “racket” this is. For instance, some of these organizations charge healthcare organizations to use the data. I find this interesting since it is the organizations publicly available billing data that the service simply puts through it’s own software grouper. Billing data is derived from a system that was oirginally designed for research purposes and later used to determine appropriate charges for healthcare diagnoses and procedures. Billing data it is indirectly derived from the medical record (via usually, non clinical financial coders).It was never intended nor designed for the purpose of evaluating the quality of care (only the appropriate charges for services). Reluctance to use these data as a measure of quality does not invalidate it’s use for the purpose for which it was intended –billing. A surgeon does not use a needle to make an incision and a scalpel to sew it up. Data is a tool and should be used apporpriately. We do not currently have sufficient clinically derived quality data but there are some notable rich sources such as clinical registries. The cost of using clinicians to abstract data from handwritten records is the primary reason. So, we need to establish our priorities. I say, pay hospitals to employ skilled people to derive the data rather than these for profit rating services that offer no benefit to the community and the public.

  8. Reminds me of when MD Anderson was the to-ranked oncology hospital by US News and World Report annual evaluation for the last 7 years because they excluded admissions from the ED which accounted for 40% of annual admissions at the hospital. Thus the mortality data, adjusted for morbidity which accounts for 32.5% of the score was worthless. Interestingly, the error was discovered by Memorial Sloan-Kettering Cancer Center which did include ED admissions that held them to the #2 slot. The error was caught in 2009 and corrected. Because the US News system averages hospital data over a number of years, this year’s top ranking was still affected by the data error.

  9. Marga says:

    Thank you for pointing out that hospitals grading themselves is like the fox grading his ability to give up chicken – in either case, we need to check on the emperor’s clothes!

  10. John Silver says:

    What a surprise, they like the Joint Commission.
    People call them Jake-o, but they’ve been Joke-o to RN’s for decades

    When’s the last time they “recommended” sane staffing ratios???????