Data Dives

Lots of ‘C’s As Hospitals Get Graded For Patient Safety

By Jordan Rau

June 6th, 2012, 9:04 AM

The cities of New York and Los Angeles grade their restaurants on cleanliness and the precautions they take to avoid making customers sick. Now hospitals are getting similar assessments for their patient safety records from the Leapfrog Group, a nonprofit devoted to patient safety.

Photo by Phil Jern via Flickr

For 2,651 hospitals, Leapfrog created a single letter grade out of 26 different measures collected by Leapfrog or Medicare officials. They included hospitals’ adherence to safe practices, such as entering physician orders into computer records to avoid penmanship errors and removing catheters promptly to minimize the risk of infections. The grade was also based on hospitals’  records of mishaps, such as bed sores, infections and punctured lungs.

Leapfrog gave 729 hospitals an “A” grade, 679 hospitals a “B” and 1,111 hospitals a “C.” Another 132 hospitals were scored with “Grade Pending,” Leapfrog’s euphemism for below a “C.”

Leapfrog plans to introduce “D”s and “F”s when it updates the ratings in six months, but didn’t want to be too harsh in its first report, said Leah Binder, Leapfrog’s executive director.

“We designed this to capture the attention of the public,” Binder said. “No one has ever given one individual score to most of the general hospitals in the country, including those that didn’t perform well.”

The American Hospital Association disputed Leapfrog’s ratings, saying in a statement that it “has supported several good quality measures but many of the measures Leapfrog uses to grade hospitals are flawed, and they do not accurately portray a picture of the safety efforts made by hospitals.”

Among the mediocre performers in Leapfrog’s study are some well-respected names. New York-Presbyterian Hospital in Manhattan and the Cleveland Clinic Hospital both got “C”s. UCLA Ronald Reagan in Los Angeles got a “Grade Pending.” Leapfrog gave “A”s to some other well-known places such as the Mayo Clinic in Rochester, Minn., and Cedars-Sinai Medical Center in Los Angeles — but also to a host of obscure community hospitals.

“The hospitals that achieved an A came from all walks of life, across the gamut of hospital types and people they serve,” Binder said. “Safety appears to be something that all hospitals can choose.”

Massachusetts, Maine and Vermont were the only three states where half of the hospitals or more got a grade of “A.” In 24 states and the District of Columbia, half or more of the hospitals got a “C” or “Grade Pending.” The worst performers were the District, Alabama, New York, Oregon, South Dakota, New Mexico, Hawaii, Arizona and Arkansas, where at least two thirds of the hospitals got a “C” or lower.

Hospital officials raised a number of objections. Dr. Michael Henderson, chief quality officer at the Cleveland Clinic, noted that much of the data was a year or two old, and many hospitals have made significant strides since then. “The question the public needs to be asking is, ‘Are you working on this? Are you getting better?’” he said.

Leapfrog’s information comes from two sources: its own surveys of hospitals that agree to participate, and data the Centers for Medicare & Medicaid Services culls from its billing records and posts on its Hospital Compare website. Dr. Shannon Phillips, a quality and safety officer at the Cleveland Clinic, said that the way Leapfrog calculated its grades, “you are automatically at a deficit if you did not participate in their survey.” The Clinic dropped out of Leapfrog’s surveys three years ago as the government began requiring more and more information to be provided to CMS.

Binder, however, said that the way the scores were calculated would not disadvantage hospitals that didn’t partake in Leapfrog’s survey.

Leapfrog, which is based in Washington, said it consulted with nine nationally known experts, including Peter Provonost of Johns Hopkins, Patrick Romano of UC Davis Davis and Ashish Jha of the Harvard School of Public Health, in designing the letter grade scoring methods.

Jha called Leapfrog’s grades “a really important step forward,” because they simplify complex measurements into things that consumers can easily understand and digest. Numerous studies have found that consumers rarely use complex quality measurements when choosing hospitals, blunting the potential influence of resources like Hospital Compare.

“As better data comes along and as time goes by, my hope is this grading will get refined,” Jha said.

Unlike a city public health department, Leapfrog can’t post its grades on the front doors of a hospital. Binder said Leapfrog hopes that groups of employers that purchase insurance will disseminate the ratings to workers and use them when selecting health care providers.

State-By-State Data

The Leapfrog Group, a nonprofit watchdog for patient safety, has given hospitals grades on how well they protect patients from avoidable mishaps. For each state, here are the percentage of hospitals that received “A”s, “B”s, “C”s or “Grade Pending”, which is Leapfrog’s category for those hospitals that scored worse than a “C”. Leapfrog did not grade Maryland hospitals, because, under a unique reimbursement arrangement, the federal government does not collect data from them.

You can sort the chart by state and each grade.

12 Responses to “Lots of ‘C’s As Hospitals Get Graded For Patient Safety”

  1. Tracy Harty says:

    Leah, nice job!!!
    We miss you, Tracy

  2. Sharon Dooley says:

    Interesting article – curious about the scorecard. Somehow do not find Maryland there. Am I missing something?

  3. WIlliam Ross says:

    Maryland is still not there.

  4. Pa says:

    Read the article. It tells in its closing paragraphs why Maryland is not included.

  5. josi says:

    Is there a way to see individual hospital’s grades in a state, like Oregon–which had lots of C s. Would like to know the identity of each hospital—I had a feeling Oregon would not score well.

  6. Tim says:

    I’ve been reading for a while that hospitals are more in the “sickness” business not the heath”care” business, and this data seems to support it. Ironically, hospitals get paid for treating illness, whether they caused it or not, so they make more money by fixing their own mistakes: medication errors, hospital acquired infections, falls, etc.

  7. Erica says:

    Sharon- Unscored hospitals in the Hospital Safety Score include all hospitals from the State of Maryland, and certain territories including Guam and Puerto Rico, which the federal government excluded from required public reporting at the national level.

    Residents of these states and territories should be very concerned that they do not have information on safety that other U.S. residents have. The Leapfrog Group encourages ALL hospitals to disclose their safety results to the public so that consumers can make informed decisions on where to seek care.

  8. Douglas Dame says:

    Measuring hospital performance is not easy. But here’s two important points on the Leapfrog methodology, that I have not seen in comments elsewhere:

    (1) Leapfrog made a completely arbitrary decision to “grade on a curve” so that, by design, half of the rated hospitals would be graded as a C or below, and the other half would be A or B. Imagine if your kid’s trigonometry teacher said “well, in my class, half of the grades will be Cs or Ds,” because that’s exactly what they’ve done here.

    (2) The Leapfrog Group has data that they gather from Leapfrog-participating hospitals on some of their chosen metrics, which is not available for non-Leapfrog hospitals. So to their credit, they use alternative data sources where possible, and re-weight the component scores based on the number of actual available metrics for each hospital. For the Process+Structure measures, which account for 1/2 of the total score, they could have up to 15 metrics available for Leapfrog hospitals, while only eight of metric are potentially available for non-Leapfrog hospitals.

    But for the non-Leapfrog hospitals, for 2 of the 8 process/structure measures for which data could be available, the maximum score achievable by a non-Leapfrog hospital is 15 of the 100 points, equivalent to the lukewarm description of “some progress.”

    For example, for Computerized Physician Order Entry (CPOE), if the AHA Survey says “FULLY IMPLEMENTED on ALL units” [emphasis added], a non-Leapfrog hospital gets a score of 15 (“some progress”), and is effectively penalized 85 (raw) points for not participating with Leapfrog. Ditto for ICU Physician Staffing, where the maximum score a non-Leapfrog hospital can get is also 15 instead of 100. (Source: Leapfrog Group, Hospital Safety Score: Scoring Methodology [final version] (June 2012). See the section on “Using Secondary Data Sources and Dealing with Missing Data,” p. 13 for Computerized Physician Order Entry, and p. 13-14 for ICU Physician Staffing.)

    Thus for the Process+Structure half of the total score, the best a non-Leapfrog hospital could do would be (100*6 + 15*2) / 8 = 78.75% of the max score potential available to a Leapfrog-participating hospital, which is (100*15)/15 = 100%.

    So on an unweighted basis, while Leapfrog hospitals are scored for the entire kit and kaboodle on the equivalent of a 0-100 raw scale, non-Leapfrog hospitals can’t score above 89.4. (100 possible for the Outcomes Measures half, 78.75 possible for Process&Structure, the combined average is 89.4 .)

    Whether this is effect is really intentional or not is unclear.

    But the bottom-line effect is inarguable .. the scoring methodology puts a 10% thumb on the scale against non-Leapfrog hospitals.

    That’s a heck of a handicap to have to make up, when by design of the grading curve half the hospitals are going to be assigned grades of C or D grades.

    Hope this adds some new insights to the discussions.

  9. rick says:

    If hospitals could refuse charity cases that walk into their emergency rooms, they’d have better grades. Those who want to repeal the new health care reform law because of the individual mandate should put the same effort into repealing any laws that “mandate” emergency room care for anyone who walks through the door. It’s hypocrisy to favor one and not the other.

  10. rick says:

    It’s amazing that Massachusetts (MA) has the highest grades in the chart above. Isn’t MA the state that enacted Romneycare? Why isn’t Mitt Romney proud of his signature legislation? Why would he be trying to hide such a record achievement?

  11. Gary says:

    Do not confuse across the board compliance with government directives with quality care.
    Many of the government mandates have little or no proof that they actually improve quality care at the bedside.
    In many cases they increase the cost of care and shift the focus of the institution from actual quality care to documentation of compliance with government regulations.
    Instead of spending time at the bedside, the nursing staff is spending their time at the computer.

  12. Jessie Collins says:

    Douglas, thank you for the detailed info. I see a few flaws in Leapfrog’s reason for publishing such a skewed and obviously biased article. They need to make crystal clear by saying upfront they are biased toward THIER participating facilities and let the Public know how this data is complied. These organizations sometimes do more harm then good!
    Gary, I so agree with you about the documentation vs quality care, but at the same time CPOE, complete implementation of an EMR, a well trained staff in how to use that EMR, will over time show a decrease in errors and give back to staff more time to spend on quality care.
    Rick, the ED over MD debate has been raging for decades, the only thing that will help here is low cost, affordable health care on the out patient level for those who do not have insurance and transportation to get to appointed care times. Don’t forget, the EMS system is impacted by lack of transportation on the part of many of these clients. No where is there an easy fix!