Each year, US News and World Report publishes its list of the top 50 hospitals in various specialties (example here). Now, an article has been published suggesting that one aspect of the methodology used by the magazine is flawed.

“The Role of Reputation in U.S. News & World Report’s Rankings of the Top 50 American Hospitals,” by Ashwini R. Sehgal, MD is in the current edition of the Annals of Internal Medicine. (You can find an abstract here, and you can obtain a single copy for review from Dr. Sehgal by sending an email to axs81 [at] cwru [dot] edu.)

Dr. Sehgal finds that the portion of the U.S. News ranking based on reputation is problematic because reputation does not correlate with established indicators of quality:

The relative standings of the top 50 hospitals largely reflect the subjective reputations of those hospitals. Moreover, little relationship exists between subjective reputation and objective measures of hospital quality among the top 50 hospitals.

More detail is provided in the article:

The predominant role of reputation is caused by an extremely high variation in reputation score compared with objective quality measures among the 50 top-ranked hospitals in each specialty. As a result, reputation score contributes disproportionately to variation in total U.S. News score and therefore to the relative standings of the top 50 hospitals.

Because reputation score is determined by asking approximately 250 specialists to identify the 5 best hospitals in their specialty, only nationally recognized hospitals are likely to be named frequently. High rankings also may enhance reputation, which in turn sustains or enhances rankings in subsequent years.

Given the importance attributed to the U.S. News ranking, this article is bound to raise concerns. I know that the folks at the magazine have worked hard over the years to make their rankings as objective as possible, and it will be interesting to see their response to Dr. Sehgal’s critique.

Reputation versus quality: U.S. News Hospital Ranking

Maybe some figures would help. According to the WHO, the Infant Mortality Rate (IMR) in the U.K. for the year 2004 was 5.3, while it was 6.7 in the U.S. and a dismal 80.2 in Pakistan. Similarly, the life expectancy in the U.K. for the year 2004 was 78.5; it was 77.4 for the U.S. and 64.9 for Pakistan. U.S. data – although vastly better than Pakistan – still occupies the bottom rung of the Western world much to the chagrin of public health experts here and amidst much public disapproval.

This may come as a surprise to some that think that the U.S. healthcare system is the most advanced and sophisticated in the world – which it is. What could possibly explain this disparity? The US spends 15.2 % of its GDP on healthcare, the UK only 8% while Pakistan spends a paltry 2.4%. What is the underlying reason for the U.S. to lag behind other Western nations despite a very robust healthcare budget? The answers to these lies in the manner healthcare resources are allocated in the U.S. In this case, it is the lack of universal and basic healthcare in the U.S. that sets it back.

Public heath experts in the U.S. have long argued this point. According to the U.S. Census Bureau, about 16% of the Americans are uninsured. According to the same source, with an uninsured rate at 19.0 percent among children in 2005, children in poverty were more likely to be uninsured. While the in the U.K. healthcare is universal for at least basic health services. Contrast that to Pakistan where there is no health coverage at all. According to the WHO, 72% of the total healthcare expenditure in Pakistan is private expenses, of which 98% are out-of-pocket expenses. This factor alone puts a damper on the frequency and extent of medical care sought by the public. These disparities in health insurance translate into poor health outcomes as noted above.

Lack of universal healthcare in the U. S. and Pakistan highlight the same fact – albeit in starkly different ways. The U.S. with all its excellence in tertiary care and R&D is still a laggard in basic health numbers in the Western world. No matter how much you spend on fancy medicine, basic health coverage for all (or lack thereof) still makes a difference. In Pakistan as in any other third world country, the conclusions are more straightforward – no health coverage means very poor collective health.

Whereas in the US healthcare has become a juggernaut that is becoming more and more unruly, it is in dire need of a jumpstart in Pakistan. Experts have pointed out various reasons for that ranging from lack of political will to poor resource allocation to inept allocation processes. On a psychosocial level, the greatest impediments to a better health care system in Pakistan come from general public malaise and institutional cynicism with the deep-rooted belief that our moribund system is incurable. Unless we can overcome these psychosocial and political hurdles through a strong leadership or grassroots effort or both, the future does not look too good.

This drama reached its crescendo when a senior health official who had been instrumental in rallying support for the polio drive was recently killed. All this comes at the heels of a fatwa by a local cleric who declared war on the United Nations, the World Health Organisation and the NGOs aiding and abetting these ‘foreign organisations’. The Northern Areas of Pakistan have witnessed many political upheavals, especially in the days of the British Raj, but they have seen few sincere outsiders who understood their problems and culture. From a public health perspective some parallels can be drawn between the attitudes of the tribal people in Pakistan and African Americans in the US against organised medicine. The Tuskegee Syphilis Study continues to fuel African American distrust to this day. It took only a few hundred study-subjects in the Tuskegee Syphilis Study to lose the African American community’s trust in medical establishment. In the case of Northern Pakistan, it was the loss of more than a dozen lives in Bajaur Agency, among other incidents, that alienated the people.

However, there is one stark difference between the two communities. Whereas the collective American conscience seems to sag under the burden of heavy guilt when it comes to African Americans, it is quite the reverse in the Pakistani case.

The media has been quick to malign the tribal community in general. The ignorance and biases of tribal people undoubtedly played a part in all this. However, to see this refusal as a war of religion, as some are trying to do, seems a little bit of a stretch. Local attitudes have been shaped by recent events like bombings, especially the one that resulted in the killing of innocent civilians and children when the US forces were on the hunt for Ayman Al Zawahiri. Similarly, one of the clerics who most vociferously denounced the vaccinations had recently lost his brother during an attack on a madrassa by the Pakistani army.

But all is not lost in this struggle. For one thing, the proportion of kids that have not been vaccinated is small. According to a government report more than 5.7 million kids were vaccinated in January and another 3 million are due very soon. The area affected by polio, part of the North-West Frontier Province of Pakistan, is also the area where vaccination refusal rate is the highest. It is small enough to make covering it a feasible project administratively and logistically for public health officials. Also, leaders from major religious political parties in Pakistan have endorsed the vaccinations. Moreover, general public opinion about the vaccines is favourable even in the tribal areas, as evidenced by the overall success rate of the number of kids already vaccinated.

Despite all this, public health workers in the tribal areas face an uphill battle. Squeezed between tribal misgivings, Western biases and the recent loss of their colleagues to terrorism, they still need to get the job done. Public health officials have to expand their role from simple vaccines pushers to community builders. It will take patience on the part of the WHO, courage and dedication by the public health personnel and, above all, acceptance from the local tribal leaders to make any headway on this issue. The challenge is unique but not insurmountable.

Their efforts produced a mortality study, their second in two years, published last October in The Lancet, Britain’s premier medical journal. The study produced a number: 654,965. This was the researchers’ estimate of probable “excess mortality” since the 2003 invasion — Iraqis now dead who would not be dead were it not for the war. The number was a product of the study, not its central point. But it commanded attention because it was appallingly, stupefyingly large. It was beyond anyone’s previous worst imagining. It was just plain hard to believe, and in the weeks following its publication, it became an oddity of science: a single number so loud, in effect, it overwhelmed the conclusions of the research that produced it…

Read more here…

"More than one billion people in the world live on less than one dollar a day. In total, 2.7 billion struggle to survive on less than two dollars per day". Eradication of poverty is a noble yet Utopian goal according to many. When Jeffery Sachs published his "

"Many such studies are also impaired by difficulties in identifying what is causing what. For example, a country’s economic distress may induce donors to give it more aid — which may, in terms of associative statistics, suggest a connection between aid and bad economic performance. But using such a correlation to prove the bad effects of aid turns the causal connection on its head. Easterly tries to avoid such pitfalls, but the statistical associations on which he draws for his comprehensive pessimism about the effects of aid do not offer a definitive causal picture." Many such studies are also impaired by difficulties in identifying what is causing what. For example, a country’s economic distress may induce donors to give it more aid — which may, in terms of associative statistics, suggest a connection between aid and bad economic performance. But using such a correlation to prove the bad effects of aid turns the causal connection on its head. Easterly tries to avoid such pitfalls, but the statistical associations on which he draws for his comprehensive pessimism about the effects of aid do not offer a definitive causal picture."

It is hard to prove negative or positive association in this sort of studies where there are too many variables involved. Often it is hard to measure progress or change in concrete terms. For some the answer lies in too-little-too-late approach by the donor countries while for others it is the corrupt and inefficient bureaucracies of the countries accepting donations that are to blame. We thus see two opposing paradigms pitted against each other. One is for big push and big money and the other is for piecemeal change that comes from within. The former premise appeals to most of the people in the third world and many whistle blowers and to people who are weary of big organizations like the World Bank, the IMF and the UN. The latter makes sense to most of the people in the Western world. Perhaps like many other debates this one is also too metaphysical to reach one true conclusion – perhaps the final answer lies in the eyes of the beholder.