How to measure up

Business schools should focus on innovation with an impact, rather than on academic rankings, Dean Dipak C. Jain says

	Dean Dipak C. Jain
	Photo © Evanston Photographic Studios

Facts, even those that come in a tidy quantitative package, can be slippery things. The numbers may seem more convincing than they really are, depending on the methodology — and agenda — that produced them.

When it comes to academic rankings, the numbers get especially complicated because market dynamics can color the assessment. Then there's the intrinsic challenge of measuring qualities such as teaching effectiveness or the merit of a person's research — not to mention thorny debates about the proper forum for publishing that work. Should a byline in a popular practitioner-oriented magazine count more, or less, than an article placed in a scholarly journal whose prestige vastly surpasses its readership?

Such factors are at least partly subjective. Many of them are also enshrined in the institutional culture of higher education.

In a recent journal article, "Sizing Up the Tyranny of the Ruler," Kellogg School Dean Dipak C. Jain outlines his views on the strengths and limits of standard measures of academic performance. He also encourages business school faculty to revise how they define themselves and the potential impact of their work. Jain believes these experts should apply their knowledge more broadly, in an interdisciplinary way, to create innovations that alleviate global challenges.

"At Kellogg, we have already seen how alliances among our scholars and colleagues in medicine and engineering are addressing problems like HIV/AIDS," Jain says, referring to the Global Health Initiative developed by Kellogg and other Northwestern University colleges. "These collaborations can make lasting contributions to the world, while showing how business schools can transform themselves into innovation hubs."

Such an ambitious vision is not typically rewarded by surveys, though. Jain believes there is room for improving these metrics, since he sees rankings as one valuable tool to help administrators, faculty and prospective students. But he cautions against the temptation to read too much into these data, and notes the design challenges facing surveys.

Determining just what to measure, and then how to measure it, is the fundamental starting point — often difficult enough to sort out. Then, too, these elements must be weighted in importance relative to each other. Another challenge is making sense of various surveys, each with its own methodology and focus, which can differ greatly, producing potential confusion rather than clarity.

The essay, co-authored by Kellogg Executive Editor Matt Golosinski and published in the March edition of the Academy of Management Learning & Education, offers a circumspect approach to the details that influence how people see business schools.

According to Jain, the rank that many surveys assign to factors such as curriculum, career management services, and starting salary for graduates are so close among top schools that there is often little substantive quality difference among these institutions.

"There are, in fact, institutional differences, but these differences are not really captured in a single number. We see differences in the focus and culture of schools, and of course greater disparity between schools across different quintiles," Jain says. The key for recruiters and prospective students is to 'go beyond the numbers' and take a close look at these institutions to select the one that offers the best fit for their aspirations and personality."

Jain believes that the rankings are helpful as "general indicators offering directional guidance" for administrators, students and others with regard to a school's perceived overall strengths and areas that may require improvement. The numbers are less helpful, he says, when one is seeking a precise assessment of curriculum and culture.

"Our article looks at how rankings and other measures can be put to best use — without these yardsticks becoming impediments to innovation," Jain says. Schools can get off course, he adds, if they overreact to a survey without thinking through its implications: Trading long-term institutional strengths for questionable short-term gains to enhance rankings performance is a strategy that Jain finds problematic.

"Responding appropriately to market signals is fine," Jain says. "Kellogg has a long history of reading these indicators and staying close to recruiters and other stakeholders to ensure that our offerings remain in step with what's needed."

However, it can prove disadvantageous if rankings divert a school from its core strengths.

"I believe a holistic approach is best," Jain says. "Administrators must pay attention to rankings, but as part of a broader set of diagnostics to refine curriculum and teaching. In the end, our primary goal should be producing leaders of real substance who put their knowledge to work in ways that make the world a better place."