U.S. News Adjusts Rankings Methodology, Emphasizes Diversity

U.S. News & World Report is once again tinkering with its rankings methodology in advance of the 2024 issue, announcing a heightened focus on student diversity. In the process, the magazine is ditching some of the rankings criteria universities have found easy to manipulate.

Amid the controversies swirling around college rankings, is U.S. News making meaningful changes or simply fiddling while Rome burns?

In a May 19 announcement, U.S. News briefly discussed changes to its college rankings for 2024. Gone are metrics on alumni giving, faculty with the highest degrees in their fields (“terminal” degrees), class size, and high school standing of the entering class.

The magazine will continue to reflect those statistics in school profiles available on its website but will not factor them into the rankings themselves.

Taking their place will be an “increased weighting on a schools’ (sic) success in graduating students from different backgrounds.” It’s not clear how the magazine will reflect such “success” or how much weight it will carry.

But a quick accounting suggests the magazine has 16% of its formula at its disposal. According to the current methodology, class size accounts for 8% of a school’s total score, while the percentage of faculty with terminal degrees and the alumni giving rate each count for 3%. High school class standing accounts for 2%. All these disappear in 2024.

A related metric, called “social mobility,” which measures how well colleges graduate students who receive Pell Grants, counts for 5% of the current score. Whether or not this falls under the heading of “different backgrounds” is unclear. The magazine failed to parse its language, though we can assume “different backgrounds” implies student diversity of various forms, including race and ethnicity.

What’s more, U.S. News promises to release “new tools” that enable students to explore college data in search of the best fit based on their individual preferences. That sounds remarkably similar to what The New York Times unveiled in March — a “build your own college rankings” tool allowing students to weigh criteria such as cost, ROI, academics, athletics, and diversity, among other factors.

Finally, the magazine notes data collected from colleges will be supplemented by publicly available information. Completing the survey, it stipulates, is “not a prerequisite for inclusion in the rankings.”

That last statement contains a clue for how U.S. News will continue to rank colleges.

Traditionally, when colleges failed to submit data to U.S. News, the magazine treated them punitively, often assigning them lower scores than they should have earned. That certainly has been evident with Reed College, which has boycotted the rankings since the 1990s. Reed’s own analysis revealed a “hidden penalty” for not playing ball with U.S. News.

More recently, a few colleges have followed Reed’s lead by refusing to participate. The Rhode Island School of Design, Colorado College, Bard College, and Stillman College all bowed out this year.

But that’s about it. We haven’t seen a mass exodus yet, especially among top-ranked colleges, which continue to participate … for now.

That’s not the case among top law and medical schools. Last year Yale Law led a boycott of U.S. News, refusing to cooperate with the magazine’s data collection. Harvard soon followed suit and was joined in rapid succession by Stanford, Penn, Columbia, Michigan, Duke, and several other law schools U.S. News regularly ranks among its top 20.

Law school officials claim the rankings incentivize them to recruit students with high standardized test scores, students who often can afford expensive test prep courses. As a result, incoming classes aren’t as socioeconomically diverse as they could be.

Rankings also discourage public service careers and disregard loan forgiveness programs when calculating student debt loads.

Around the same time law schools were digging in their heels, leading medical schools, including Harvard, Columbia, Chicago, Johns Hopkins, Penn, and Stanford — all highly ranked by U.S. News — followed a similar path by boycotting the magazine’s data collection.

Like law schools, medical schools cited problems with the magazine’s misguided assumptions related to admissions criteria and career choices.

Nonetheless, U.S. News continues to rank law and medical schools using public data. Its recent rankings revealed few changes among top institutions – just a minor reshuffling of the deck.

Perhaps anticipating a wave of boycotts among undergraduate colleges, the magazine is issuing a preemptive strike by declaring the rankings will go on with or without their involvement, just as they are for law and medical schools.

The more colleges refuse to participate, the less likely U.S. News will be to penalize those that don’t.

One elite college that refused to participate, at least temporarily, was Columbia. The university got caught with its Ivy-laden hand in the rankings cookie jar, much like other institutions guilty of “gaming” the rankings to gain a more favorable spot.

In 2022, a Columbia professor published a blog post claiming the university doctored data related to class size, faculty with terminal degrees, student-faculty ratio, and retention rates.

Amid the controversy, Columbia neglected to submit data ahead of last year’s rankings, and U.S. News dropped the university from its No. 2 spot to No. 18.

Starting with the 2024 list, U.S. News will no longer have to worry about fabricated data related to class size, terminal degrees, and high school class rank. Of course, all that data appears within the Common Data Set, statistics on admissions, financial aid, graduation rates, and student demographics, among other measures.

Although it’s voluntary, most elite colleges do submit a Common Data Set. Columbia didn’t, which made it easier to fudge the numbers. Since the rankings controversy, the university has committed to issuing a report.

Alumni giving, which ostensibly measures alumni satisfaction with their education and outcomes, isn’t included in the Common Data Set. Colleges can easily finagle those figures, too. Albion College, for example, spread out senior gifts to the annual fund over five years to pad percentages.

For U.S. News, however, this now becomes a moot point.

Again, it’s unclear how U.S. News plans to evaluate a college’s “success in graduating students from different backgrounds.” It might consult the Integrated Postsecondary Education Data System (IPEDS), a mandatory submission of data to the National Center for Education Statistics that includes information on college completion by race and ethnicity.

Assuming more colleges eventually refuse to cooperate with U.S. News, it makes sense for the magazine to rely on publicly available data and not information offered by institutions. Should the day come when a critical mass of colleges, especially the ones dominating the upper echelons of the rankings, opts out of the U.S. News rankings altogether, the magazine will no longer have its special sauce: the peer assessment survey, which still accounts for one-fifth of a college’s score.

If and when that day arrives and U.S. News has to rely solely on data that’s available to everyone, what value will the rankings add other than a sophisticated aggregation and analysis? Can’t consumers draw their own conclusions based on this same data? That’s what The New York Times is banking on.

Meanwhile, we’ll see if the magazine’s revamped formula results in any surprises next year, especially among the top 20 national universities and liberal arts colleges. Based on prior methodological rejiggering, it’s unlikely. As the French say, plus ça change, plus c’est la même chose — the more things change, the more they stay the same.