How reliable are QS rankings? Close look at parameters, processes and outcomes raises doubts

NEW DELHI: A high position in one of the international university rankings can do a lot for an Indian university, even fetch it near-freedom from domestic regulatory control. It also allows institutions to market themselves to students.

But a closer look at the ranking method and parameters, along with other activities of ranking agencies, raises doubts about the processes involved in the ranking exercise and the possible biases in them. This is especially true of Quacquerelli Symonds (QS), headquartered in the United Kingdom.

It is not immediately apparent from the thicket of headlines on the QS rankings – there is a “World University Rankings” and smaller ones that are region or theme-specific – but the methodology relies heavily on “perception”, judged through surveys.

Plus, a 2021 paper from the University of California, Berkeley, about QS ranking of Russian institutions found an uncomfortable correlation between QS’ consultancy clients and their positions. Frequent clients jumped many places up the ranks over the five years from 2016 to 2021, rising far more rapidly than non-clients. Even more troublingly, the rise was mainly on the strength of teacher-student ratio data that did not match with records anywhere else, including the Times Higher Education (THE) rankings and the Russian government.

Why look at rankings?

First, a disclosure: Careers360 has its own ranking system and publishes annual rankings by discipline. For this, it relies on data from a host of disclosures made to various public bodies and also directly to Careers360 itself. Any data shared with us is cross checked against disclosures made to regulators and other bodies.

The reason for looking into international rankings now is this: global rankings have a disproportionate influence on Indian higher education. An institution’s place in rankings by QS, Times Higher Education (THE) or the Academic Ranking of World Universities (ARWU) can determine the extent of regulatory control over it, the courses it can launch and where it can hire teachers from.

Multiple higher education initiatives and norms have international rankings a criterion for selection, including the Institute of Eminence (IoE) scheme, the graded autonomy scheme and the University Grants Commission’s regulations on recruitment of foreign faculty. Under the first two, institutions in the top 500 ranks of international ranking lists are practically freed of regulatory control.

But a close look at the ranking methodology and revenue stream of one – QS – casts the objectivity of the exercise into doubt.

QS Rankings: 50% perception

As much as 50% of the weight is assigned to perception parameters -- "academic reputation" with 30%, "employer reputation" with 15% and "employment outcomes", 5%. For these, the QS conducts a survey among respondents whose names and contact details are supplied by the institution being assessed.

These surveys can be open to manipulation. A professor shared with Careers360 a 2021 email from a university seeking their "help and support for the QS Academic Reputation Survey". The email requests the recipient to suggest “at least 5 academics from outside India” and willing to respond to QS’ survey queries. The same professor had recommended academics before and so, this time, was requested to find new names.

Asked about fairness and the reliance on perception in an email interaction, QS’ Simona Bizzozero wrote, At QS, we take rigorous steps to ensure the accuracy, fairness, and integrity of our survey processes. All institutions participating in the rankings are allowed to submit 400 academic and 400 employer contacts. However, institutions are strictly prohibited from canvassing for nominations. To further minimise bias, surveys are sent directly from QS to these contacts, ensuring respondents complete the survey independently. Additionally, respondents can nominate up to 10 domestic and 30 international institutions rather than selecting only one, which reduces the likelihood of skewed responses. To safeguard the process further, QS employs a stringent validation system that actively monitors, reviews, and detects anomalous responses and patterns. Any irregularities identified are excluded from the results. QS has a clear and enforceable policy around survey participation, and any breaches of our guidelines can result in sanctions, including temporary exclusion from rankings consideration. Through these robust measures, we ensure our survey results reflect a fair, balanced, and unbiased assessment, maintaining the integrity and credibility of the QS rankings.”

$100 ‘token of appreciation’

Another email – this one far less carefully-worded than the one above – severely undermines the claims of fairness and probity. Former Indian Institute of Technology (IIT) Delhi director and current group-VC of Birla Institute of Technology & Science (BITS) Pilani campuses, V Ramgopal Rao posted an email on professional networking site, LinkedIn, that appears to do exactly what QS says it prohibits – “canvassing”.

The email is purportedly from The Korea Advanced Institute of Science and Technology’s (KAIST) Chemical and Biomolecular Engineering department and requests the recipient to participate in "brief survey to gather insights from esteemed professors like yourself".

After this, the author of the email quickly gets down to business. “Additionally, we would greatly appreciate it if you could take a few minutes to complete a QS survey to help provide feedback on our department’s reputation within the academic community. To express our gratitude for your time, we would be happy to provide you with a $100 (USD) token of appreciation upon completion of the two surveys," says the email. The goal of the email is clear – payment in exchange for a review of the department on QS.

Rao’s own comment on the email is equally striking. While he did not share the details of the sender or those of recipients, he wrote: “I see a few thousand faculty names in their email list. Even if a few 100 accept their office, I guess they will be fine."

V Ramgopal Rao's post on an email related to the QS World Ranking (Image: LinkedIn / Rao)

On the surveys, Bizzozero further wrote: “As outlined earlier, while institutions can provide 400 academic and 400 employer contacts, these form only a portion of the overall surveyed pool. QS maintains an extensive, independent database of academic and employer contacts, who are also invited to participate in the survey annually. This approach ensures that survey responses are drawn from a broad and diverse range of participants, reducing any potential institutional bias. It’s also important to clarify that the Employment Outcomes indicator does not rely on survey data. Instead, it is derived from Graduate Employment rates and Alumni Impact, ensuring a robust and independent evaluation of graduate success. By combining institutional submissions with our independently curated databases and ensuring the integrity of the survey process, QS rankings achieve a balanced and credible representation of institutional performance.”

QS vs other rankings

Another 10% of the weightage goes to international students ratio and international teachers ratio – a parameter that, in the Indian context, allows private universities charging usurous fees to leapfrog many public institutions with longstanding repute on the rankings. Yet another 5% is assigned to the “International Research Network”, which QS owns. Institutions awarded high research scores by QS seem to struggle on other rankings and research indices.

In response, Bizzozero said: “Discrepancies in rankings across different indices such as QS, NIRF, and the Nature Index can be attributed to differences in methodology and the specific indicators each ranking evaluates. The QS rankings assess institutions on multiple dimensions, including academic reputation, employer perception, internationalisation, and research impact (using metrics like citations per faculty). Other indices may place a stronger emphasis on localised or specific research outputs. This variance in focus can result in differing rankings for the same institution. It is important to note that no single ranking provides a complete picture. By understanding how different metrics and priorities shape each ranking, institutions can identify areas of strength and opportunities for growth.”

Also read 'I would invalidate the very exercise of ranking institutions,' says scientist

In contrast, the education ministry’s National Institutional Ranking Framework (NIRF) privileges data from the institutions. As a government initiative with strong links to regulatory bodies, NIRF is also uniquely well-placed to obtain data. Participating universities submit data in affidavits, giving such disclosures the legal heft absent in submissions made to private entities.

On the wildly different outcomes in government, other international rankings and its own, QS said, “Different rankings, such as QS and NIRF, use distinct methodologies and evaluate institutions through different metrics. QS rankings are designed to assess the strengths of comprehensive universities globally, considering factors such as academic and employer reputation, internationalisation, and research impact. Rankings serve as valuable tools for stakeholders to understand institutional performance across specific areas. The differences observed between rankings arise from variations in focus and methodology, offering institutions diverse insights that can inform their strategic decision-making. By interpreting results from multiple rankings, institutions gain a well-rounded perspective of their strengths and areas for improvement.”

QS Rankings: Clients rise quickly

There is another cause for concern. In 2021, Igor Chirikov from UC Berkeley’s Center for Studies in Higher Education studied QS ranking of Russian universities. Russia requires universities to disclose all large procurement contracts on a government portal. Chirikov found that of 28 ranked universities, nine were clients of QS – the ranking agency was a frequent consultant for them – the rest were not or only occasionally so.

Chirikov analysed the rankings of the clients and non-clients over five years, 2016-2021, and found that client institutions had improved their rankings dramatically in comparison to non-clients. The average ranking of the nine frequent clients rose from 609 in 2016 to 418 in 2021 – a jump of 191 positions. Over the same period, the mean rank of the 19 occasional or non-clients rose from 582 to 508, a change of mere 74 positions.

The researcher examined THE rankings as well – eight of the nine QS clients and 17 of those that were not were also ranked by THE. In this case, the mean rank of the eight fell, from 706 to 858 – by over 151 places – and the non-clients actually fared better, falling only 102 places.

There were more startling revelations. Looking at data against each ranking parameter, Chirikov found that in the case of client institutions, the biggest leap was in faculty-student ratio. QS data shows a 14.31 point improvement in the faculty-student ratio score of the nine client institutions over five years, propelling them up the ranks.

But, the very same institutions show a decline in faculty-student ratio, not just on THE but also in data reported to the Russian Ministry of Higher Education.

Also read CBSE wants international boards reined in; letter to education ministry seeks directions for AIU

In response, QS said: “At QS, we operate with absolute integrity to ensure the transparency and independence of our rankings. Our rankings operations are completely ringfenced from any commercial influence. This separation is safeguarded by strict policies, including employment contracts for staff involved in rankings compilation and analysis, which contain specific clauses prohibiting undue influence. Breaching these clauses is a fireable offense. Additionally, QS Consulting contracts explicitly clarify that our consultancy services are not focused on improving rankings. Instead, they aim to help institutions enhance key performance areas such as research impact, teaching resources, internationalisation, student employability, and sustainability . While these efforts may lead to improved performance over time, rankings are inherently dynamic; even as one institution progresses, others may advance at a faster pace, influencing relative positioning. QS has built its credibility over 34 years of serving the higher education sector and students worldwide. Ethical conduct is not just a core value for us—it is both a moral and business imperative. Our commitment to integrity underpins our long-term success and the trust placed in us by universities, students, and stakeholders globally. Trust is the cornerstone of our reputation, and any compromise in this area would harm both our mission and our business. By maintaining rigorous ethical standards, QS continues to serve as a reliable, impartial, and valuable partner to the global higher education community.”

Rising profits

The UC Berkeley paper from 2021 says that 22 of the 28 Russian universities had spent USD 2.8 million on QS-related services over previous eight years.

No such analysis is easily possible in India as much of the relevant data is simply not publicly available. However, QS has an Indian subsidiary and according to data from the ministry of corporate affairs, it reported gross revenues amounting to Rs 8.53 crore and profit of about Rs 93 lakh in 2018. Within five years, by 2023, it was making Rs 42.78 crore in revenues and Rs 5.23 crore in profits.

The THE has clients too but unlike QS, the THE’s clients are essentially advertisers. A second disclosure: Careers360 has a similar model where clients advertise on the publications. Chirikov’s paper shows that THE’s model might make it less vulnerable to manipulation.

The QS is open about its for-profit status. “For context, all major organisations which produce and publish rankings are commercial entities,” said Bizzozero. “Producing robust, trusted rankings requires significant resources to ensure methodological rigour, data integrity, and continual improvement. Our ranking methodologies are fully transparent and publicly available on our website, including detailed explanations and calculations, which can be accessed through our Support Pages (support.qs.com). To further promote understanding and transparency, we conduct regular webinars for institutions, providing opportunities to clarify our processes, address queries, and gather feedback. Additionally, QS actively seeks input from the higher education sector and our external Advisory Board, which helps ensure our methodologies remain fair, relevant, and reflective of evolving educational priorities. This commitment to transparency underscores QS's role as a trusted partner to institutions worldwide, including those in India.”