Analysis: Why we shouldn’t trust global university league tables

Global league table rankings have become increasingly prominent as a means to judge the performance of universities, and hence as a comparative indicator for prospective students. Cambridge remains among the top ten universities in the world of the three main league tables for 2012. Cambridge ranks seventh in the Times Higher Education (THE) World University Rankings, fifth in the Shanghai rankings and second in the QS World University Rankings after having been knocked down last month from first place.

It is clear that there is substantial discrepancy in rankings by the different systems. The three biggest ranking systems share a common aim but adopt different means, different methodology and data to proxy univeristy performance. Since it is often difficult to interpret precisely the information given in the tables, the rankings are not as useful as they purport to be. There exists no reliable, all-encompassing and objective ranking table.

A key problem is that the rankings systems only encompass the world's top 700-1000 higher education institutions; a very small fraction of the 17,000 universities in the world. The THE, which has a team of around 50 staff working on the rankings, admitted that it does not visit every institution in the process of scoring them, let alone all other universities. Essentially over 16,000 universities will never obtain any rating at all. This is problematic given the universities rankings disease - if you are not in the tables, you do not exist.

There is also too much focus on research and not enough focus on gauging the teaching quality, which misses the point of what a league table should achieve. In the Shanghai ranking, there is no indicator to proxy the teaching quality. In the QS league table, the only proxy is the staff to student ratio, which is only an indirect measure of teaching quality. The THE tries harder by including surveys on teaching as one of their main indicators with a weighting of 34.5%. But all in all there is a tendency to judge all higher education institutions according to criteria only suitable for judging top research universities.

As a result of the focus on research, the ranking systems largely favour English-speaking universities. Academic research published by English institutions are cited far more often than other languages and perform better in the tables.

Moreover the indicators used are both biased and flawed. Shanghai's methodology favours universities which perform high in the sciences and thus is not fit for comparing universities which emphasise humanities subjects. Its data cover 21 broad subject areas which are heavily dominated by natural sciences, medicine and engineering, neglecting humanities subjects entirely.

Overall the indicators used are subjective and many of the precise calculations involve trying to compare indicators measured in different dimensions. There is limited availability on accurate data, leading to a tendency to 'count what can be measured rather than measuring what counts'. The weightings of the indicators are subjective, and even methods of data collection vary among the different ranking systems. This leaves severe doubts about the robustness and comparability of data used to compile these league tables.

How important are league tables to society anyway? According to the editor of the THE World University Rankings, Phil Baty, the rankings are useful for "informing student and academic decision-making and helping university leaders and investors make strategic decisions". But for students, the league tables do not fulfil their aim of helping them compare universities accurately. Commenting on the THE World University Rankings 2012, Dr Wendy Piatt, Director General of the Russell Group, said: "All league tables have their limitations, and students should look beyond rankings when choosing a university or degree course."

Most academics express little concern about league tables. Professor David Baulcombe from Trinity College, Cambridge says that league tables "are not at all important to me but I realise that it cannot do any harm when the government, funders, collaborators and potential employees look at them and see that Cambridge ranks near the top in most of them".

Overall, it remains difficult to capture in objective numerical values the performance of universities, and the global league tables certainly do not achieve this. Though improvements to the methodology behind ranking universities are being researched, they have made little impact on improving rankings systems so far. As a result, global university league tables remain unreliable, inaccurate and untrustworthy.

Gwen Jing - Deputy News Editor