Understanding the Dynamics of the Shanghai Ranking

Transcription

1 Understanding the Dynamics of the Shanghai Ranking Domingo Docampo,, Spain Atlantic Research Center for Information and Communication Technologies Northern Institute Charles Darwin University October 29, 2015.

2 Outline of the Talk Who cares about rankings? Shanghai Ranking (ARWU), Why? Dynamics of the Shanghai Ranking. Social Sciences in the ARWU. ARWU as a benchmarking tool Australian Universities in ARWU and the Times Higher Education Ranking.

3 Impact of the Shanghai Ranking German s perception of its universities was shattered with the first release of the SJT rankings in 2003 (Hazelkorn) Exzellenzinitiative, 2005 ( 1.9b) to reclaim Germany s historic leadership position in research (Chambers).

4 Reactions to the Shanghai Ranking

5 Rankings will not evaporate All current university rankings are flawed to some extent; most, fundamentally, but rankings are here to stay, and it is therefore worth the time and effort to get them right (Alan Gilbert, 2007).

6 The Shanghai ranking conundrum Irreproducibility of the Shanghai Academic ranking results (Scientometrics, 2005). A simple calculation shows that merging the universities in Paris that are mainly oriented towards 'hard sciences' and Medicine, would lead to an institution that would roughly be at the level of Harvard (Famous diatribe published in Scientometrics in 2009).

7 Up there World Institution Ctry Total alu awd hici s&n pub PCP 1 Harvard University USA Stanford University USA Massachusetts Institute of Technology USA University of California, Berkeley USA University of Cambridge UK California Institute of Technology USA Princeton University USA Columbia University USA University of Chicago USA University of Oxford UK Yale University USA University of California, Los Angeles USA Cornell University USA University of Pennsylvania USA University of California, San Diego USA University of Washington USA The Johns Hopkins University USA University of California, San Francisco USA University of Wisconsin - Madison USA The University of Tokyo JPN

8 From raw data to ARWU scores For each indicator, the highest scoring institution is assigned a score of 100, and other institutions are calculated as a percentage of the top score. The statement arguably points to a linear relationship between absolute and relative (to the highest achiever) scores. Unfortunately, nothing seems to be linear in ARWU

9 University The Knot HCR HiCi James Cook University 1 10 Macquarie University 4 20 University of Copenhagen 9? Swiss Fed Inst of Technology Zurich??? 40 University of Washington ~25 ~50 University of California, Berkeley ~36 ~60 Stanford University ~49 ~70 Harvard University

10

11 Dynamics of a scoring system When improvements become more difficult (easier) closer to the maximum performance ever achieved, a progressive (regressive) scoring table should reward improvement with relatively larger (shorter) increments at the highest levels of performance

12 ARWU scores evolution % 20% 15% 10% 5% 0%

13 ARWU SCORES AUSTRALIA MELB ANU QUEEN UWA MONASH SYD UNSW ADEL MACQ TASM FLIN

14 The PUB Indicator PUB: Number of articles indexed in the SCI-Expanded and the SSCI on the previous year. A special weight of two was introduced for articles indexed in the Social Science Citation Index.

15 PUB Indicador computation Special weight applied to SSCI papers: oc: Papers listed only in the SCIE. cs: Papers listed both in SCIE and SSCI. os: Papers listed only in SSCI. The weighting scheme could be one of the two extremes: (1,1,2) or (1,2,2), or be rather special. It is indeed special: (1,1.5,2)

16 Social Science in ARWU 2015: the language divide 40% 35% 30% 25% 20% 15% 10% 5% 0% AUS NLD NOR UK ZFA NZL USA CAN IRL SWE ISR FIN BEL DNK CHL ESP PRT TWN SGP DEU ITA BRA FRA KOR JPN CHN IND RUS

17 We are not that different NUMBER OF RESEARCH UNIVERSITIES vs GDP UNIVERSITIES R² = 0.90 KOR ESP AUS CAN UK ITA FRA CHN JPN USA TWN NZL SWE BEL NLD WORLD'S GDP SHARE

18 Comparative Analysis of Higher Education Systems NUM UNIV f(gdp ) COUNTRY FIRST QUARTILE LAST QUARTILE USA China Japan Germany UK Korea Australia Portugal 1 4 4

19 And yet we are different FIRST QUARTILE AVERAGE HIGHER EDUCATION SYSTEMS IN THE SHANGHAI USA RANKING UK CAN AUS FRA FIN NOR BEL DEU NLD JPN SGP ITA AUT SAU KOR HKG CHN TWN ZAF ESP IRL PRT BRA NZL IRN GRC POL MLY CHL RUS TUR HUN THA IND R² = 0.53 CHE DNK SWE ISR FOURTH QUARTILE AVERAGE

20 FIRST QUARTILE AVERAGE HIGHER EDUCATION SYSTEMS: THE RANKING R² = 0.58 UK CHE SGP SWE CAN USA BEL AUS DEU HKG NLD FIN DNK ZAF KOR FRA NOR AUT ESP ITA NZL IRL ISR TWN RUS JPN TUR GRC CZE CHN SAU PRT CHL IND IRN MAL BRA POL HUN THA FOURTH QUARTILE AVERAGE

21 ARWU 500 vs THE 500 THE RECKONING DIFF THE ARWU United Kingdom Italy Australia Republic of Ireland New Zealand Germany France Spain Japan China United States of America

22 Universities ctry hici S&N PUB pcp score fte Swinburne University of Technology AUS University of New England AUS Murdoch University AUS Southern Cross University AUS Charles Darwin University AUS University of Waikato NZL University of The Sunshine Coast AUS Edith Cowan University AUS Charles Sturt University AUS University of Canberra AUS Victoria University of Melbourne AUS Australian Catholic University AUS Federation University Australia AUS Lincoln University NZL University of Southern Queensland AUS

23 Universities ctry hici S&N PUB pcp score fte Charles Darwin University AUS Swinburne University of Technology AUS University of New England AUS University of The Sunshine Coast AUS Southern Cross University AUS Murdoch University AUS Federation University Australia AUS University of Canberra AUS Victoria University of Melbourne AUS Edith Cowan University AUS Charles Sturt University AUS University of Waikato NZL Australian Catholic University AUS University of Southern Queensland AUS Lincoln University NZL

24 R² = 0.75 R² = 0.71

25 SURVEY 50% 60% RELATIVE WEIGHT 30% 30% 30% 7.5% 2.5% THE RANKING TEACH RES CIT INTER INDUS score rkg University of Technology, Sydney Queensland University of Technology Charles Darwin University University of Newcastle James Cook University University of Wollongong University of Tasmania Griffith University Flinders University Macquarie University Deakin University La Trobe University University of South Australia Swinburne University of Technology Murdoch University Southern Cross University Edith Cowan University University of Canberra

26 Conclusion We now understand the dynamics of the Shanghai Ranking. Evidence based rankings may offer help as institutional benchmarking tools. Ignoring them is no longer an option for research universities.