Technical Report 02/2013 Universities from Australia and New Zealand and the 2013 edition of the Shanghai ranking Domingo Docampo Lawrence Cram December 12, 2013. Abstract The emergence of international academic rankings is one of the most interesting phenomena in the field of comparative analysis of higher education. The growing influence of the Shanghai ranking led its many critics to show strong reluctance in using it as a source of analysis and improvement, mainly because it was generally thought that its results were not reproducible. Once we have found a way to accurately replicate the results of the ranking, we are in a position to shed light into the performance of whole Higher Education systems. This technical report presents the results of Australian and New Zealand universities in the 2013 edition of the Shanghai ranking. Keywords Academic rankings Shanghai arwu world class universities Australian universities Australia Australian Higher Education System New Zealand universities New Zealand New Zealand Higher Education System 1 Introduction The Academic Ranking of World Universities (ARWU) a.k.a the Shanghai Ranking, evaluates the research performance of academic institutions on the basis of numerical measures related to individual outstanding achievements and institutional research throughput. The accessibility of the sources of the raw data and the fact that the hierarchy of universities generated by ARWU roughly aligns with perceptions of the historical and recent performance of elite research universities have contributed to its acceptance and success (Docampo and Cram, 2013). A first hand account of the Shanghai ranking indicators can be found in Liu and Cheng (2005), and is summarized in Section 2 following Docampo (2011). A reliable account of the ranking procedures can be found in Docampo and Cram (2013). For the sake of clarity, a summary of the procedure is also reproduced in section 3. This report analyzes the performance of Higher Education institutions from Australia and New Zealand on the indicators measured by the Shanghai ranking (2013 Domingo Docampo Universidad de Vigo, Atlantic Research Center for Information and Communication Technologies; Campus Universitario, 36310 Vigo, Spain. Tel.: +34-986-812134 E-mail: ddocampo@uvigo.es Lawrence Cram Australian National University, University House; 1 Balmain Crescent, Acton ACT 0200, Australia. Tel.: +61-2-6125-5334 E-mail: Lawrence.Cram@anu.edu.au
2 edition). We have not included three private institutions that do not show a measurable scientific production: Avondale College, Melbourne College of Divinity and the Batchelor Institute of Indigenous Tertiary Education. To collect the necessary the data for the computation of the ARWU raw scores, we have checked the electronic sources shown in Section 4. The list of affiliation names and organizations-enhanced corresponding clues are shown in Section 5. Section 6 shows the procedure for the computation of the ARWU indicators. It is divided into two subsections, the first one dealing with the basic five indicators and the second one explaining how to compute the composed indicator (PCP). Finally, we present in Section 7 the results of 47 universities from Australia and New Zealand on the Shanghai ranking indicators. Data on the universities included in the ARWU 500 (2013 edition) have been extracted from the Shanghai ranking web site (a total of 24 institutions). Data for the remaining 23 universities have been estimated using the procedures described in Section 6. 2 ARWU ranking ARWU ranks universities on six indicators of academic performance. Individual indicators Alumni Total number of graduates from an institution winning Nobel Prizes or Fields Medals in Mathematics. Award Total number of the staff working at an institution at the time of winning Nobel prizes in the sciences, or Fields Medals in Mathematics. HiCi Total number of highly cited researchers in broad subject categories found at the web site of the Institute of Scientific Information 1. Collective indicators N&S Total number of articles published in Science and Nature in the past five years. PUB Total number of articles indexed by Science Citation Index-Expanded and Social Science Citation Index in the previous year. PCP Total scores of the previous five indicators divided by the number of full-time equivalent academic staff. 3 ARWU ranking methodology ARWU ranks universities individually or into bands by sorting on the total score, i.e. the linearly weighted sum of 6 indicator scores derived from the corresponding raw data by transformations that have been calibrated and explained by Docampo (2013). To produce the annual Shanghai ranking report the following steps are required: 1. Assemble new raw data Scan the relevant data sources (see Section 4) for updated information on new Nobel prize winners (every year) and Fields medalists (every four years), new awardees (expected in 2014) and revised affiliations of the current list of HCR for Thompson ISI Highly Cited researchers, and the institutional details of Nature & Science 1 http://www.isihighlycited.com
3 publications and ISI indexed publications in the prior year. Do your best to obtain the institutional FTE (full-time equivalent) faculty numbers used in the calculation of the PCP (per-capita performance) indicator. 2. Assess and determine exceptions Once per decade, rescale indicators for historical Nobel prize winners and Fields medalists according to the decadal aging procedure. Allocate institutions specialized in the humanities and the social sciences to the class that uses special weights to exclude Nature & Science publications. Decide which entire countries and which special institutions will be excluded from the determination of PCP because the relevant values of FTE have not been found. 3. Combine raw data and apply scaling Sum and/or average raw data over the relevant time windows according to the published algorithms. For every indicator other than PCP, multiply each value of the processed raw data by a fixed scaling factor so that Harvard University has a scaled raw score of 10000. Calculate an intermediate quantity for PCP by dividing the weighted sum of the scaled raw scores by the FTE for the California Institute of Technology, and apply a scaling factor so that this intermediate quantity for CalTech is 10000. 4. Compress the scaled raw data For each indicator, including PCP, compress the dynamic range of the scaled raw data by taking its square root to form the indicator score. 5. Calculate the total score and determine ranking/banding Combine the indicator scores using the relevant indicator weights, and linearly rescale so that the total score for Harvard University is 100. Rank institutions by total score, and determine for publication the ranking sequence, the membership of bands, and the excluded institutions. 4 Electronic sources Data were gathered from the following Electronic Sources: Nobel Prize Laureates available at http://nobelprize.org/ Data on Field Medalists available at http://en.wikipedia.org/wiki/fields Medal Data on Highly Cited Authors available at http://www.isihighlycited.com Data on scientific production available at http://www.webofknowledge.com. Preprint copies of our published articles are available at https://www.researchgate.net/profile/lawrence Cram/contributions/ https://www.researchgate.net/profile/domingo Docampo/contributions/ 5 Affiliation names We have conducted electronic searches using the organizations enhanced (og) names feature of the Web of Knowledge. According to the last release notes (THOMSON- REUTERS, 2013), og names precisely identify research published from a specific organization using the Organization - Enhanced searching to quantify an organizations output including naming variants. The names we used are shown in Table 1. It is not unlikely that some disambiguation problems with affiliations may still persist,
4 so we warmly invite comments on any apparent errors or uncertainties in the list of institutional name variants that comprise the WOK OG names. Institutional Name Auckland University of Technology Australian Catholic University Bond University Central Queensland University Charles Darwin University Charles Sturt University Curtin University of Technology Deakin University Edith Cowan University Flinders University of South Australia Griffith University James Cook University La Trobe University Lincoln University Macquarie University Massey University Monash University Murdoch University Queensland University of Technology Royal Melbourne Institute of Technology Southern Cross University Swinburne University of Technology The Australian National University The University of Adelaide The University of Auckland The University of Melbourne The University of New England The University of New South Wales The University of Newcastle The University of Queensland The University of Southern Queensland The University of Sydney The University of Western Australia University of Ballarat University of Canberra University of Canterbury University of Notre Dame University of Otago University of South Australia University of Tasmania University of Technology, Sydney University of the Sunshine Coast University of Waikato University of Western Sydney University of Wollongong Victoria University of Wellington Victoria University of Melbourne WOK og name Auckland University of Technology Australian Catholic University Bond University Central Queensland University Charles Darwin University Charles Sturt University Curtin University of Technology Deakin University Edith Cowan University Flinders University of South Australia Griffith University James Cook University La Trobe University Lincoln University Macquarie University Massey University Monash University Murdoch University Queensland University of Technology Royal Melbourne Institute of Technology RMIT Southern Cross University Swinburne University of Technology Australian National University University of Adelaide University of Auckland University of Melbourne University of New England University of New South Wales University of Newcastle University of Queensland University of Southern Queensland University of Sydney University of Western Australia University of Ballarat University of Canberra University of Canterbury Univ Notre Dame Australia University of Otago University of South Australia University of Tasmania University of Technology Sydney Sunshine Coast University University of Waikato University of Western Sydney University of Wollongong Victoria University Wellington Victoria University Table 1 Universities from Australia and New Zealand: institutional name and Thomson Reuters organizations enhanced name Besides the names from the table, three institutions require to look up another og name: og=rmit in the case of the Royal Melbourne Institute of Technology, og=victoria univ for Victoria University of Melbourne, and finally, og=univ notre dame and og=notre dame univ for the University of Notre Dame.
5 6 Computation of the indicator scores Since all the scores depend on the raw scores in relation with the raw scores of the best performer institution, the first step is to evaluate the raw scores for each institution on the five non composed indicators: Alumni, Award, HICI, S&N and PUB. For a complete account of the computation of raw scores the reader is referred to Docampo (2013). 6.1 Computation of the five basic indicators For the computation of the non composed indicators (Alumni, Staff, HiCi, S&N, and PUB) we need the raw scores of the best performer. Harvard University achieves the maximum score on the five non composed indicators in the 2013 edition of ARWU: its raw scores are shown in Table 2. Alumni Award HICI S&N PUB 35.00 33.375 191.00 446.00 15472.00 Table 2 Harvard University raw scores in 2013 Let us now summarize the precise operations with the raw scores described in steps 4 and 5 of Section 3. Let H be the number of points (raw score) of Harvard on any of the five indicators, and X the number of points achieved by any other institution. Estimated scores are computed through the same formula: EST = 100 X H (1) 6.2 PCP computation There is no easy way to access reliable data on equivalent full-time faculty for all the institutions listed in ARWU. However, in the case of Australian universities, there are public and reliable data published by the Department of Education (AUSTRALIAN- GOVERNMENT, 2013). The number of Full Time Equivalent Academic Staff used in ARWU for Australian universities is computed as the aggregate number of Full Time Equivalent Faculty at the levels above Senior Lecturer and Lecturer (Level C). Data from the 39 Australian universities included in the analysis are shown in Table 3, where the acronyms of the three columns stand for: ASL: Full Time Equivalent number of Faculty above Senior Lecturer. LLC: Full Time Equivalent number of Faculty at Lecturer Level C. FTE: Full Time Academic Staff for the calculations of the ARWU data.
6 Institution FTE Faculty ASL LLC FTE Australian Catholic University 143 141 284 Bond University 148 30 178 Central Queensland University 98 126 224 Charles Darwin University np np 235 Charles Sturt University 138 169 307 Curtin University of Technology 388 282 670 Deakin University 285 305 590 Edith Cowan University 111 163 274 Flinders University 236 203 439 Griffith University 393 357 750 James Cook University 207 187 394 La Trobe University 246 253 499 Macquarie University 303 225 528 Monash University 730 661 1391 Murdoch University 142 149 291 Queensland University of Technology 398 312 710 RMIT University 293 358 651 Southern Cross University 73 88 161 Swinburne University of Technology 148 147 295 The Australian National University 545 316 861 The University of Adelaide 388 318 706 The University of Melbourne 865 520 1385 The University of New England 108 105 213 The University of New South Wales 769 718 1487 The University of Newcastle 257 260 517 The University of Notre Dame Australia 85 124 209 The University of Queensland 740 506 1246 The University of Southern Queensland 99 105 204 The University of Sydney 899 664 1563 The University of Western Australia 480 345 825 University of Ballarat 44 np 235 University of Canberra 119 95 214 University of South Australia 213 315 528 University of Tasmania 223 241 464 University of Technology, Sydney 266 299 565 University of the Sunshine Coast 49 54 103 University of Western Sydney 263 270 533 University of Wollongong 281 237 518 Victoria University 132 149 281 Table 3 Full Time Equivalent Staff of Australian universities in ARWU 2013. Source: Australian Government, Department of Education. The case of New Zealand is not so clear cut, so we approach the computation of FTE figures in the following way: first we searched the official pages of the Government and found figures for total staff, including non academic staff. We then searched the annual reports of the New Zealand universities and found FTE figures for academic staff. Results of that search are shown on Table 4. The table includes data on ARWU FTE figures extracted from the official results of ARWU 2013 (they can be obtained by reversing equation 4), except the figure from the University of Auckland that was extracted following the same procedure but from the official results of ARWU 2012. Academic values are far larger than the ones ARWU uses for universities in New Zealand, as Table 4 shows.
7 Staff numbers year academic general ARWU FTE Massey University 2012 1,072 1,900 673 The University of Auckland 2011 2,050 2,734 929 University of Canterbury 2012 690 1,097 402 University of Otago 2012 1,162 2,180 406 Victoria University of Wellington 2012 817 1,017 463 Auckland University of Technology 2011 1,070 993 Lincoln University 2012 235 383 University of Waikato 2012 637 876 Table 4 Institutional figures for FTE academic and total staff: universities from New Zealand. Source: Annual reports of New Zealand universities To arrive at a suitable procedure to estimate the values of FTE figures for the institutions not listed in ARWU 2013, we performed a hierarchical linear regression using the data on Table 4, using academic and general as predictors and ARWU FTE 2013 as the dependent variable. Although the sample is too small to comment on statistical significance of the regression analysis, the fact that just one of the predictors (academic as expected) explains around 97% of the variance in the sample tells us that it may not be the worse choice for the estimation of FTE figures. Anyway, we would warmly welcome any suggestion that could help in arriving at the appropriate FTE figure for each New Zealand Higher Education institution. Using the results of the linear regression we get these estimations for the FTE academic staff of the three remaining universities: Auckland University of Technology (528), Lincoln University (116) and University of Waikato (314). Now, to get the PCP values for all the universities included in the analysis, both from Australia and New Zealand, we first compute the weighted sum of the squares of the estimated scores, evaluated through Equation 1, on the first five indicators; let s call that value WSS. WSS = 0.1Alumni 2 + 0.2(Award 2 + HICI 2 + N&S 2 + PUB 2 ) (2) Let s call WSSCT the value of the squared sums for Caltech, the university with the highest score in the indicator PCP. Let FTECT be the Full Time Equivalent Staff of Caltech (278 in 2013). To estimate the PCP indicator of the university X, with a Full Academic Staff of FTEX and weighted scores sum of WSSX, the following operation will be carried out: PCP = 100 WSSX FTEX WSSCT FTECT = 100 FTECT WSSX WSSCT FTEX To compute formula 3 we introduce the value of 278 for the Full Time Equivalent Academic Staff of Caltech. Since it follows from the 2013 ranking data that WSSCT = 29680, we have that (3) FTECT W SSX = 0.30605 PCP = 30.605 WSSCT F T EX (4)
8 7 Results We show in Table 5 the results of the 47 institutions under analysis. It is perhaps worth pointing out that the entrance score in ARWU 2013 was close to 9.4. INSTITUTION Alu Awd HICI N&S PUB PCP score rk The University of Melbourne 17.7 13.4 24.0 24.4 62.5 27.1 30.2 1 The Australian National University 14.1 19.4 32.3 24.0 43.0 29.3 28.9 2 The University of Queensland 13.1 0.0 21.7 22.5 60.0 26.5 25.5 3 The University of Western Australia 14.1 14.2 25.8 14.8 45.8 27.3 24.9 4 The University of Sydney 15.1 0.0 19.1 19.2 62.5 23.9 24.7 5 The University of New South Wales 0.0 0.0 20.4 15.5 55.0 21.5 20.9 6 Monash University 0.0 0.0 14.5 19.6 55.0 22.1 20.6 7 The University of Auckland 14.1 0.0 10.2 12.7 39.9 19.9 16.4 8 The University of Adelaide 15.1 0.0 10.2 9.8 39.8 22.4 16.2 9 University of Otago 0.0 0.0 14.5 10.5 34.7 26.6 15.0 10 Macquarie University 0.0 0.0 16.2 12.1 30.2 21.6 14.3 11 University of Tasmania 0.0 0.0 10.2 14.9 26.6 20.4 12.7 12 James Cook University 0.0 0.0 10.2 13.6 26.6 21.7 12.6 13 Flinders University of South Australia 16.0 0.0 10.2 5.4 26.9 20.5 12.5 14 The University of Newcastle 0.0 0.0 12.2 6.6 30.1 20.0 12.1 15 University of Wollongong 0.0 0.0 10.2 8.7 29.4 19.4 11.9 16 Griffith University 0.0 0.0 7.2 7.4 33.5 17.5 11.7 17 Swinburne University of Technology 0.0 0.0 12.5 13.2 19.7 21.4 11.5 18 University of Canterbury 0.0 0.0 7.2 9.2 26.0 19.5 10.7 19 Curtin University of Technology 0.0 0.0 0.0 10.7 32.0 17.8 10.6 20 University of Technology, Sydney 0.0 0.0 14.5 2.4 26.1 17.2 10.6 20 Massey University 0.0 0.0 10.2 4.2 25.3 14.6 9.7 22 La Trobe University 0.0 0.0 7.2 4.0 26.7 17.1 9.6 23 Victoria University of Wellington 12.0 0.0 0.0 7.4 24.0 16.9 9.4 24 Deakin University 0.0 0.0 0.0 6.6 29.7 17.2 9.2 25 Queensland University of Technology 0.0 0.0 0.0 1.5 33.5 17.3 9.0 26 University of New England 0.0 0.0 7.2 6.2 17.2 18.5 8.2 27 University of South Australia 0.0 0.0 0.0 3.5 27.9 16.8 8.2 27 University of Western Sydney 0.0 0.0 0.0 5.6 25.8 15.7 8.1 29 University of Waikato 0.0 0.0 10.2 0.0 17.8 15.7 7.4 30 Royal Melbourne Institute of Technology 0.0 0.0 0.0 0.0 24.1 13.0 6.3 31 Murdoch University 0.0 0.0 0.0 3.7 18.8 15.4 6.2 32 Charles Sturt University 0.0 0.0 0.0 2.8 17.7 14.0 5.7 33 Charles Darwin University 0.0 0.0 0.0 2.6 15.5 14.1 5.2 34 Southern Cross University 0.0 0.0 0.0 1.5 14.5 15.8 4.9 35 Victoria University of Melbourne 0.0 0.0 0.0 0.0 16.7 13.6 4.8 36 University of Canberra 0.0 0.0 0.0 3.3 13.1 12.7 4.7 37 Auckland University of Tehnology 0.0 0.0 0.0 0.0 16.0 9.5 4.3 38 Edith Cowan University 0.0 0.0 0.0 1.5 13.3 11.1 4.2 39 Lincoln University 0.0 0.0 0.0 0.0 12.1 15.4 4.1 40 The University of Southern Queensland 0.0 0.0 0.0 1.5 12.3 11.9 4.1 40 Bond University 0.0 0.0 0.0 0.0 11.3 14.1 3.8 42 University of the Sunshine Coast 0.0 0.0 0.0 0.0 11.0 14.8 3.8 42 Australian Catholic University 0.0 0.0 0.0 0.0 12.6 10.3 3.7 44 University of Ballarat 0.0 0.0 0.0 1.5 9.5 8.3 3.1 45 University of Notre Dame 0.0 0.0 0.0 0.0 8.6 11.1 2.9 46 Central Queensland University 0.0 0.0 0.0 0.0 9.1 8.3 2.7 47 Table 5 ARWU scores and ranking of universities from Australia and New Zealand
9 References AUSTRALIAN-GOVERNMENT (2013) Selected Higher Education Statistics Staff 2012 Full-time Equivalences. Department of Education web site, URL http://docs.education.gov.au/documents/selected-higher-education-statistics-staff- 2012-full-time-equivalence Docampo D (2011) On using the Shanghai ranking to assess the research performance of university systems. Scientometrics, 86(1):77 92. Docampo D (2013) Reproducibility of the Shanghai academic ranking of world universities results. Scientometrics, 94(2), 515 524. Docampo D, Cram L (2013) On the Internal Dynamics of the Shanghai ranking. Scientometrics, accepted DOI:10.1007/s11192-013-1143-0 Liu NC, Cheng Y (2005) Academic ranking of world universities: Methodologies and problems. Higher Education in Europe, 30(2), 127 136. THOMSON-REUTERS (2013) Web of knowledge v5.9 release notes. URL http://wokinfo.com/media/pdf/webofknowledge5-9rn.pdf