Source: Getty

Op. cit. effect: departmental citation counts correlate with RAE ratings

The research excellence framework could be replaced by citation analysis, but only in the hard sciences and only if results are not used to rank quality, a study has concluded.

Four physicists and mathematicians compared the results of the 2008 research assessment exercise - the REF鈥檚 forerunner - with citation data from Thomson Reuters.

In a published on the arXiv pre-print server, 鈥淐omparison of a citation-based indicator and peer review for absolute and specific measures of research-group excellence鈥�, the authors find a high correlation between the total citation count for papers submitted by a department to the RAE and that department鈥檚 quality rating determined by the assessment panels.

探花视频

The correlation is particularly high for larger groups in the hard sciences.

The REF is intended primarily to inform the distribution of quality- related research funding based on an aggregation of peer-review scores at departmental and institutional level. For this reason, the paper says that 鈥渢he use of citation-based indicators may offer a much cheaper, and less intrusive alternative to [peer review] for large research groups in the hard sciences鈥�. But it cautions that citation analysis would be 鈥渇ar less reliable for the social sciences and humanities鈥�.

探花视频

One of the paper鈥檚 authors, Ralph Kenna, deputy director of Coventry University鈥檚 Applied Mathematics Research Centre, told 探花视频 that in practice it would be 鈥渦nacceptable鈥� to replace peer review with citation counts in some subjects and not others because departments might try to choose which panel to submit to based on whether they expected to do better via peer review or citation analysis.

鈥淚t would stymie curiosity-driven research and further distort the academic landscape,鈥� he said.

The paper also warns against using citations as a measure of departmental or institutional quality. This is because quality should be a per-head measure, but 鈥渁lmost paradoxically鈥� citations-per-head correlate only weakly with RAE quality scores.

The formal inclusion of citation analysis in the 2014 REF was rejected after a feasibility study on the grounds that citations were 鈥渘ot sufficiently robust鈥� to replace peer review - including in the sciences. According to Graeme Rosenberg, REF manager for the Higher Education Funding Council for England, worries included anomalies such as negative citations, the possibility of self-citation and 鈥渃itation clubs鈥� and the time lag between publication and the accumulation of citations.

探花视频

However, the issue is likely to be revisited ahead of the next REF, which is expected in 2020.

paul.jump@tsleducation.com