Scholarly recognition or social impact: not both?

The proposed Research Quality Framework aims to assess both the quality and impact of academic work. From what I understand, the designers have assumed that outputs demonstrating quality may be different from outputs demonstrating impact.

I think this is a wise decision, because recently I obtained data allowing separate quantitative assessments of scholarly use and social impact of my own work - and the two are even less correlated than I anticipated.

To assess scholarly use, I used figures on citations to my journal articles, as listed in the ISI Web of Knowledge. My most highly cited social science paper has 34 citations, the next 23, the third 16 and so forth. These figures allow me to pinpoint which particular articles had the most scholarly recognition by this measure.

There are limitations to citation counts. The Web of Knowledge includes a select list of journals, which for social sciences picks up less than 20 percent of citations, according to work by Linda Butler, head of the Research Evaluation and Policy Project at ANU and an expert in bibliometric analysis. A lot of journals aren't included, nor are citations from books.

Citations aren't a direct measure of quality, but rather a surrogate. However, other ways of assessing quality, for example direct peer assessments of publications, research grant successes and invitations to give keynote addresses, don't as easily allow for quantitative comparisons of research outputs.

To assess social impact, I used figures on website hits on my publications, most of which are available in full text on my site, in 2006. These figures show which article had the most hits, which one was second, and so forth. The top articles had thousands of hits in 2006.

There are limitations to website hit figures. The university's data has some statistical anomalies, but the comparative ranking of different items seems unaffected. More fundamentally, does someone looking at your article on the web represent a social impact? It could involve a careful reading or a passing glance. However, the same reservations can be raised about citations.

Some of the website hits are from scholars, but judging from correspondence I receive, most are not. Some hits are from students but many are from outside the university system.

Does someone reading your article constitute a social impact? Not necessarily, but sometimes it has an effect. Though relatively few readers offer any feedback to authors, I know that many people use my writings for obtaining insights and making decisions, because quite a number of them tell me in unsolicited correspondence.

I'm told the RQF is not likely to endorse website hits as a measure of impact. I agree that website hits are far from perfect, but they do have the advantage of allowing quantitative comparisons of individual publications.

So I ended up with two lists: a list of my most highly cited articles and a list of my papers with the most website hits. I omitted my books from the comparison because only some of them are online.

The striking thing is that these two lists are almost entirely different. There's not a common single item in the top 10 on each list. The divergence between the lists suggests either that scholars are not the primary users of my website or, if they are, they don't often cite what they read.

The highly cited articles are mostly about academic issues and debates, whereas the top-rating webpages are about issues of practical relevance, such as defamation, or topics that receive a lot of media attention, such as plagiarism and scientific fraud. International journals with high impact factors published most of my highly cited articles, not surprisingly, but not a single such journal published any of the top items for website hits, several of which appeared in nonrefereed journals.

My preliminary conclusion from this exercise is that for someone like me writing on different topics in different fields, the work with greatest scholarly recognition may well be quite different from the work with greatest social impact.

In most cases, social impact depends on work being accessible beyond academic journals on shelves or databases. In my case that is primarily through open web access; it can also occur through media stories, reports to client groups, textbooks, public talks and personal contact, either by the researcher or via students and followers.

No doubt there are some scholarly outputs - such as prominent books - generating both significant scholarly recognition and social impact. But it would be a mistake for researchers to assume they can achieve high scores on both criteria with the same piece of work.

Having obtained article-specific data on scholarly recognition and social impact, and noted their divergence, I am left with a question: what priority should I put on researching and publishing in areas and journals likely to be influential with scholars versus areas and outlets more likely to be used outside the academy?

Brian Martin is in the Faculty of Arts, University of Wollongong.

 

I thank Lee Astheimer and Linda Butler for helpful comments.