Where is the evidence? Grey literature, research and policy

Posted by Amanda Lawrence on 27 February 2013

Julian Thomas

Swinburne Institute for Social Research, Swinburne University of Technology

Keynote speech by Professor Julian Thomas presented at the conference Where is the evidence: policy, research and the rise of grey literature, held in Canberra, October 2012.


The question, ‘where is the evidence?’ responds to the familiar contemporary call for evidence-based policy. Evidence-based policy may be more of an idea than a reality, but the answer to the question turns out to make a difference: it matters where the evidence is, the form it takes, and who is in a position to find it. In this talk, I want to introduce some of the issues this intriguing question raises. I hope this will be a useful way of framing some of the discussion to come, and also to position our interest as researchers working on the topic. (You’ll see that we have distributed a project brief.) I’ll talk mainly about grey literature produced in university environments, especially what we can call the new practices of digital scholarship, but the approach I suggest should also be relevant to non-academic contexts.

Episodes such as the debate over the 2010 report of the International Panel on Climate Change expose an expectation that the evidence informing public policy debate should be peer-reviewed, and that it should be found in reputable journals by established publishers. The problem was that much of the IPCC’s evidence was actually not like this: it was published as “grey literature”, produced and distributed more or less outside the protocols of systematic research publishing. In the case of the review of the 2010 IPCC report, the review found that the Panel could not rely solely on the scholarly journal literature, because vital information was published elsewhere.

Grey literature, in the form of reports, blogs, papers or presentations, is produced by a diverse array of institutions, including university-based centres, think tanks, consultancy firms and private research agencies, NGOs, international bodies and government agencies. It is often distributed on a website, outside conventional channels, without the involvement of established publishers. So when we are thinking about grey literature, we are usually thinking about one or more of these three aspects — the nature of the document, the source of the document, and how it has been distributed. There is nothing neat about the distribution of these attributes. Documents often occupy a place on a grey-scale spectrum, rather than an unambiguous position on one side or another of the divide.

Beyond the question of definition, I think there are some potentially useful ways of conceptualizing grey literature. One is to look outside the domain of information science, and draw upon the scholarship of the informal economy, beginning with the work of development economists and anthropologists who began to take seriously “off the books,” unregulated economic activity in the developing world in the 1970s. Subsequent work has pointed to the persistence of informal economic activity, its flourishing in the West, and the creative tensions between the formal and the informal sectors. Grey literature can be understood as a kind of informal publishing, something accessible and possible outside traditional industry structures. Like other informal activity, it is resilient, enabled by new technologies, and it is also deceptively important for the formal sector, with which it is necessarily entangled.

Another way of thinking about the distinctive features of grey literature is to consider what some cultural sociologists would call the ‘field’ in which it is produced: the networks of people, relationships and institutions that make it possible. In the world of conventional publishing, the field is composed of publishers, editors, distributors, retailers, agents, and authors (although they are rather disconnected from the other elements). Power has shifted among these figures, with publishers increasingly squeezed on both sides, by retailers and by agents and their authors. The field of grey literature is almost unrecognizably different from book and journal publishing. It consists of funding agencies, consultancy firms, universities, research centres, think tanks, public sector bodies and NGOs. But here also there are distinctive, and somewhat fluid, power relations at work, especially between funding bodies, universities, and the creators of grey literature. The result is a complex web of competing incentives.

With these perspectives in mind, we can begin to describe the conditions that have led to the profusion of grey literature. These are technical, institutional and economic:

- Clearly, digital publishing and distribution have enabled the low cost production of grey literature, using the tools readily at hand. Some of those tools are perfect for the task; some less so. (PDFs, for example, were originally devised in order to meet the needs of ‘the paperless office’ as it was envisaged in the early 1990s, rather than the more complex challenges of networked publications).

- The popularization of ‘evidence-based policy’ necessarily creates continuing interest in readily available, up-to-date, relevant research findings. The demand may be more imagined than real — but all policy development work stands to benefit from easy access to current research.

Research funding models embody multiple, potentially contradictory impulses and objectives: for example that outcomes should be accessible; that they be published in high quality outlets; that researchers should increasingly collaborate with industry and government; and that publicly-funded work should have tangible ‘impact’ both within and outside the academy. The effect of these is to provide some incentives to publish informally. Grey literature is also clearly a response to widely recognised deficiencies in the formal research publishing system:

- The cost of journals and books

- The inaccessibility of journals outside universities and particularly well resourced libraries

- The delays in publishing

- Uncertainty as to the quality assurance value of peer reviewing: what does peer reviewing tell us? (Not that the data has been checked or that the references are correct: more a matter of relevance.)

We often think about grey literature in terms of what it fails to be: not peer-reviewed; lacking bibliographic information or other useful metadata; escaping collection and preservation. But it is important to remember that the deficiencies of grey literature from one perspective may at the same time be advantages from another. Grey literature is an opportunistic creature of the dynamic ecosystem of publishing, and in particular the various, competing incentives and disincentives for publication. This means that there is a certain complexity — a double edge to grey literature.

For example we often associate grey literature with open access publishing, and clearly the desire to make research accessible in a timely fashion is one of the main incentives for producing a report or a working paper. But there is no necessary link between informal publishing and open access. As my colleague Amanda Lawrence has shown in her recent historical sketch of the issue, the desire to restrict or control access is also an incentive A hard-to-find technical report may be deliberately so: a great deal of research remains intentionally restricted in circulation, for commercial or other reasons. On the other hand, the relative simplicity of grey literature publishing practices makes preservation and access relatively straightforward compared to the increasingly restricted, platform-specific domain of commercially-published electronic books.

These pressures help to explain an apparently contradictory situation, where, while the academic research system is increasingly organised and formalized, with strong incentives for publication in peer-reviewed form, and evolving systems for measuring research quality and impact, at the same time we see a remarkable proliferation in informal academic publishing. 

The proliferation of grey literature has been developing for some time. Ten years ago, a few of us at Swinburne University’s Institute for Social Research set up Australian Policy Online, an online gateway to public-policy related new research. We created APO because we had seen the proliferation of websites created by mostly small research centres across Australia. Many but not all the centres were in universities. Lots of them were producing interesting work in the form of technical reports or working papers. Most of their websites, on the other hand, were not amazing examples of information architecture. The problem was that simply posting something online didn’t actually make it all that accessible. We played with the search engines of the time, and it was clear that lots of work was not easy to find if you didn’t already know about it, and it tended to become increasingly lost as time went by. So we built a database of information and links to these materials, organised them around broad policy areas, and hoped that people both within but especially outside universities would subscribe to a free weekly email listing of what was new. After many years and several technological revolutions, there are more than 17,000 subscribers to the newsletter, and 85,000 visitors to the site each month. Almost half of APO’s users are in the public sector, 22% are in education and 17% are in NGOs. We had no way of articulating it at the time, but we were effectively building a kind of policy library for grey research literature.

We keep improving APO, but we realized a while ago that one site can only go a small way towards solving the problems around grey literature. Some of these problems are to do with law and policy: grey literature is subject to copyright law, which currently prevents sites like APO from providing more useful linked, full text archives of relevant material. Some of the problems are technical. These range from the limitations of the tools creators and distributors use, to the absence of infrastructure for collection and preservation. Some problems are institutional: a lack of co-ordination. But there is also a lot we simply don’t know about who uses and produces grey literature, and about how we might use it better, and how its functioning in our research and innovation systems could be improved. I think we’ll learn a lot about those matters today.