From digital deluge to digital dust

Posted by Amanda Lawrence on 26 February 2013

By Amanda Lawrence

Article first published in the Australian Library and Information Management Association's Incite Magazine, Vol 34 Issue1/2 (Jan/Feb 2013)

 

No longer are publishers the primary conduit through which content must pass before it is disseminated to the world. This is a radically new paradigm and despite our best efforts, libraries (and many other related institutions) have not been able to keep pace with the impact this shift has had on collection development and long term access.

We now live with a deluge of content – a complete contrast to the old paradigm of scarcity and cost – but we do not yet have any real solutions for collecting online content or for preventing its subsequent disappearance from the internet. This problem goes to the heart of what libraries have traditionally done – selected, collected and provided access to content, both today and in the long term.

This issue manifests in various ways. One important area, and one that is attracting its fair share of attention in Europe and the USA, is that of informal publishing of research and public policy documents – also known as grey literature.

Grey literature as a term may not be familiar to some librarians but it is well-known for others, particularly those in special libraries and researchers in various disciplines. Grey literature is a collective noun for reports, papers and other documents produced by organisations including governments, universities, NGOs and professional companies, outside of commercial publishing channels. It often lacks bibliographic control, can be hard to find and difficult to catalogue. Yet in the area of public policy a great deal of extremely important material that is regularly discussed in the media, parliament and cited in research articles is produced and disseminated informally as grey literature.

The internet is now estimated to contain over 14 billion pages and growing every second. From early on librarians responded quickly to the abundance of online content with the creation of web archives. While extremely valuable, web archiving has not been sufficient for providing access to many digital documents, some of which are similar in length, quality and importance to formally published articles and books. For the sake of our democratic processes, and to protect the very fundamentals of scholarship and evidence, these documents need to remain publically accessible, searchable, citable and locatable on a long term basis.

The digital deluge is now turning to digital dust as a great deal of important content has not been archived and is no longer available online. There are estimates that at least 30% of online content has disappeared over the last decade and this is not just facebook updates and celebrity tweets but also government documents and information, policy material, research, statistics and more. The reasons for this situation are multiple: on the one hand producers are failing to take care of their content - websites are updated, content is moved and removed at whim; on the other hand libraries are too often unaware, unwilling or unable to collect and catalogue digital documents.

There are many complex issues to overcome in resolving this now dire situation including the limitations of Australian copyright law, questions of financial responsibility and document management, selecting and evaluating content, developing adequate infrastructure, metadata systems and more.

Through my work on Australian Policy Online, a digital library of policy grey literature, we have developed a research project funded by an Australian Research Council linkage grant called Grey Literature Strategies (greylitstrategies.info). Led by Prof Julian Thomas and Prof John Houghton with partners from the National Library of Australia, the National State Libraries Australasia, the Australian Council for Educational Research and the Eidos Institute, the project aims to enhance the value and access of policy grey literature in Australia. This research taps into the Reinventing Libraries project of NSLA as well as having great relevance for academic libraries, special libraries, online collections and repositories.

I will be presenting on this issue in more detail at the Information Online conference in February and will also be conducting a workshop, with Jessica Tyndall, investigating how librarians and information professionals could better respond to the complex problems presented by online content. I hope Incite readers will join us at this session as any solutions will require a collaborative and collective response. If you can’t make Information Online please get in touch anytime if you would like to know more or contribute to the research.

This article was first published in the Australian Library and Information Management Association's Incite Magazine, Vol 34 Issue1/2 (Jan/Feb 2013)