D-Lib Magazine

January/February 2013
Volume 19, Number 1/2

'Oh, you wanted us to preserve that?!' Statements of Preservation Intent for the National Library of Australia's Digital Collections

Colin Webb, David Pearson, Paul Koerbin
National Library of Australia

Point of contact for this article: David Perason, dapearso@nla.gov.au

doi:10.1045/january2013-webb

Abstract

Clarifying preservation intentions is likely to be a good starting point for preservation planning for diverse digital collections. This applies both in terms of identifying what needs to be kept and what does not warrant the use of limited preservation resources, and in terms of opening up conversations about what is required in order to achieve preservation intentions. This paper describes an approach being explored by the National Library of Australia to negotiate formal and reviewable statements of 'preservation intent' for each of the digital collections in its care with those responsible for those collections. The paper looks at the relationship with the widely discussed concept of 'significant properties', as well as the other benefits that the approach is delivering. The paper also looks at the preservation intent statements for archived web collections at the NLA as an illustrative case.

Introduction

A small tale to set the scene:

The other day, a young person asked a pointed question, 'Do you believe in significant properties, Grandpa?' The question was asked with a rather patronising smile commonly accompanying questions like, 'Do you believe in Santa Claus ... or fairies?'

I had to think for a few moments about how best to answer. 'Yes, I do, if by "believe in" you mean "think it is a useful concept likely to have practical relevance to digital preservation planning". But not if you mean "think it is the magic that will deliver all your wishes when you open your eyes". In fact, if you would like to sit down and listen, I will tell you what I think about it, and how the concept could be pragmatically applied—because that's what the National Library of Australia is exploring right now!'

Unsurprisingly, my young person decided he didn't have the time, and went off to laugh about silly old codgers who are stuck with impractical ideas that lead nowhere. I suspect he is probably still blogging on this fruitful subject.

Like most things to do with managing digital collections, effective ways of making preservation decisions are evolving. Let's be quite clear about what we mean by that statement: we (the digital preservation community) have no settled, agreed procedures for the full range of digital preservation challenges, nor even tentative plans we confidently expect to ensure adequate preservation for the next 100, 200 or 500 years. We don't know with certainty what we have to do now, let alone what will have to be done in 20, 50 or 100 years in the future. So we are necessarily speculating and making up proposals for action.

Obviously there are some contexts in which this is less true. Some managers of digital information have several decades experience of successfully looking after data collections and maintaining satisfactory access. However, we are talking about the library field—and to minimise the likelihood of offending anyone else, these comments will focus on the National Library of Australia (NLA). When it comes to preserving access to the digital collections that the NLA has corporately decided to collect, we don't know precisely what we will need to do. In fact, for some collections we barely even know the extent or detail of what we have and are collecting.

That's not quite the same as saying 'we don't know what we are doing'. We are looking for practical approaches that appear most likely to work, while recognising we are unlikely to foresee and forestall every problem. We are experienced experimenters familiar with operating speculatively.

We take the view that methods and solutions do not exist in isolation: consideration of them is meaningless without the context of what has to be achieved.

The latest manifestation of this approach to preservation planning is the development of what we at the NLA have labelled 'preservation intent statements', which we have been talking about since 2009 ¹. These statements record the organisation's intentions for specific classes of digital content expressed as a result of consultation between curators (or collection managers) and digital preservation specialists.

This concept is closely related to the idea of 'significant properties' that has been widely discussed in the preservation community for well over a decade. One source described significant properties as:

'... those aspects of the digital object which must be preserved over time in order for the digital object to remain accessible and meaningful. An institution with curatorial responsibility for digital objects cannot assert or demonstrate the continued authenticity of those objects over time, or across transformation processes, unless it can identify, measure, and declare the specific properties on which that authenticity depends. Nor can it undertake the preservation actions required to maintain access to those objects, unless it can characterise their current technical representations with sufficient detail'. ²

At least in part, the approach described in this paper is a response to the difficulties we and others have encountered in trying to apply the significant properties concept as a starting point for preservation planning ³.

We have come to a tentative conclusion that recognising and taking action to maintain significant properties will be critical, but that the concept can be more of a stumbling block than a starting block, at least in the context of our own institution. We believe reference to significant properties in preservation planning requires some prior consideration of both the purposes for which digital content has been collected and the purposes of providing preservation attention. In effect, we are asking how can we know what attributes of digital materials we need to preserve if we haven't articulated why we are preserving them?

The purpose of this paper is to explain the NLA's approach to developing agreed statements of preservation intent, as well as the apparent benefits, both generally and with specific reference to managing web archives. In doing so, the NLA is inviting dialogue with others with an interest in progressing practical preservation planning and exploring how such policy-oriented planning can be effectively systematised.

Experience suggests that stating the obvious opens a door which we can all see but to which none of us has yet been able to find the handle. Our preservation intent methodology looks like a very modest step because that's exactly what it is. But we believe it is both necessary and helpful for getting started in practical preservation planning.

Preservation Intentions and Preservation Intent Statements

The NLA's digital collections are made up of digital objects collected by all curatorial areas of the Library. These collections also include copies created by the NLA for various access or management purposes.

Like many other public institutions but perhaps unlike some other collectors of digital content, the NLA collects specifically for the purpose of providing access. While access to some content may be restricted for legal or other reasons, the Library's default collecting purpose is to support ongoing access to the nation's documentary heritage of information resources likely to have enduring value. Therefore, preservation issues and judgments usually include questions about how adequate access can be defined and how long access needs to be maintained.

The Library's preservation intent methodology is simply to engage collection curators in making explicit statements about which collection materials, and which copies of collection materials, need to remain accessible for an extended period, and which ones can be discarded when no longer in use or when access to them becomes troublesome. Curators are also asked to make broad statements clarifying what 'accessible' means by stating the priority elements that need to be re-presented in any future access for each kind of digital object type in their collections.

This is basically a call to take responsibility for deciding what happens to the collections. In the first instance, it is largely an invitation to curators to analyse the objectives of their collections and to identify why different parts of their collections were acquired. In practice, curators are being asked to consider the implications of their collection development policies and collecting decisions in terms of the realities of providing and maintaining access.

Importantly, the approach aims to accommodate the specific needs and characteristics of each collection, while looking for common ways of describing things so that patterns can be efficiently and effectively recognised and planned for. This approach accepts that users and values will almost certainly change over time. However, it assumes that useful action is more likely to arise from making the best estimates we can for the foreseeable future than it is from refusing to make any decisions about priority uses and values.

One difficulty of the approach is that those responsible for collecting often see things in terms of genres, workflows and intellectual entities, whereas collection management and preservation decisions typically deal with types of file formats and individual files. Consequently, the same file types may be perceived as having different roles and importance in different collections. While this is a challenge, an explicit aim of the approach is to bridge the distance between these different, and equally legitimate, conceptualisations of the collections.

All preservation programs, whether dealing with digital or non-digital collections, need to understand the objectives they are set up to serve. All preservation programs are confronted with the challenge of articulating how long collection objects need to be kept, and what properties are significant to their value and function and therefore need to be maintained. Digital preservation programs have highlighted the importance of these issues but they are just as real in traditional library preservation programs concerned with storing, protecting, repairing and copying other kinds of materials such as books, manuscript papers and pictures ⁴.

However, this is an especially pressing issue for digital collections because of their scale, the uncertain timeframes for taking action, the risk of deterioration not being apparent until it is too late, and the likely costs of recurring preservation treatment.

Significant Properties and Preservation Intent Statements

The NLA's Digital Preservation Program was an early and convinced advocate of a significant properties approach, recognising that preservation planning decisions about methods, quality assurance (QA) and accountability hinge on clear statements of what has to be achieved (even though there are some who argue that this is impossible) ⁵.

There are no evident orthodoxies or assumed outcomes to build upon. Curators who assume that a service provider—external or even internal—knows what 'preservation' means for diverse digital collection objects can expect to be disappointed. Responsible officers in the chain of planning, decision, action and evaluation need to educate themselves on the intentions, desired outcomes, challenges and possibilities, and then set about broadening and deepening the understanding of everyone else in the planning and management chain. In this context, we looked—and still look—to some concept of significant properties as a key to preservation planning.

However, the NLA experienced great difficulties in defining the significant properties of various kinds of digital objects when approached as an abstract, 'once and for all time' exercise, and eventually put the concept to one side until a more promising way of applying it could be found.

Initially, the approach of working with curators to identify their preservation intentions was intended to include discussion of the significant properties needing to be maintained. However, it quickly became clear that these two issues should be dealt with separately, for two main reasons:

1. Firstly, because the complexity of defining measurable levels of properties required to fulfil a preservation intention almost inevitably became a barrier to engaging with the critical but simpler questions of:
   • How can we describe and classify a collection, which makes sense to the collection managers?
   • Who is responsible for authorising what happens to any specific collection materials?
   • Do we need to keep these materials accessible? If so, for how long?
   • In general terms, what would an adequate level of accessibility look like (such as, integrity or fidelity of bits, viewing functionality, editing functionality, navigating, and the ability to manipulate content)?
   • Who is responsible for preservation actions (i.e. digital preservation specialists, curators, ICT staff or another party)?
   • Can we identify any issues, including operational or non-collecting issues, which may hinder preservation efforts in the future?

2. Secondly, because the full definition of significant properties will be most pertinent closer to the time of taking preservation action. Potential kinds and levels of change to digital objects can be tested and evaluated using the action options available at the time of such action. While preservation programs will ultimately have to define the desired significant properties they are responsible for preserving, we now believe this is likely to be best informed by, and based on, practical experience with preservation actions.

Current Activity and Methodology

The NLA's Digital Preservation Program has recognised a need to develop a more comprehensive, sophisticated, accurate and useful understanding of what it has to manage. This began with a collection profiling exercise, which gathered broad information about general rates of collecting, what the digital collections contain, how they are organised and managed, and key issues for different collection areas ⁶. The exercise led naturally to the question of preservation intentions—how can they be teased out and how could they be expressed as formally agreed preservation intent statements?

The next step involved refining the types of questions which would be useful for the management and preservation of digital objects (see above). The various collection curators and a digital preservation specialist discussed and came to an agreement on how their collection could be described and characterised. This dialogue resulted in the drafting of 'plain language' statements in consultation with the NLA curators and collection managers. These were recorded on a wiki. Although it has taken some time to engage across the organisation, the NLA now has statements of preservation intent at some level of detail for all of its digital collections.

This activity is a key to preservation planning ⁷. Without it, we are left floundering between assumptions that every characteristic of every digital item has to be maintained forever (almost certainly an impossible expectation), and assumptions that it is good enough to store data safely and let future users worry about how to access it (almost certainly an inadequate response). The NLA believes that its responsibilities lie between these extremes, seeking as good an understanding as we can currently develop about what needs to be achieved, so we can start identifying priorities and plans for achieving preservation objectives.

In summary, this approach doesn't try to solve the deep, dark problems of digital preservation but it does give us some more information and some vital organisational engagement and recognition of shared responsibility to work with.

Benefits

The apparently modest nature of the preservation intent statements methodology belies some very significant potential benefits. In the NLA's case, a number of benefits are already emerging, including the following.

1. It is common for preservation discussion and debate to move straight into a focus on the 'how'. This is unsurprising, given the necessary involvement of and reliance on the technical development and expertise of the ICT profession. However, the preservation intent approach allows us to focus some attention on questions that are at least as critical as 'how', including questions of what we have to preserve, why we want to preserve it, and when action will be needed to pre-empt or mitigate threats to the objectives we value.
2. Engagement of curators in making decisions about how their collections should be managed has led to growing recognition of accountability for collection management decisions.
3. Consideration of other stakeholders including users and what they might be looking for in using digital collections.
4. Gaining a better understanding of the digital collections, including development of some common ways to describe them and opening up conversations about their current and intended status.
5. Identification of materials that have been 'parked' on physical carriers (such as CD-Rs, floppy disks and data tapes), network drives or email attachments, rather than in managed mass storage.
6. Progressive recognition of and engagement with the level of detail required for preservation management. For example, different manifestations or versions of the same content may require different preservation intent statements within the same collection, because of the presence of annotations, change tracking or different sources.
7. Understanding the purposes of different kinds of copies, which is quite an issue in the NLA where we have many types of copies with ambiguous descriptors such as 'preservation copy', 'master copy', 'modified master' and so on. These descriptors may refer to different functions in different collections and workflows, causing confusion in preservation planning.
8. Creating an opportunity to look for common issues and common patterns across and between collections.
9. Creating an opportunity to map out possibilities and have a first go at formalising some digital preservation processes.
10. Starting a process to confront the issue of what acceptable loss or change might be.
11. Recognition across the organisation of the need for good preservation practices at all stages of the collection life cycle. These practices include good documentation of purpose, ways of documenting losses and changes, and conscious decisions about the extent to which a 'reversibility' principle can be applied.
12. Demystification of digital preservation across the organisation and a basis for more effective communication with corporate management about issues, realistic expectations, priorities and timeframes for allocating resources.
13. Recognition that a commitment to ongoing review of preservation intent statements is intrinsic to the process, as intentions, values, users and uses will change over time.
14. Keeping the door open for technical development and new approaches. This is a starting point for now and for the near future, that does not necessarily close off options for doing things differently in the long term.
15. By focusing on intentions, the approach creates a useful space between objectives and actions that should allow us to critically evaluate available action options when they are needed.

Preservation Intent Statements for Archived Web Collections

The NLA has developed preservation intent statements for all of its digital collections. However, web collections present some interesting challenges that warrant special examination because of the volume and complexity of content over the creation of which the preserving institution has no control. Moreover, preservation planning for web collections may be less well addressed globally than it is for many other kinds of digital collections. At the NLA, web collections are considered to be a high priority for preservation action, but are also likely to be among the most challenging.

The NLA's web collections fall into three broad categories.

1. Selectively gathered 'titles' which make up PANDORA, Australia's Web Archive ⁸.
2. 'Whole Domain Harvests' which aim to capture a broad periodic snapshot of the Australian Web Domain.
3. Coherent bulk collections, such as '.gov.au' seedlist derived collections.

Of these three collection types, for legal reasons only the PANDORA content is currently available for public access, although work is underway to make the '.gov.au' collection accessible.

Content in the NLA's web collections is obtained by harvesting copies from the web. However, the NLA's processes frequently generate archived copies which are not completely accurate representations of what was presented on the live web. In the case of our PANDORA collections, the software we use rewrites the code, so we have archived copies that are intrinsically different from the originating live web versions ⁹. On the other hand, in the NLA's experience of collections that were acquired using the Heritrix web crawler the results are more direct copies but the software often produces incomplete or inadequately rendered copies ¹⁰. Either way, we have had to face the fact that the archived copy is different in some degree from the entity on the live web.

This produces some obvious questions regarding which aspects (properties, attributes, characteristics) should be given priority for preservation, and how much change may be acceptable in an already modified representation.

A second challenge in determining preservation intentions for web collections is the combination of scale and diversity. The Library holds around five billion files or 184 terabytes of data and currently collects around one billion files or more than 40 terabytes of data each year ¹¹, so there is a huge amount of material to be considered and great diversity to be accommodated. This diversity refers to the content and the purposes for which it was collected; it also extends to the file formats and structures used. Within the PANDORA archive, for example, more than 300 file formats have been identified, while many formats have not yet been identified because of the limitations of format identification tools and the idiosyncratic ways in which many objects were originally created ¹². The scale and diversity of the existing web collections make it much more difficult to establish classes of materials than for more homogenous collections such as the NLA's Pictures or Oral History collections.

A third complicating factor for preservation planning is the complexity of the individual objects being considered. Most web pages are compound or complex objects made up of numerous digital entities: HTML, executable programs, style sheets and so on. Determining preservation intentions for each of these and accounting for their individual contributions to the overall functioning and value of the 'whole composite' (if such a thing exists) takes us back into the heart of the significant properties problem we have already referred to. We have had to look for ways of making useful and meaningful statements of intentions that account for this kind of complexity without being debilitated by it.

Even for relatively simple web objects, those responsible for articulating preservation intentions and planning action need to consider requirements for maintaining bit-level objects, content, and context such as links, possibly to a greater extent than for most other kinds of collected digital materials.

A fourth factor expected to complicate preservation action, although it is perhaps less challenging for identifying preservation intentions, is the wide variance in tolerance of tools for browsing, collecting and rendering web content. As mentioned above, many of the harvested and archived web files are idiosyncratic, leading to unpredictable changes when preservation action such as migration is applied to large numbers of files. This will seriously complicate the translation of intentions and significant properties into consistent, automated action.

Finally, the use of large-scale container file formats such as WARC ¹³, which can hold many millions of files, invites preservation planning at the container level, potentially leaving the diversity of formats, intentions and properties unaddressed within the container.

Facing these challenges, the NLA's approach to identifying and articulating its preservation intentions for its archived web collections is described in the following section. For the purposes of this paper, we are going to consider only the PANDORA archive, as we have more information about it than our other web collections and it contains our earliest content. The archive's cultural significance is reflected in its inclusion in 2004 on the UNESCO Australian Memory of the World Register ¹⁴.

Summary Preservation Intent Statement for the NLA's Selective Web Collection

What follows is the example of the Preservation Intent Statement for the NLA's Selective Web Collection.

Practical Application

Discussion of preservation intent is not a theoretical exercise. A number of file types in the NLA's web archive are already known to present access problems, given the way the content is collected and managed in the archive. There are several problematic formats including:

1. RealMedia files—the archive includes many sites containing .rm and .ra files delivered in their original live web environment via a master file (.ram) which pointed to the actual media files hosted on a RealMedia server. In order to collect such files, the NLA required publishers to separately deposit the relevant .rm and .ra files, which the Library then hosted on an Real Time Streaming Protocol (RTSP) enabled server. Internal support for this server has failed and the NLA is currently not able to deliver the RealMedia content it has collected.
2. Virtual Reality Modelling Language (VRML)—content in this format requires a plug-in (such as Live Picture Viewer), which has not been incorporated into browsers. Content in this format cannot be delivered from the NLA web archive at this time.
3. Shockwave—older sites with Shockwave files, such as Director (.dir), appear to be missing the required plug-in component.
4. Quicktime VR (Virtual Reality)—content no longer seems to function in the current operating environment.

This means that the NLA is facing a number of practical preservation use cases where we need to decide whether to take action, what kind of action to attempt and how to evaluate the outputs of action. Our current Preservation Intent Statements tell us if we want to keep the content accessible for an extended time period; if we want to view the content; and whether to attach importance to other access modalities such as the ability to manipulate content.

Put simply, such a preservation intent classification may lead us to assess the following action options:

1. Take no action; or
2. Replace the failed access software; or
3. Migrate the content to another format that does preserve the presentation of the properties we decide are significant; or
4. Work out some way to emulate the presentation of those significant properties of the content; or
5. Maintain the bits and documentation that will support action in the future when more effective action options may become available.

The NLA is currently engaged in a large-scale effort to build workflows and infrastructure that will support a range of preservation management requirements for all of its digital collections through a multi-year Digital Library Infrastructure Replacement project. Preservation intent commitments, along with ongoing risk monitoring and other factors to be recorded in knowledge-bases, will drive workflows in the delivered preservation management systems.

While there are potential challenges in coding intent information for programmatic interrogation and use, within the next few years the NLA expects to have operational systems in place using its preservation intent information.

Conclusion

The approach described in this paper is based on a conviction that methods and solutions in digital preservation do not exist in a policy vacuum. Rather, they are only meaningfully discussed as solutions to problems that threaten or frustrate the organisation's explicit access intentions.

This approach still refers to significant properties as a critical part of the preservation planning process. We believe it puts the full identification and evaluation of significant properties into a more useful context: that is, being later in the decision-making process.

At the very least, we aim to be in a position to know whether preservation action will be needed and whether a bit-level preservation copy will be good enough when access to content is lost.

As with most preservation processes, evaluating and articulating preservation intentions is likely to be an ongoing process requiring proactive management and periodic review. Our current conception of it still requires further development but it has given the NLA a useful start in real preservation planning.

Without some kind of understanding between curators and preservers, we are doomed to recurring nasty surprises based on mismatched expectations.

Acknowledgement

The authors wish to thank Libor Coufal and Tina Mattei from the NLA for their suggestions and support. We also wish to thank Lynn Benson (NLNZ), James Doig (NAA) and Cal Lee (UNC) who provided helpful comments. We also appreciate the encouragement provided by Gina Jones (LoC), Andrea Goethals (Harvard Library), Clément Oury (BnF), Barbara Sierman (KB) and Tobias Steinke (DNB), all co-members of the IIPC PWG (International Internet Preservation Consortium, Preservation Working Group).

Notes

¹ See Long, A. S. and Pearson, D. (2009) 'I Say Emulate; He Says Migrate—Are Emulation or Migration Feasible Preservation Strategies?' (Presentation for the IIPC (International Internet Preservation Consortium) Open Day, San Francisco on 7 October 2009) National Library of Australia Staff Papers. See also, del Pozo, N., Long, A. S. and Pearson, D. (2010) 'Land of the lost': A Discussion of what can be Preserved through Digital Preservation'. Library Hi Tech, Vol. 28, Issue 2, pp. 290—300.

² InSPECT Investigating Significant Properties of Electronic Content. See also, the concept of Transformation Information Properties, similar to significant properties, in Reference Model for an Open Archival Information System (OAIS). Magenta Book, Issue 2, June 2012.

³ See Bradley, K. (2007), 'Defining Digital Sustainability'. Library Trends 56 (1): 148—163. See also, Low, J. T. (2011), 'A Literature Review: What Exactly Should We Preserve? How Scholars Address this Question and Where is the Gap'.

⁴ Virtually all preservation treatment decisions—such as deciding whether and how to rebind books, bleach stains on paper, deacidify paper artefacts and so on—involve or should involve some consideration of what gives the object value to users, so that this can be protected and preserved to the full extent possible, while minimising the extent to which the value of the object is jeopardised or compromised by preservation treatment. Similar principles apply to copying treatments such as microfilming and digitisation. Judgment about the adequacy of a particular copy as a preservation surrogate depends largely on the extent to which the valued components of the original have been captured and re-presented. Without any agreed sense of priorities regarding what is of value, preservation action decisions almost inevitably lead to conflict, as happened when library microfilming programs in some cases destroyed newspapers that had been—at least in the judgment of some users—inadequately represented by microfilm copies.

⁵ See Yeo, G. (2010), '"Nothing is the Same as Something Else": Significant Properties and Notions of Identity and Originality'. Arch Sci (2010) 10: 85—116.

⁶ Elford, D. (2009) Collection Profiling Project. Internal National Library of Australia paper (unpublished).

⁷ See Becker, C., Kulovits, H. and Rauber, A. (2010), 'Trustworthy Preservation Panning with Plato'. ERCIM (European Research Consortium for Information and Mathematics) News, Vol.80, January 2010, pp. 24-25.

⁸ PANDORA, Australia's Web Archive.

⁹ Content for the PANDORA Archive has been collected using three harvesting softwares since the program was established in 1996. At first the Harvest indexing software was used and then for a brief period the offline browser WebZIP was also used. Most of the content however, since the late 1990s, has been collected using HTTrack. As an offline browser HTTrack rewrites links as part of the harvesting process so that they function as relative links in the context of the Archive. The software also changes URLs that include dynamic database calls into a static HTML form that is functional in an 'offline' context. This means that archive URLs may have a different file name, URL structure or file extension than the original URI. Moreover, some quality assurance work can involve curators making some changes to code within the page in order for the archived instance to be as complete and functional as can be reasonably achieved. For example, in order to make FLV format videos to work some customised 'embed' code is added to replace the original script in order to deliver the video file within the context of the archived page. Finally, a post-processing program written for PANDORA re-writes external links so that they deliver a message indicating that an external linked page has not been archived, also providing a link to the live page address.

¹⁰ Heritrix does not permit the same degree of 'hand-crafted' quality assurance work as is undertaken for the PANDORA collection since the content is stored in WARC packages rather than as static HTML files in a file system structure. Consequently there is a far greater reliance on and acceptance of what Heritrix is able to collect and also on what the archival delivery system (Wayback) is to be able to render. While some quality assurance and 'patching work'—identifying and adding missing files—is undertaken on the Heritrix crawls for NLA collections (i.e. whole .au domain crawls and gov.au seed list crawls) it is not to the same degree of scrutiny that is done with the PANDORA model, not the least because Heritrix crawls are conducted for much large scale collections rather than for the discrete site harvesting for PANDORA that lends itself to more quality assurance scrutiny and manual intervention.

¹¹ The figures quoted include the Australian domain harvests of 4.8 billion files or 177 terabytes of data and the PANDORA collection of around 160 million files or seven terabyes of data. It should also be noted that the figures for PANDORA are only indicative based on the derived display copy of the archival content and when the various master copies are also counted the actual PANDORA figures may perhaps be multiplied threefold.

¹² Elford, D. (2011) PANDORA File Format Analysis. Internal National Library of Australia paper (unpublished). Elford's paper identifies idiosyncrasies such as duplicate filepaths and filenames, inconsistent mime types and extensions, formats apparently non-compliance with standards, etc.

¹³ WARC ISO 28500:2009 Standard.

¹⁴ UNESCO Australian Memory of the World Register, Register No. 14.

¹⁵ The potential for confusion with the NLA's current descriptors for digital copies should be evident from this discussion.

¹⁶ An analysis of the first three Australian domain harvests (2005—2007) established these percentages of broadly aggregated mime types; see Koerbin, P. (2008), 'The Australian Web Domain Harvests: A Preliminary Quantitative Analysis of the Archive Data'. As PANDORA, although selective, may be understood as representing a sub-set of the larger domain harvest collections, the same broad percentage breakdown has been extrapolated to the PANDORA collection for the purpose of this paper.

About the Authors

Colin Webb retired in September 2011 from the position of Director, Web Archiving and Digital Preservation at the National Library of Australia after more than 30 years in preservation management roles in the Library and the National Archives of Australia. He introduced the first digital preservation program at the NLA in the early 1990s, and in 2002 put together the first UNESCO Guidelines for the Preservation of Digital Heritage. He retains a keen interest in policy responses to digital preservation challenges and the changing nature of information.

David Pearson is currently Manager of the Digital Preservation section at the National Library of Australia. He has worked in the digital preservation field for the last 11 years and in cultural institutions for over 19 years. During this time he has project managed or worked on many aspects of digital preservation, including policy, software and tools development, system and metadata requirements, data recovery from obsolete media, and building and fitting out computer rooms. David holds an honours degree in archaeology from the Australian National University, Canberra. He has written a number of articles in academic journals on both conflict archaeological and digital preservation issues.

Paul Koerbin has been involved with the development and operation of the National Library of Australia's web archiving program since its inception the late 1990s. He was part of the team that developed one of the first workflow systems for selective web archiving and currently manages the web archiving program at the NLA. He has published papers and given many presentations, in Australia and overseas, on web archiving operations and practice. He holds a graduate qualification in library and information studies from the University of Tasmania and a PhD from the University of Western Sydney.