Bit Rot & Long Term Access

Over the last weeks at several events Paul Wheatley of the British Library showed examples of bit rot in an image collection of the BL.

Planets did lots of R&D into preservation and long term access and provided background and breakthrough thinking on many of the technical challenges.

But Paul's tenacity on this subject bit rot triggered me to open a Problem, Requirement, Use Case sequence in the "what we need" section of the OPF wiki. 

http://wiki.opf-labs.org/display/REQ/Home

We need to make serious efforts to drill down into every individual potential element and risk for Bit Rot. Create Use Cases and take action on these Use Cases and develop practical tools and/or workflow to assure we can prevent Bit Rot. I am sure that many of the implications of the Use Cases are not necessarily big development projects and often more a case of creating scripts that can compare "checksums" or "open and save as new" type of activities inside a repository. Todays hardware is subject to a very short technical life cycle and by far not as reliable for keeping the integrity of the bits as we sometimes like to believe.

Understanding the challenge and a continuous discussion amongst experts who know how to mitigate the risks is crucial. Do look at this wiki entry as starting point to share thinking and resolution of Bit Rot problems

 

 

Vint Cerf's view on Bit Rot

Preservation Topics: 

Comments

Maurice de Rooij's picture

LOCKSS is the first thing that comes to mind when I read the word Bit Rot. The people at Stanford have done extensive research on this subject and we could learn a lot of them.

Some might oppose to how LOCKSS works ingestion-wise and/or workflow-wise, but IMHO the "Lots of Copies Keep Stuff Safe" principle is our main weapon in the fight against Bit Rot.

http://lockss.stanford.edu/lockss/Home

Yes, preventing bit rot is a requirement for digital preservation. In fact, unless you can do it well, everything else is a waste of time. To most people's surprise, it is an unsolved and (at large enough scale) unsolvable problem. I pointed this out in a 2007 blog post, a paper at iPRES2008 (at the British Library) and most recently in articles in ACM Queue and Communications of the ACM. The LOCKSS system is designed around the restrictions of copyright law. As a side-effect, it has lots of copies of the content. By contrast, most digital preservation systems want to maintain as few copies as needed to reach a reliability target. I'll be talking about this topic at Screening the Future 2011 next month.
Bram van der Werf's picture

In essence technology (hardware and software) all suffer the same challenge as traditional engineering products.  Maintenance and the need for it, is an immediate result from actual usage.

Recently at a Software Preservation event organised by JISC http://softwarepreservation.jiscinvolve.org/wp/2011/02/18/7-feb-outputs-... , I stated " Untended software is like an empty house in the jungle: it soon falls apart" The high humidity and and ants (as Andy Jackson tweeted during the meeting), will make the house rot. People living in the house would have fought against these attacks and they would have done this with relative standard maintenance processes such as ventilation and cleaning.

In many technology driven projects and processes in libraries and archives, maintenance is a highly underestimated activity. There is a  tendency to look for high level solutions that will help librarians and archivists without needing to know too much about the underlying technology and bits. While actually preventive maintenance (this requires understanding of what is under the hood) is needed.

Lockss is a fine solution for those institutes where it fits their preservation policy, it has data maintenance functionality built in.  Many institutes made different choices that better fit their policies, systems and resources. Hopefully all will have a high level of awareness that usage and maintenance are effective in preventing bit rot.

I strongly believe in preservation solutions and tools (http://lockss.stanford.edu/lockss/How_It_Works, http://dpsp.sourceforge.net/, http://wiki.opf-labs.org/display/KB/List+of+Digital+Preservation+Tools%2...  etc.) and a diversity is actually fine as well, as there are still too many unknown risks. A diversity of solutions is a very effective high level mechanism that prevents us from all making the same mistake. There are still too many uncertainties, making it hard to feel 100% confident about a solution.

But I would also strongly advocate for "technology and bit maintenance professionals" in libraries and archives, similar to paper preservation experts. Because lots of the maintenance tools are already there and only wait for skilled operators/engineers.

I was thinking about the tools that are available to us recently, and noticed that we seem to have a capability gap that belongs in this discussion.

At the moment I have tools that let me check for viri and other nasties, I have tools that let me 'measure' my object repeatedly to make sure its not changed, I have tools that will help me identify the format of the object, and to figure out the object:application relationship, I have tools that will help me extract metadata from the object so I can catalogue technical aspects it in detail... 

But I don't have a systematic way of validating that the object renders. I can do it manually, open the jpg, or pdf, or whatever, and I can manually verify that 'yes' the object rendered as I expected.

What would be fantastic would be a bilk 'loader' that have some feedback logging. You could present it with a pile of suitable objects, and it would return a validation message 'file.jpg opened without error and was presented to the jpg viewer with no detected problems'.

Would this be viable? Perhaps parts of it are a little hopeful… Could we really rely on an emulated renderer? Or even an emulated vision/cognition mechanism to support this render validation function?

I'm not sure, but I do know that as a repository grows in size the likelihood that we will ever have a ‘human’ to undertake enough ‘openability / renderability’ testing across my content decreases, (not least to fulfil a statistically significance number of tests to make it worth doing), a tool of this nature becomes paramount.