Feed aggregator

The following is a guest post by Jennifer Clark, intern with NDIIPP at the Library of Congress.

I am a first-year Master of Science student at the Graduate School of Library and Information Science at the University of Illinois at Urbana-Champaign through their online distance-learning LEEP program. I am pursuing a specialization in Data Curation which focuses on data management best practices with course work in digital preservation, information modeling and representation objects. I am particularly interested in how institutions decide what needs to be saved and how, as well as how those decisions affect patron access and their future interaction with the digital object.

Though digital preservation issues have recently grabbed my attention, I have been familiar with many digital preservation problems for years. As a hobby, I regularly participated in National Novel Writing Month, a month-long race that occurs in November to frantically write a novel of 50,000 words in 30 days. To date, I have written four draft novels, and all were born-digital. During those late nights, I became painfully aware of the stability issues that can occur with born-digital objects and how my computer might decide to interact with my precious, burgeoning “Great American Novel.” Without fail, right around 25,000 words, my word processor would corrupt my file. To this day, I have multiple backups of each of my novels in a variety of formats, but only after many heart-stopping incidents.

When I began my degree I started to wonder, if I had so much trouble preserving and stabilizing four relatively small files that held such personal significance, how do we as a profession begin to preserve and stabilize the millions – or billions – of files that make up our national digital heritage? Moreover, though I immensely valued my novels because I had a blood, sweat and tears connection with them, did I feel the same way about digital documents that did not hold a sentimental value, like my tax returns or my lease? How do librarians assist people in assigning value to intangible, impersonal and quickly-fleeting digital objects?

There has been a cultural shift in the last few years that has deemed digital objects as something “less-than.” Stories run weekly about how eBooks are poor substitutions for real books, and many musical artists are going back to releasing vinyl, believing the experience to be somehow better than listening to an MP3. We even attempt to make our digital photographs look like they were taken by an old Polaroid with the help of tools like Instagram. Admittedly, there is something to be said for experiencing an original, but what happens when the digital object IS the original? Do we still value it less?

We have already begun the tough work of ensuring that our traditional cultural heritage will be preserved for many generations. There is no argument that President Lincoln’s letters or a Civil War soldier’s accounts should be preserved. It’s easy to connect to physical items because they live in our physical realm – our bookshelves, our walls, and our museums, but digital objects often feel too-far removed. They are separated from us by our computers, our software, and our social media. Although we have discovered ways to slow-down or halt deterioration of traditional information objects like letters and photographs, the same cannot be said of the deterioration of born-digital objects. In many ways, these objects are much more fragile, and we should be in a much greater rush to preserve.

As part of the GSLIS Alternative Spring Break program, I’ll be conducting interviews with the people at NDIIPP to better understand how they propose we tackle our digital preservation problem and begin to value our digital objects. I want to know their opinions on the progress being made with digital preservation efforts as well as how their personal projects fit into the big picture. I hope to gain some insight into how we can accept our new digital norm to preserve what is most important before it’s too late, and I’ll be writing two blog posts to summarize my findings.

If there is one lesson that has been reinforced again and again in my short time at GSLIS, it is that our future roles will not be librarians, but advocates. I hope my week at NDIIPP will teach me how to be a better advocate for digital preservation at home, at school, and in my community.

 

Page edited by Becky McGuinness

View Online Becky McGuinness 2013-03-19T15:53:17Z

Paul Wheatley drew attention to this question on Stack Exchange yesterday:

“PDF is almost a de facto standard when it comes to exchanging documents. One of the best things is that always, on each machine, the page numbers stay the same, so it can be easily cited in academic publications etc.

But de facto standard is also opening PDFs with Acrobat Reader. So the single company is making it all functioning fluently.

However, thinking in longer perspective, say 50 years, is it a good idea to store documents as PDFs? Is the PDF format documented good enough to ensure that after 50 years it will be relatively easy to write software that will read such documents, taking into account that PDF may be then completely deprecated and no longer supported?”

I tried to respond, but fell foul of Stack Exchanges login/password rules, which mean I’ve created a password I can’t remember. And I was grumpy because our boiler isn’t working AFTER it’s just been serviced (yesterday, too), so I was (and am) cold. Anyway, I’ve tried answering on SE before and had trouble, and I thought I needed a bit more space to respond. My short answer was going to be:

“There are many many PDF readers available implemented independently of Adobe. There are so many documents around in PDF, accessed so frequently, that the software is under constant development, and there is NO realistic probability that PDF will be unreadable in 50 years, unless there is a complete catastrophe (in which case, PDF is the least of your worries). This is not to say that all PDF documents will render exactly as now.”

Let’s backtrack. Conscious preservation of artefacts of any kind is about managing risk. So to answer the question about whether a particular preservation tactic (in this case using PDF as an encoding format for information) is appropriate for a 50-year preservation timescale, you MUST think about risks.

Frankly, most of the risks for any arbitrary document (a container for an intellectual creation) have little to do with the format. Risks independent of format include:

• whether the intellectual creation is captured at all in document form,
• whether the document itself survives long enough and is regarded as valuable enough to enter any system that intends to preserve it,
• whether such a system itself can be sustained over 50 years (the economic risks here being high),
• not to mention whether in 50 years we will still have anything like current computer and internet systems, or electricity, or even any kind of civilisation!

So, if we are thinking about the risks to a document based on its format, we are only thinking about a small part of the total risk picture. What might format-based risks be?

• whether the format is closed and proprietary
• whether the format is “standardised”
• whether the format is agressively protected by IP laws, eg copyright, trademark, patents etc
• whether the format requires, or allows DRM
• whether the format requires (or allows) inclusion of other formats
• the complexity of the format
• whether the development of the format generally allows backwards compatibility
• whether the format is widely used
• whether tools to access the format are closed and licensed
• whether tools to access the format are linked to particular computer systems environments
• whether various independent tools exist
• how good independent tools are at creating, processing or rendering the format

and no doubt others. By the way the impact of these risks all differ. You have to think about them for each case.

So let’s see how PDF does… no, hang on. There are several families within PDF. There’s the “bog-standard” PDF. There’s PDF/A up to v2. There’s PDF/A v3. There are a couple of other variants including one for technical engineering documents. Let’s just think about “bog-standard” PDF: Adobe PDF 1.7, technically equivalent to ISO standard ISO 32000-1:2008:

• The format was proprietary but open; it is now open
• it is the subject of an ISO standard, out of the control of Adobe (this might have its own risks, including the lack of openness of ISO standards, and the future development of the standard)
• it allows, but does not require DRM
• it allows, but does not require the inclusion of other formats
• PDF is very complex and allows the creation of documents in many different ways, not all of which are useful for all future purposes (for example, the characters in a text can be in completely arbitrary order, placed by location on the page rather than textual sequence)
• PDF has generally had pretty good backwards compatibility
• the format is extremely widely used, with many billions of documents worldwide, and no sign of usage dropping (so there will be continuing operational pressure for PDF to continue accessible)
• many PDF creating and reading tools are available from multiple independent tool creators; some tools are open source (so you are not likely to have to write such tools)
• PDF tools exist on almost all computer systems in wide use today
• some independent PDF tools have problems with some aspects of PDF documents, so rendering may not be completely accurate (it’s also possible that some Adobe tools will have problems with PDFs created by independent tools). Your mileage may vary.

So, the net effect of all of that, it seems to me is that provided you steer clear of a few of the obvious hurdles (particularly DRM), it is reasonable to assume that PDF is perfectly fine for preserving most documents for 50 years or so.

What do you think?

Who are you?

 I am Catherine Jones. I am an Information Systems Project Manager in the Scientific    Computing Department at the Science and Technology Facilities Council (STFC) in the UK. 

STFC is one of the UK seven Research Councils and provides both funding for research in Astronomy, Particle and Nuclear Physics and Space Science and large scale scientific Facilities (http://www.stfc.ac.uk/home.aspx).

Your role in SCAPE?

I am the STFC project manager responsible for planning and coordinating all STFC activities on SCAPE. STFC contributes in four main areas in SCAPE: we lead the work in the Research Data Testbed, we build tools for preserving research data, we are part of the policy representation discussions and we lead work in developing guidelines for best practices in preservation of scientific data in the context of SCAPE work.

At the moment I am working on Policy Representation. There are two aims of this activity. Firstly to identify and provide guidance on which topics need to be considered and addressed within preservation policy for a whole organization or a particular content/collection to assist those people who need to write policy in this area. The second is to produce a set of machine understandable statements and potentially actionable statements which can be used by the PLATO planning tool and the SCOUT watch tool. I am working on describing a process to enable this translation of natural language policy into the machine understandable statements, together with a sample set. This is a challenge as humans don’t need every implicit fact made explicit as computers do!

Why is your organisation involved in SCAPE?

STFC is publically funded and hence the data produced and managed here should be preserved for the long term. This data forms part of the Record of Science alongside scholarly articles. By participating in the SCAPE project with partners across other sectors we can share experience and practice.

What are the biggest challenges in SCAPE as you see it?

A challenge that the Research Data Testbed is starting to consider and work on is the preservation of the context for research data. For many types of research data it is not enough to preserve the object to enable use/reuse in the future, other additional pieces of information also need to be preserved, or linked to in a permanent way. This is a challenge for the creation of these links without taking into consideration how these may be preserved over the longer term.

What do you think will be the most valuable outcome of SCAPE?

I think that the work done on the watch tool SCOUT which will enable certain conditions to be monitored in the wider environment is a welcome addition to the tools and infrastructure available for those concerned with digital preservation. To be able to use SCOUT effectively, then a particular organization will have considered and decided on the preservation objectives and underpinning policy, thus helping to ensure that digital objects are kept and hopefully functionally preserved for future use.

Contact information:

Catherine.jones@stfc.ac.uk
Skype handle: cm_j0nes
http://www.stfc.ac.uk/home.aspx

Preservation Topics: SCAPE

Who are you?

 I am Catherine Jones. I am an Information Systems Project Manager in the Scientific    Computing Department at the Science and Technology Facilities Council (STFC) in the UK. 

STFC is one of the UK seven Research Councils and provides both funding for research in Astronomy, Particle and Nuclear Physics and Space Science and large scale scientific Facilities (http://www.stfc.ac.uk/home.aspx).

Your role in SCAPE?

I am the STFC project manager responsible for planning and coordinating all STFC activities on SCAPE. STFC contributes in four main areas in SCAPE: we lead the work in the Research Data Testbed, we build tools for preserving research data, we are part of the policy representation discussions and we lead work in developing guidelines for best practices in preservation of scientific data in the context of SCAPE work.

At the moment I am working on Policy Representation. There are two aims of this activity. Firstly to identify and provide guidance on which topics need to be considered and addressed within preservation policy for a whole organization or a particular content/collection to assist those people who need to write policy in this area. The second is to produce a set of machine understandable statements and potentially actionable statements which can be used by the PLATO planning tool and the SCOUT watch tool. I am working on describing a process to enable this translation of natural language policy into the machine understandable statements, together with a sample set. This is a challenge as humans don’t need every implicit fact made explicit as computers do!

Why is your organisation involved in SCAPE?

STFC is publically funded and hence the data produced and managed here should be preserved for the long term. This data forms part of the Record of Science alongside scholarly articles. By participating in the SCAPE project with partners across other sectors we can share experience and practice.

What are the biggest challenges in SCAPE as you see it?

A challenge that the Research Data Testbed is starting to consider and work on is the preservation of the context for research data. For many types of research data it is not enough to preserve the object to enable use/reuse in the future, other additional pieces of information also need to be preserved, or linked to in a permanent way. This is a challenge for the creation of these links without taking into consideration how these may be preserved over the longer term.

What do you think will be the most valuable outcome of SCAPE?

I think that the work done on the watch tool SCOUT which will enable certain conditions to be monitored in the wider environment is a welcome addition to the tools and infrastructure available for those concerned with digital preservation. To be able to use SCOUT effectively, then a particular organization will have considered and decided on the preservation objectives and underpinning policy, thus helping to ensure that digital objects are kept and hopefully functionally preserved for future use.

Contact information:

Catherine.jones@stfc.ac.uk
Skype handle: cm_j0nes
http://www.stfc.ac.uk/home.aspx

Preservation Topics: SCAPE

It’s been a long wait, but version 2.1.0 of JHOVE2 is now out! Sheila Morrissey writes:

Version 2.1.0 of JHOVE2 includes 3 new format modules, 1 new identifier module, 1 new displayer module, and several bug fixes and enhancements from the Issues page on the JHOVE2 wiki.

The new format modules included in this release are for the ARC, WARC, and GZIP formats.

The new Identifier module uses the UNIX “file” utility, giving JHOVE2 users the choice of employing either DROID or file for identification of file formats.

The new XSLDisplayer module (which extends XMLDisplayer) can do XSLT transformations on the XML output before displaying it.

This release also reflects a new milestone in the JHOVE2 development community. The new format and identifier modules are the contribution of developers from institutions (Bibliothéque Nationale de France and NETARKIVET.DK) beyond the original project participants (California Digital Library, Portico, and Stanford University Libraries).

The release notes are available on the project site.

Congratulations to everyone who helped bring this release out!

Tagged: jhove2, software

We digital archivists warn about the risk of losing data with the assumption that the threat of loss is enough to stir people to action. But while most everyone has their own experience with data loss, people have a way of tucking past pain away rather than remaining hyper-vigilant about something similar happening again. Or maybe we console ourselves with the thought that was we might lose isn’t really all that important.

Periodically, it’s worth devoting full awareness to how much our digital files actually do mean to us. I trolled through Flickr to find some poignant expressions of people desperately seeking lost data.

It happens every year, by quin.anya, on Flickr

There but for the grace…, by pcorreia, on Flickr

Lost Laptop computer, by Steve Rhodes, on Flickr

reward for OQO, by Steve Rhodes, on Flickr

Day 10: Lost, by RemixDave, on Flickr

Laptop stolen, by Bahi P, on Flickr

Lost iPhone, by Steve Rhodes, on Flickr

Finally, a first semi-public demo instance is available to the OPF community. The current version features

- an overview on basic emulation services; different emulator + OS platforms are available for testing. The next bwFLA release will feature a sophisticated user management, i.e. users can start with a base image, clone this image as a dedicated user machine and further develop it to a dedicated rendering environment for certain digital artifacts;

- bwFLA / EaaS as a digital curation tool for dig. art by the example of Transmediale (http://www.transmediale.de/) CD-ROM art.

Access to the demo is password protected. Password and a quick overview of the demo features can be found in the members-area of the wiki: http://wiki.opf-labs.org/display/PT/bwFLA+test+demo+instance

If you are interested in the bwFLA workflows, our current use-cases and yet unreleased bwFLA features join the Webinar held by our dear college Annette on Tuesday, March 26 2012 at 12:00 BST / 13 CET.

Please register at: http://opf-bwfla-webinar.eventbrite.com/.

Preservation Topics: Emulation AttachmentSize demo.png282.28 KB cdroms.png348.16 KB

A Galaxy (S4) of new features
ITWorld Canada (blog)
When you consider that it'll require a video format in order to work (or worse, a completely new file format), I'm not sure why you'd actually want to do this...video is much more immediate. One of the features that's been getting a lot of talk is the ...

and more »

Many of the tools and practises developed for the digital forensics field can be integrated into digital preservation techniques. This is particularly true regarding:

• processes for securing, analysing, and appraising material prior to ingest into a repository or digital archive.
• donated personal digital archives, where physical hardware and media are acquired, rather than digital content.

Digital Forensic best practises deal with:

• Acquisition: activities to secure and preserve the state of physical and digital evidence. These include disk imaging, metadata creation, and producing authentic copies for examination. These techniques can also be used to "secure and preserve the state" of physical media and digital content.
• Examination: a rigorous, systematic examination of data to locate information of interest to the investigation. Methods here include duplicate detection, identifying system files from Operating Systems, programs, etc., detecting encryption, detecting personal data, and time line analysis. These techniques can complement existing characterisation methods in the digital preservation field.
• Analysis: an often manual analysis of extracted data, evaluating it for relevance to the investigation. A digital preservation practitioner's activities might include assessing the relevance digital material to the collection, finding or removing personal information, or creating a content profile.

These forensic tools and methods combined with established digital preservation tools and techniques can provide a pre-ingest workflow that:

• secures data at the point of acquisition.
• allows tools to be run on imaged copies protecting the source media and data.
• provides the metadata required to make informed decisions regarding the content.

What those decisions are in practise depends on variable factors including the type of access to be provided, permissions granted by the rights holder, and institutional policy.

One scenario might be that a subset of content is extracted from the image and ingested into a repository. Access to this material could be provided through an emulated environment, the choice of environment and rendering software informed through metadata gathered during the pre-ingest process.

When requirements and permissions allow however a potentially exciting emulation opportunity may present itself. It is possible to virtualise some of disk images created from the original media. The image must contain a supported working operating system and the process isn't certain to be successful. When this approach works access to the material can be provided through an emulated version of the original machine, which may have belonged to an author or researcher. This article in The Atlantic, and this one from The New York Times describe a nice example of the use of these or similar techniques to preserve the personal digital archive of Salman Rushdie.

The OPF is hosting two hackathons focussing on these themes:

• 15-17 May, Copenhagen, Denmark
• 3-5 June, Chapel Hill, North Carolina, USA (agenda and registration available soon)

Preservation Topics: EmulationOpen Planets Foundation

Page edited by Becky McGuinness

View Online Becky McGuinness 2013-03-18T10:23:51Z

Page edited by Becky McGuinness

View Online Becky McGuinness 2013-03-18T10:23:51Z

Page edited by Carl Wilson - "Tidied up, added some guidance on what material to bring (and what not to bring). Added further reading."

View Online Carl Wilson 2013-03-16T01:18:20Z

Page edited by Carl Wilson

View Online Carl Wilson 2013-03-16T00:45:12Z

File attached by Carl Wilson

JPEG File avatar.jpeg (2 kB)

View Attachments Carl Wilson 2013-03-15T20:57:10Z

The following is a guest post by Tess Webre, an intern with NDIIPP at the Library of Congress.

In a recent meeting, some colleagues and I discussed the age in which individuals should start understanding the basics of digital preservation. I suggested that, with children creating digital files earlier and earlier, it should be taught as early as possible. The question, of course, is how to get youngsters interested in preserving their data. Fortunately, while doing some research I was able to find a digital preservation fairy tale in the digital archives of the Brother’s Grimm. Here is the never-before published tale of Snow Byte and the Seven Formats (movie rights pending). I promise it will make a great bedtime story – they’ll fall right to sleep.  (I would like to thank my wonderful classmate Sara Allen for her invaluable contribution and illustrations)

Snow Byte and the Seven Formats

Written by Tess Webre, Illustrations by Sara Allen

nce upon a time in a land far far away there was a beautiful princess named Snow Byte. Snow Byte was a wonderful child who loved and was beloved by all who knew her, except for one. Her father, the king, had married a new wife and, as is often the case in fairy tales, her stepmother was evil. In fact, she was evil because she was a sorceress named Obsolescence, but the king didn’t know that. He wasn’t a very observant king.

Obsolescence was not only evil, she was shallow. She was obsessed with being the most beautiful woman in the entire kingdom. She created an enchanted smartphone with a magic mirror app that would tell her who was the fairest in the land. One day, Obsolescence looked at her magic mirror app.

“Mirror, mirror on the phone, who’s the fairest in the town?”

“You are fair, my Queen this is true. Yet, Snow Byte is fairer than you.”

Furious with this news, Obsolescence grew determined to kill Snow Byte. She used her enchanted smartphone to make all of Snow Byte’s digital files inaccessible by destroying all of her storage media. She then called Snow Byte into the chamber to demand that she run an errand for her. The errand would require Snow Byte to go through the enchanted forest, filled with trolls.

“Don’t worry, there is a spell that you have in your files that will save you from harm.”

Before leaving, Snow Byte checked and found that all of her files were inaccessible, thus she had no spell. Fortunately, she kept archived copies of this data in a trusted digital repository, and after a few phone calls and a few transfers, she was able to get her archived spells.

Wandering through the forest, Snow Byte came across the worst kind of troll in the forest, an Internet Troll. The troll immediately started berating her on her choices and life decisions. She tried to use the spell to repel him, but found that it didn’t work.

“Please,” scoffed the troll, “the queen gave me an antidote for that spell a while ago. She really doesn’t want you around. It might be because you are wearing that dress. It’s a really ugly dress.”

Angry, Snow Byte was about to defend her dress, when a group of woodland creatures came up to her. They told her that the queen had sent this troll to harass her to death and that the best thing she could do was ignore it. Snow Byte turned her back on the troll and walked away.

Snow Byte continued to wander through the forest, depressed that she could not come home, when she found a cottage. Finding no one at home Snow Byte walked right in to discover that there were seven tiny beds. She noted that on each bed was the name of a different file format.  There was WARC, TIFF, TXT, PDF, JPEG, MPEG and, of course, DOC.

Shocked by the unsanitary manner in the cottage, even for developers, Snow Byte immediately started organizing and cleaning. With the help of some random woodland creatures that followed her around, she created a strict metadata schema to organize all their files and objects, and updated their storage media. This was so exhausting that she fell asleep when she was finished.

When the seven formats came home from data-mining they were shocked to see the cottage so clean and a girl in the house. They woke her up and demanded she explain herself. Snow Byte told them about her wicked stepmother and the formats had a vote to let her stay.

Obsolescence, upon finding out that her rival was still alive, grew so angry that she decided to kill Snow Byte herself. She disguised herself as an old woman and programmed a poisoned app on her enchanted smartphone that would cause Snow Byte to fall into a coma. Using her sorceress powers, she found the cottage and waited for the formats to go off data-mining for the day.  She knocked on the door and Snow Byte answered.

“Deary, I just made this new smartphone and I wondered if you wanted to try it.”

“Is that an Apple phone?”

“No deary,” Obsolescence replied, “it’s completely different.”

Snow Byte took the phone and was amazed by how quickly it worked, how wonderful the touch screen responded, and all of the marvelous apps.

“Try this app, deary,” said Obsolescence, pointing to the poisoned app, “I think you will love it.”

Snow Byte opened the app and instantly fell into a coma. Obsolescence was so pleased with herself that she wrote a note saying that she had successfully killed Snow Byte and that the seven formats could do nothing about it. She skipped all the way home only to fall off a cliff and die.

When the seven formats came home they found Snow Byte in a coma and were flabbergasted. They found the note and were shocked.

“Something must have happened with this smartphone!” DOC declared.

At that moment a handsome prince named Dublin rode by and asked why there was a commotion. The seven formats told him what happened.  At that moment, one of the seven formats located a file called “poison app” but found that it contained no real information they could understand because it was in a proprietary file format that none of them could read.  Dublin took a look at the file and found it to have an XML wrapper of metadata. It showed that only the kiss of a prince would wake Snow Byte.

“This is a bit uncomfortable,” he said, “I apparently have to kiss her to wake her up.”

“Wait, what?” asked the formats, “you can’t expect us to believe that.”

“Yeah,” said Dublin, “you see how this file is gibberish. Well, if I decompress the file I find that there is a separate metadata wrapper on the spell she cast. It’s right here.”

Dublin showed the phone to the formats and it displayed this:

<rdfs:comment>
death only prevented by kiss from prince she will then wake up
</rdfs:comment>

“It clearly shows a prince kissing her and then her waking up. I mean, I don’t want to be a jerk or anything, but it’s clearly there.”

Dublin leaned down and kissed Snow Byte and she instantly woke up.  The formats rejoiced and Snow Byte thanked the prince. They fell in love, were married and lived happily ever after because they always preserved their data.

Page edited by Becky McGuinness

View Online Becky McGuinness 2013-03-15T14:46:49Z

Page edited by Becky McGuinness

View Online Becky McGuinness 2013-03-15T14:45:31Z

The Signal is a very good blog on digital preservation. It has a serious limitation, though: it’s published by the Library of Congress, which as a government agency has to stay neutral on businesses and products. I heard at the recent OPF Hackathon that people who write for it have been required to take out comments endorsing or criticizing specific products.

I don’t have that limitation. In Files that Last, I name names and make recommendations. Here are some things in FTL that you’ll never find on The Signal:

• “Sometimes content providers decide to stop supporting old DRM systems which require you to have online access, making the stuff you paid for suddenly useless. Major League Baseball did this in 2007 with its videos.”
• “According to several websites, in 2012 Sky News yanked a story which was embarrassing to Formula One Racing. It was ‘withdrawn for further review’ and later restored to the website in a redacted form. In some countries, removing or rewriting news stories because of governmental censorship is routine.”
• “Amazon’s use of the word ‘purchased’ for Kindle content is an outright lie.”
• “iPhoto is hostile to intelligent users and digital preservation. Flee from it.”

If you want to see more statements on digital preservation with no punches pulled, you’ll be able to in April when the book comes out.

Page edited by Becky McGuinness

View Online Becky McGuinness 2013-03-15T13:21:11Z