Digital data is accumulating at an unprecedented rate. The need for reliable, long-term storage solutions is becoming increasingly urgent. Traditional methods are struggling to keep up with the demand.

However, new technologies are emerging to address this challenge. From holographic storage to glass panel data preservation, these advancements promise to revolutionize how we store information. Meanwhile, sectors like museums, businesses, and healthcare are all grappling with this data deluge.

The Growing Data Challenge

The year 2039 may seem distant, but Ian Crawford at the Imperial War Museum is already planning for it. He oversees a large project to digitise a vast collection of pictures, audio, and film. With around 24,000 hours of video and 11 million photographs, it’s a huge task.

Preserving these historical sources digitally is crucial as the original copies degrade over time. Crawford said, “When you’ve got the only copy, you want confidence that your storage system is reliable.” Advanced scanners can now capture documents and films in great detail, generating large data files.

Data Explosion Beyond Museums

This surge of data impacts not just museums but many sectors. Businesses need more space for backup data, hospitals must store records, and governments require storage for increasing information volumes. Simon Robinson from Enterprise Strategy Group says that data volume for most organisations doubles every four to five years.

Data for long-term storage isn’t kept in traditional centres with servers and blinking lights. Instead, it’s often stored on tape. The latest format, LTO-9, can hold 18 terabytes of data – equivalent to nearly 300 standard smartphones. The Imperial War Museum uses a Spectra Logic tape system, which can hold up to 1,500 LTO tapes.

Innovative Holographic Storage

In Chiswick, London, a startup called HoloMem is developing a long-term storage system using lasers to burn tiny holograms into light-sensitive polymer. Chief Executive Charlie Gale highlights that holograms can store data in multiple layers, unlike magnetic tape. “That’s really kind of the superpower of what we’re doing. We can put more information in less space than ever before,” Gale says.

The polymer used by HoloMem can handle extreme temperatures, from -14C to 160C, whereas magnetic tape needs stricter climate control. Tapes need replacing every 15 years, while the polymer lasts at least 50 years. Once written, the data in the polymer can’t be tampered with, making it highly secure.

HoloMem’s prototype, capable of storing and retrieving data, is set to be ready later this year. Gale is confident that HoloMem’s costs will be competitive, using standard components to keep prices down.

Microsoft’s Glass Storage Project

Microsoft is also moving beyond magnetic tape with its Project Silica, which stores data on glass panels. Using powerful lasers, tiny structural changes called voxels are created in the glass to store data. A 2mm thick piece of glass, about the size of a DVD, could store over seven terabytes of data.

The glass panels are stored on racks, accessible by small robots. Richard Black from Microsoft says glass is highly durable and almost immune to temperature and environmental changes. This makes it a promising medium for long-term data storage, potentially lasting hundreds to thousands of years.

Project Silica could eventually integrate into Microsoft’s cloud computing service, Azure. However, it still has significant development ahead before it’s market-ready.

AI’s Role in Data Management

The Imperial War Museum is exploring the use of artificial intelligence (AI) to manage its digital library. Recently, they tested AI to identify different models of Spitfire planes from photographs. Ian Crawford believes AI could be very useful in cataloguing, which would otherwise take humans hundreds of years.

AI’s ability to sift through massive amounts of data has underscored the importance of data storage. Businesses can now perform analytics to find valuable insights in archived data, a shift from merely storing data ‘just in case’. Simon Robinson mentions that there’s now a solid business reason to retain data for potential future analysis.

Innovation in data storage is not just a necessity but a revolution in handling the ever-growing deluge of data. The advent of holographic and glass storage technologies promises more efficient, durable, and scalable solutions.

As institutions like the Imperial War Museum and tech giants like Microsoft push the boundaries, the future of data preservation looks bright. With AI’s growing role in data management, not only will storage solutions become more robust, but the ability to extract valuable insights from archived data will also enhance.

The journey of data storage is evolving rapidly, and these new technologies are paving the way for a more secure and manageable data-filled future.