Double Take: Mapping Great Britain: Long-term archiving for the future
November 2007 by DoubleTake
Ordnance Survey creates a blueprint for long term archiving and disaster recovery with the help of BridgeHead Software’s HT FileStore, improving storage management and reducing the cost of primary storage.
Back in 1791 the British Government realised that in planning adequate defences to repel any invasion, the South Coast of England needed to be comprehensively and accurately mapped. So it instructed its Board of Ordnance – the defence ministry of its day – to speed the necessary survey work. That decision led to the mapping of the whole of Great Britain in detail, and is also the source of the intriguing name ’Ordnance Survey’ – an organisation which has grown to become the world’s leading map-maker and a major provider of digital geographic information.
Today it is a self-financing £100-million-a-year civilian organisation producing computer data products and paper maps for business, leisure, administrative and educational use. It is still part of Government, but it covers its operating costs by selling its products and services or licensing others to use its copyright material.
Although best known for its paper maps, computer data already accounts for more than 90% of Ordnance Survey’s business. Digital product customers include the Land Registry, Forestry Commission, Police and Fire services throughout Britain and other Government departments. It is also involved in a collaborative project to produce a geographical blueprint of Great Britain, the Digital National Framework.
Ordnance Survey is creating the definitive national geographic database, digitally mapping the changing face of Britain for perpetuity. Its considerable investment in global positioning systems, digital aerial photography and new surveying techniques enables it to constantly update, enhance and maintain the database with at least 5,000 changes every day.
High definition digital aerial photography provides a significant part of this project, with camera data taking up more than 20% of Ordnance Survey’s primary disk storage space in 2005 and this is expected to grow by 20%, a further 40 terabytes (TB), per year. Ordnance Survey keeps all the raw images and corresponding edited JPEG files for an infinite length of time and as a consequence, it faces a huge problem of never-ending growth of high volume data files and its associated storage.
Dave Lipsey, Information Systems Infrastructure Manager at Ordnance Survey, said: “The project to digitally map Great Britain presented us with a storage challenge that requires us to quickly access terabytes of large files far into the future without crippling our infrastructure. We needed a solution for eternal storage, otherwise the project itself could not be completed and our IT system would be brought to its knees.”
Historically, Ordnance Survey used online disk storage, adding hardware to absorb the data growth, a strategy based on the falling price of disk. However with the increased use of digital graphical data, the SAN based arrays running on Windows 2003 servers, soon felt the strain. Not only is Ordnance Survey creating millions of new images each flying season, but also the underlying files are massive; each raw file is 700 megabytes (MB) and edited JPEGs are a minimum of 90MB.
The continual creation of image files that needed to be retained forever was accumulating on the SAN faster than it could be backed up. The system was starting to get more errors with unavailable tapes and backup time windows were being exceeded, putting other systems at risk. Ordnance Survey also estimated that within four to six months, the data being received would surpass the backup capabilities, putting the organisation at risk in the event of a disaster.
Ordnance Survey needed to act shrewdly and quickly to tackle escalating storage and management costs and reduce backup windows. It had almost come to a physical hard stop and it had reached capacity limits of its online disk hardware. In addition, although the cost of online storage was coming down, Ordnance Survey did not benefit from the cost reductions as the data management costs soared.
“We knew that throwing capacity at this problem was not the answer. The best solution for reducing the burden of the growing digital geographical data storage needs was automated archiving to long life removable media. A new system would need to operate openly, integrate into our existing infrastructure, and provide multiple copies for backup and robust disaster recovery,” Lipsey explained.
Working with system integrator, OptoMedia, Ordnance Survey selected BridgeHead’s storage agnostic, automated, policy-based archive software, HT FileStore and Plasmon’s UDO (Ultra Density Optical) storage technology for secure, long term archiving. The combined solution stores and manages the massive lifelong digital archives for years without the need to migrate data regularly to new storage media.
Plasmon’s UDO WORM (Write Once Read Many) technology ensures that Ordnance Survey’s geographic and historical information is kept for many decades with unquestionable data authenticity and unmatched media longevity. The stability of UDO media dramatically reduces archive management overheads. By deploying UDO, Ordnance Survey can comfortably operate on an 8-12 year data migration cycle, which is much greater than the typical 2-3 year lifecycles of disk or tape.
Ordnance Survey collects more than 40TB of digital photographic data every flying season, with files of over 1 gigabyte (GB) not uncommon. BridgeHead’s HT FileStore provides long-term data management and uses intelligent automated policies to locate these new files and archive them to on-site and off-site Plasmon G-Series UDO libraries. After a given time, it checks that the archive is good and then leaves a stub impression of just 1KB on the server in place of the original larger raw file, thus reducing the huge cost and management burden of primary storage and at the same time allowing fully transparent access to the file in the archive. The stubbed files are also indexed so users can perform in-depth searching of the archive. Once found a file or files will be restored to the required location, allowing them to be worked on.
The ability to archive to multiple copies and multiple locations was a major factor in the selection of HT FileStore. Its self-replicating archive feature eliminates nearly all the expense of performing separate backup and disaster recovery procedures and, combined with UDO media; it provides a robust disaster recovery solution. HT FileStore’s policies migrate digital files to the on-site G638 UDO library (19TB) and at the same time, copy them to the smaller Gx24 UDO library at the disaster recovery site using a Bridgeworks iSCSI bridge.
Of using HT FileStore, Lipsey said, “Simply put, it’s like an automatic valve, controlling the flow of data in, out and within the system policing the data flow through the defined policies. Rather than manually guessing which files are available to be archived or deleted, the process in now completely automated.”
In total, Ordnance Survey will be using more than 1,300 pieces of 30GB UDO media to archive historic information. HT FileStore’s policy based media management and automated file migration will ensure that the lifecycle of all the media pieces are managed and that the data will be moved to newer technology at the appointed time in the future.
Ordnance Survey estimated that the previous hard disk storage system was costing around £2,500 per TB. The proposed BridgeHead/Plasmon package would only cost around £1,200 per TB in the short term, and in the longer term, this would fall to a mere £600 per TB.
“The combined solution from Bridgehead Software and Plasmon is the latest in archiving software and long term storage media technology, and will save Ordnance Survey almost £500,000 in storage costs,” said Lipsey. “The project will breakeven in four years, with an investment return rate of 13.5%, a figure that is well above the Government minimum of 5% and our own benchmark of 7%.”
By using BridgeHead’s automated archiving software, HT FileStore and Plasmon’s UDO long-term storage media, Ordnance Survey has already saved 20TB of capacity in the first few months of operation. HT FileStore has enabled Ordnance Survey to manage its media transparently across both production and DR sites, providing a resilient archiving approach by using multiple copies to support both the disaster recovery and access needs of the business. The result is an archive infrastructure that is easy and cost-effective to manage. As the HT FileStore approach to archiving is adopted more widely within Ordnance Survey, the benefits of having multiple copies and the ability to perform multi-site media management will continue to save the organisation time and effort which can be deployed elsewhere to support its strategic IT objectives.
Ordnance Survey’s maps and paper archives from its 200 year history are available today, and it can now work towards ensuring that its newest digital data is around for at least another 200 years, providing a continual record of geographical transformations.
Lipsey explained: “Plasmon’s UDO storage media has a life expectancy of 50 years, unparalleled in technology available today. And BridgeHead’s HT FileStore enables us to securely migrate data to any future technology as and when it becomes available, no matter which vendor develops it.”
The system will influence the approach to archiving throughout Ordnance Survey’s infrastructure. The digital geographic archiving project is a precursor to being the standard for rolling out the combined HT FileStore/UDO automated archive solution for other departmental archiving and disaster recovery requirements such as its SAP and Oracle applications.
“Overcoming this challenge using the BridgeHead and Plasmon archiving solution has provided us with a blueprint for our archiving needs moving forward,” concluded Lipsey. “We implemented this for the Digital Air Camera storage requirements first, as this was identified as requiring the most immediate archiving solution.”
“Working with Ordnance Survey has shown us the true meaning of storage forever. You can never underestimate the importance to any business of having access to your data, whenever that might be needed in the future,” added Tony Cotterill, CEO of BridgeHead Software.
- Double Take Software : A practical guide to business continuity planning
- Multimedia International Services selects Double-Take Software to ensure Business Continuity