​Leading UK law firm Shoosmiths recently replaced their document management system (Hummingbird DM6) with a new solution based on MacroView DMF and Microsoft SharePoint (see the  Case Study).  A critically important part of this project was the migration of nearly 3 million existing documents.  Challenges included the need to move multiple versions of most documents, to preserve existing metadata and document-level permissions – as well as coping with the differences in how SharePoint operates compared to the previous traditional DM system.  In this article we look at how these challenges were handled so that the migration was a success.

The central requirement was to migrate just under 3 million documents from Hummingbird DM6 to SharePoint 2013 in such a way that existing versions, metadata and access permissions were maintained.  Shoosmiths had a total of 1,300 users across a number of departments.  They wanted to stage the migration on a department-by-department basis, which meant that all the documents relating to a department had to be migrated so that users of that department could arrive at work on Monday morning and start working on their documents with the new DM system.

Migration Software

The software chosen to handle the migration was the SharePoint Migration Tool from Clockwork Software. This tool streamlines the transfer of documents to SharePoint 2010 or 2013 from any SQL Server-based Enterprise Content Management System (or EDRMS, DMS, CMS) where the documents are stored as files on a network drive or file share.  This is the case with most popular traditional document management systems - including the Hummingbird DM6 as used by Shoosmiths and also DOCs Open, OpenText eDocs, iManage Worksite, FileNet, etc.

The Participants

The detailed work on the migration was undertaken by a team with members from Shoosmiths and from ClearPeople – a London-based partner of MacroView.  MacroView Professional Services supported the work of this team with specialist assistance in the use of the Clockwork SharePoint Migration Tool, consultancy support in the use of MacroView DMF and also by undertaking some small customisations of the MacroView DMF software (see more detail below).

ClearPeople worked with Shoosmiths to create a design for the new SharePoint document store.  This design used separate SharePoint web applications for Client / Matter, Precedent, Admin and Collaboration. The Client / Matter web app used a separate SharePoint site and document library for each Client, with a SharePoint folder tree corresponding to each Matter.  These sites, libraries and folders were provisioned automatically.

Coping with Volume

Satisfying the requirement for a department to be able to start using the new DM system on a Monday morning meant that up to 500,000 documents needed to be migrated over a weekend. Key to achieving this was the way multiple instances of the Clockwork SharePoint Migration Tool can be run simultaneously.  By running up to 10 instances of the Tool on each Web Front End and Applications server Shoosmiths were able to migrate up to 80,000 documents per hour. This made full test runs practical and allowed departments to be migrated, with validation testing, over a weekend.

“We migrated just under 3 million documents, almost 1 TB of data, over the course of 9 weekends.  Using 40 instances of the migration tools running in parallel across 4 SharePoint servers allowed us to condense almost 1,950 hours of migration into just under 67 hours, that’s a saving of over 78 days’ work! The tool was very stable and got the job done.”

Systems Developer, SHOOSMITHS LLP

 Figure 1: Staging the migration over multiple weekends

Metadata and versions

The Clockwork SharePoint Migration Tool handled the transfer of metadata attributes for each migrated document.  As part of this transfer existing Hummingbird value sets were mapped to new sets of values for SharePoint lookup columns. The Clockwork Tool allows a migration developer to specify these mappings and data extractions using SQL and provides a framework for assembling and applying the resulting SQL commands. 

The Clockwork Tool also handled the transfer of Creator, Creation Date, Last Editor and Last Edited Date for the migrated documents. In also populated the Last Edited, Last Editor andVersion Comments for each version. The Clockwork Tool automatically resolved values forCreator and Last Editor to the corresponding Person or Group values in SharePoint.

Shoosmiths used both Major Versions Only and Major and Minor Versions document libraries, with the Clockwork Tool handling migration to both types of library.

Unique Document Numbering

As each migrated document was loaded into SharePoint it was assigned a new unique document ID by the MacroView unique numbering provider. This provider replaces the out-of-the-box SharePoint provider, and numbers documents in much the same way as a traditional DM system – i.e. sequentially across the whole document store.  In addition the Clockwork SharePoint Migration Tool recorded the existing Hummingbird unique document ID as a custom metadata attribute for each migrated document.

The document search panels in the new MacroView DMF user interface were customised so that a document could be found using either its previous Hummingbird Document number or the new SharePoint document ID.

Although the capability was not utilized during the Shoosmiths migration, the Clockwork Tool can also be configured to record the existing Hummingbird document number as the value of the SharePoint Document ID, provided the MacroView unique numbering provider is in place.

Security

Within the previous Hummingbird system, each document effectively had its own access permissions. While SharePoint allows each document to have its own unique access permissions, this tends to lead to unacceptable performance. The preferable approach in SharePoint is to assign permissions at the document library or folder level, and have documents inherit that permission.

As part of the migration the permission structures for existing documents were analysed and document libraries were provisioned with corresponding security. The Clockwork SharePoint Migration Tool was then used to set custom permissions for only those documents that did not fit within these library-based security structures.

Coping with the Differences between Hummingbird and SharePoint

The virtual folders of Hummingbird DM allow a document to appear to exist in many different places. By contrast, SharePoint requires documents to be located in a specific library.  However the Link to Document capability of SharePoint allows an effect similar to virtual folders – virtual copies of a document that are actually stored in one library can be created in multiple other document libraries and folders anywhere in the SharePoint farm.

The Clockwork SharePoint Migration Tool was configured to migrate a document to its “best fit” location in SharePoint, and then to create Link to a Document items in all the other relevant locations. The Clockwork Tool logging tables were used to prevent successfully migrated documents from being migrated more than once, while still allowing Link to Document items to be created.

MacroView Professional Services customized the menu that appears when a MacroView DMF user right-clicks on a Link to Document item so that the menu contained a Navigate to Target Location item. Selecting this new menu item navigated the MacroView tree-view display of SharePoint so that the user was located on the area wherein the ‘master’ copy of the document was located.  The resulting effect was akin to using the Linked Item feature within Hummingbird.

Striking the Right Balance between Flexibility and Control

To say that document migrations are complex undertakings is an understatement! It is also true that no two migrations are the same.  In this regard an important aspect of the Clockwork SharePoint Migration Tool is its Proving Facility, which allows a migration developer to ‘try out’ proposed new extraction and mapping commands on a test document base, before attempting to run against a live document store.

Once proven the various extraction, mapping and control settings are grouped into Configuration Sets. The Clockwork Tool then assists in applying these Configuration Sets, to allow a large migration to proceed in a series of well-controlled stages.

For more information contact solutions@macroview.com.au