One of the recurrent themes at record keeping events is the frustration the record keeping community expresses at not having “a seat at the table”. I used to shift a little uncomfortably in my own seat because much of the frustration was directed at “IT people” like me.

The solutions of today have a nasty habit of becoming the problem of tomorrow and IT’s success in automating the production of all kinds of data has led to the problem loosely known as big data.

Big Data is the dilemma created when the means to create data vastly outweighs the capacity to manage that data. The volume, velocity and variety of information so outstrips our capacity to manage that information many parts of the organisation have given up trying. 

Some records departments have solved the problem by drawing their own line in the sand and, using a piece of perplexing logic, maintaining that if it isn’t in the record keeping system it isn’t a record. There may be thousands of emails flying around the network unmanaged, but so long as the letters are still registered, there is a small part of the madness that has familiar order. It could be argued this is not drawing a line in the sand but burying your head in the sand.

Meanwhile, IT people keep providing more storage and place their faith in the power of their concordance index. More computing power, faster indexes. If a search engine can find a needle in a haystack, surely that is all that is needed? This is the sorcerer’s apprentice solution. We are so busying enabling we are too busy to manage.

Six years ago I believed in the power of the search engine but now am convinced it is only one side of the solution. Google has proven the power of search but the search is only effective when Google places you in the right haystack.

I have also had a humbling realisation it is the often neglected record keepers who are responsible for “harvesting” the fields and placing the hay into haystacks. In other words, they give context to the content that the search engines so easily find.

The challenge for record keeping is to shift from an artisan model of hand crafting each document to a system of automated classification. It is enacting machine mediated contextualisation to a human standard. 

For record keepers this is not a great change in role but it is a significant change in mindset. There is no longer a one-to-one relationship with the record keeper and the document. The record keeper becomes an organisational ontology manager training a rules engine that takes the place of the previous manual process. Where the rules engine is not confident, the record keeper trains the engine how to recognise the context and apply the rule.  From that point on, another exception moves into the rule set.

There are software products emerging (and rapidly) that offer the promise of semantic indexing. Following the now-predictable Gartner hype cycle, at first they will be overhyped and then underestimated and we will find ourselves surrounded by them.

For record keepers, recognising the seismic shift that will bring them back to the table is a critical perspective.  There are many record keepers on the point of retirement and many who follow “the rules” rather than realise the time is rapidly approaching when they will be called upon to make the rules.  It will be a time when record keepers will be invited back to the table. I just hope there are still some remaining who will answer the call.

Andy Carnahan is Customer and Information Services Manager, Wingecarribee Shire Council, NSW. Andy will speaking on the topic “How to get a seat at the Big Data table - an overview of the emerging methods that will put record keeping back on the map and put business back in control of its data assets” at the upcoming Ark Event EDRMS for the Australian Government 2014. The event is over two days 30-31 October in Canberra. Details at http://www.arkgroupaustralia.com.au