Overcoming the big data barriers

view from behind a fence

Big data can transform businesses of all kinds, as long as they’re set up to take advantage.

The big data revolution is here: we’ve all heard about it and it’s transforming the way businesses manage their data. It’s opening up new possibilities and opportunities, but at the same time posing many challenges.

When CA Technologies surveyed 1,000 business professionals, more than half said they had implemented some form of big data project. However, when asked what was holding them back, the top answers were:

  • Complexity: Managing such a large implementation is a major undertaking
  • Managing multiple solutions: Multiple solutions create complexity and reduce transparency.
  • Cost: For businesses with limited funds, the cost of implementation often represents too big a hurdle.
  • Difficulty reintegrating analysis: Having the data is all very well. The challenge is integrating it back into the systems. Many organisations are balancing multiple platforms which do not necessarily gel together.
  • Lack of visibility of information and processes: Data overload is an enormous problem. Rather than a lack of data, businesses are just as likely to complain about too much.
  • Inability to build the infrastructure: Big data analytics requires an overhaul of equipment and infrastructure. Implementation is a major project in itself and many business owners say they don’t even know where to start.

The problems are particularly felt by small and medium sized businesses. These often struggle to process the data they are collecting and lack the resources and the knowledge to manage it effectively.

A common complaint among business leaders is that they simply don’t know the questions to ask in order to bring about successful transformation. The quick answer is to hire a data analyst and while that might be a simple, and quick fix for most larger corporations, a smaller enterprise will have to think carefully about whether or not it can afford the investment.

Of course, that analyst may well, over time, inject revenue back into the business and more than justify the initial investment, but that takes time. A CEO struggling to balance tight budgets and while he might see the benefits of big data, he might not feel in a position to implement change. In order words, it’s a bit like trying to rescue a big plasma TV screen when your house is on fire. Yes, you’d like to have it but you have somewhat more pressing concerns.

A change of attitude

IT has long been seen as a cost centre.

The age of big data requires a rethink – to see the IT department as a revenue-generating enabler of business. It is now doing much more than just recording and storing data, it is enabling engagement and opening up valuable new sources of revenue. To get the most out of their IT departments, businesses must see them as a strategic partner and invest the time, money and resources it needs.

The people problem may be one of the most difficult issues to overcome. Even when systems have the ability to mine data and produce incredibly deep insights, much of that information may be left behind. For example, a company may develop enhanced analytics which demonstrates multiple layers of customer engagement online. However, many managers remain stuck in an old attitude of ‘how many clicks’ they received, and so miss crucial details.

Technology can only show us the door. Attitudes must develop if we are to walk through it.

Kickstarting change

The first step to change, then, will be to address office culture. Individuals must understand what the data is telling them and how it can help; they must learn to see the IT department as a key strategic element of the operation rather than just an administrative function.

The approach to data management will have to change out of all recognition. Traditional data systems managed data which had been carefully curated. Access was restricted to just a few individuals and reports were relatively limited.

In the big data world, information comes in all shapes and sizes.

New technology makes this possible. A new generation of tools offer ways to store and process data which would otherwise have fallen through the cracks. Systems such as Hadoop, for example, can store and process data which would have previously have been lost. Automation tools can take care of many of the repetitive, mundane and lengthy tasks which would previously have consumed man hours and resources.

As these multiply, though, they create complexity with many businesses operating across multiple platforms. Because traditional data centres are not set up to store the data or adequately mine information from it.

To solve the problem, operators are shifting to cloud and multi-cloud strategies. Bringing in data from multiple sources and presenting it in one easy to manage system. This reduces the number of different platforms businesses need to work on. Enhancing data visibility by putting all the information they need in one place. Alongside the technical shift, organisations much adapt their own culture and outlook to facilitate smooth inter-departmental collaboration.

Businesses can become much more adept at harvesting information. After all, technology such as the Internet of Things enable a business to bring in data from almost anywhere. For example, black box devices can help with vehicle fleet management. Sensors can track the progress of packages or the condition of machinery. The more forward-thinking organisations are rapidly increasing their communication points to improve the yield of their data gathering.

It represents an enormous undertaking. Businesses are going through a transformative phase both in the software at their disposal, and their approach to data management. IT departments are changing in structure and organisation and taking a position much closer to the hub of operations.

However, for all the gains which have already been made there is much more to come. Market penetration is still in its early stages as organisations struggle to work out how to make use of big data. It leads to a divergence between the early adopters and those who are stuck in the past. Those which have made the move will gain a crucial competitive advantage which will show across their organisation.