Big Data: Trends You Should Be Part Of

There has been a significant shift within the technology sector in the last year as more and more businesses of all sizes have woken up to the importance of data as the prime mover of development, discovery and – above all – increased profit. Understanding the data that is in a company’s silos, understanding whether it can be extracted and used, and if so, how it can most effectively be extracted and used, has led to a fundamental re-appraisal of the nature of corporate assets, and the development of strong data-use trends.

Kicking off the Big Data London conference on September 21^st, Conference Chair Mike Ferguson gave a concise summary of some of the trends in Big Data that are helping to move the corporate needle on data, and especially data sharing, to transform the way companies operate. These are the trends that will help define the “Work data-richer, not harder” approach to business, which will likely transform the business landscape over the next five years.

We’re not saying your business will definitely fail or suffer if you don’t ride these trends. There’s just a strong likelihood that companies who make the most effective use of the data they have will have the power to find more and better business opportunities in an increasingly tough, small-margin economy.

And Now, The Headlines…

Some of the headline trends Ferguson identified as important right now may feel familiar – but some will raise eyebrows.

• He raised the importance of the ongoing adoption of multicloud solutions to data issues, but also mentioned that there was a growing backlash on spiralling costs. “CFOs are beginning to squeal that they want full visibility on costs when it comes to multicloud projects,” he told the conference. That’s a trend that will make for an increasingly rocky negotiation landscape in the next year to eighteen months, with technologists having to justify expenditure on such multicloud projects not only in terms of business benefits, but in terms of hard, bottom-line outlay.
• Both data management platforms and data governance were trending, said Ferguson – understandably so as the amount of data collected by companies increases exponentially, particularly when it comes to unstructured data. While it’s important to understand that Ferguson didn’t mention this explicitly, it’s easy to infer from this growing trend that companies without a data management platform by around this time in 2023 may find themselves in increasing levels of stagnation and a forest of data governance issues. Acting sooner, rather than later, on data management and governance can only make sense and make the transition more smoothly manageable – assuming, probably correctly, that the data management issue will only intensify as technology and the data landscape improves and expands respectively.

Less Latency, More Data Democracy

• Ferguson noted that there was an increasing demand for data with significantly lower latency. That might well push the market towards automated data classification engines in the next few years, to speed up processes in which the latency of a human factor has always previously been a drag on delivery.
• He also told the conference that the demand for democratization of the data and analytics space was increasing. “The desire to share and reuse analytical products – like machine learning – is increasing,” he said.

• One of the most fundamental paradigm shifts that Ferguson identified for Big Data London delegates though was the spreading of data-hunger. “The hunger for data and insights probably started in the finance departments of businesses,” he explained, “but now, every department wants data and insights, every department wants that data edge on the competition. The question is how you get that data and those insights when the data landscape is increasing complicated, with data on various clouds, data in various software-as-a-system platforms, and so on.”
• That was made especially complicated, he said, because there was no industry standard to which companies had to adhere – and to which they could look for guidance – for data-sharing and integration across data catalogs. So, he forecast, we will see the evolution of more automation in data catalogs, extending not only to the search and output functions that are already commonplace, but also to auto-classification of data, which will be a big step forward (assuming it works with at least human reliability and consistency).

• IT departments, Ferguson predicted, will have to fundamentally change their remit as data becomes a more central focus in all parts of any business. They will be practically required by the shift of paradigm into going out and about in the business to more effectively manage data flow and management from system to system, office to office, cloud to cloud, and catalog to catalog.

The Always-On Business

• And finally, to set the mood of the two-day event and set delegates on the paths to making effective connections, he asked them how the new, more software developer-style approach to would impact the approach of data and analysis.
• While delegates would be steered one way or another depending on the weight they gave to upcoming presentations at the event, he suggested that the new approach would have to be integrated into applications if business wanted to see a data-powered digital transformation into “the always-on AI business” – the practically perfect, data-rich, profit-making, value-delivering, redundancy-lean business that will be the core of the next generation of enterprises.