Data warehousing has reached an impasse. Born in an era of monolithic systems walled off from the outside world, the data warehousing technologies that have dominated the last thirty years stand today as an unsightly reminder of the wear and tear taken by increasingly vain attempts to help them stumble forward into the modern world.
The legacy data warehouse has struggled because it suffers from the limitations of an approach and architecture defined by the constraints of the physical world, a world in which resources were limited, applications ran in isolated silos, and changes were infrequent and carefully controlled.
The limitations born from that world are today the chains that shackle the data warehouse, tying it to limitations of the past and preventing it from addressing the realities of the present and future. Simply put, the legacy warehouse was not designed for the volume, velocity, and variety of data and analytics demanded by the modern enterprise.
Data analytics, once accessible only to the largest and most sophisticated global enterprises, today is a top priority for marketing, finance, development, and sales at organisations of all types and sizes. Stories of the outsized impact of data analytics only increase the demand for better insights, for example, the story of how utilities can realise a 99 percent improvement in accuracy by harnessing big data.
However, the growing demand for data analytics is shining a spotlight on the painful inadequacy of legacy data warehouses. For one, they were never designed for the sheer volume and pace of data that exists today. In large enterprises the volume of data generated within the organisation is overwhelming. For example, in just one year Rolls-Royce generates over three petabytes of data in the manufacturing of fan blades for turbines. Mining that data is essential to design, manufacturing, and after-sales support. Exploding volumes of data are not just a large enterprise problem even mid-sized organisations that may not generate such volumes of data internally have access to large external data sources such as data.gov.uk.
Traditional data warehousing, with its chains of the physical world, is prohibitively expensive and inflexible, limitations that become painfully obvious with the scale of todays data. The limitations force difficult capacity planning exercises and large upfront expenditures to acquire sufficient capacity in advance to avoid a painful, disruptive exercise to scale your data warehouse.
Not only is the scale of data a challenge for organisations, so is the demand for faster results. Not so long ago users would have been happy to get updated analytics on a weekly or daily basis. Today that would be seen as a failure users expect results in seconds and expect those insights to be based on data that is continuously updated, even as the number of people with direct access to data and analytics grows.
Again, the limitations of traditional warehousing become painfully clear. Prohibitively slow batch processing has crippled business intelligence (BI), and the inability of legacy architectures to scale to handle growing numbers of users ruins response times for everyone.
How do organisations cope with the challenge of adapting legacy data warehousing to address these challenges? Unfortunately, most have struggled mightily to do so. The complexity and labour required to keep these legacy solutions from collapse add to the strains not only on IT but also on the business. It’s no wonder that less than half of business executives in a recent UK survey expressed confidence in the insights they have from their data and analytics solutions.
If enterprises are to continue to grow, they will need a radically different approach to data analytics – one that leaves behind legacy shackles from the physical world and ascends to the cloud.
[easy-tweet tweet=”A cloud-based solution should be able to operate painlessly at any scale” hashtags=”Cloud,DataAnalytics”]
The promise of cloud for data analytics is a solution that frees organisations from the limitations and constraints around which legacy data warehouses were designed. A cloud-based data warehouse should not only eliminate limits to resources and the complexity of systems and infrastructure; but it should also fundamentally eliminate the friction and complexity of providing the entire organisation with fast, flexible access to all of the data that they need to make the best possible decisions.It should empower enterprises to shift their focus from systems management to analysis.
To deliver this, a cloud-based solution should be able to operate painlessly at any scale, effortlessly able to adapt immediately to dynamic changes in data volume and analytics workloads. It must also make it possible to combine diverse data, both structured and semi-structured, to bring together data silos that prevented users from accessing data locked away in a sprawl of enterprise databases, CRM systems, general ledgers, and web applications. It must do all of this at a cost and efficiency that is connected to the value delivered, not tied to a legacy model of upfront costs and ongoing management burdens.
Such a data warehouse changes the way that analytics are made available to users. Instead of an assembly line in which multiple steps are required before data is available for users and their analytics, a new approach can emerge that can be thought of in terms of distillation: a system in which data is available immediately and at every step of refinement, supporting data exploration and experimentation in the same system that also supports business intelligence and reporting.
This not only changes the experience for users, but it also changes the experience for the business. Data insights can be always up to date and directly accessible to everyone who needs them. Further, they can be richer and more comprehensive because they can combine data from all possible sources to create insights that would have been inaccessible before.
This new world holds the promise of reinvigorating data analytics, ending the struggle faced by enterprises today. Although it is difficult to forecast the changes taking place across the data landscape, the promise of clouds on the horizon could mean new life for your business.
Thibaut Ceyrolle is Vice President of Sales for EMEA at Snowflake Computing. Thibaut has long been at the forefront of disruptive technology, with prior experience in driving the cloud revolution at Hewlett-Packard in 1998. He quickly ascended through the ranks of digital transformation proponents DevoTeam and BMC Software, before serving as VP of EMEA Sales at Bazaarvoice.