Big Data and its business benefits have been bandied around since the term entered the Gartner Hype Cycle in 2011 so it’s no surprise that research carried out by Vanson Bourne indicates that 86 percent of organizations now have big data systems in place. But the technology landscape is becoming more complex with nascent technology emerging every month and solution providers attempting to influence the technical decision by addressing architecture concerns such as volume, velocity, variety and veracity.
[easy-tweet tweet=”Is operating big data in the cloud really a viable option?” hashtags=”cloud, tech, bigdata, business”]
But how do businesses choose the solution that works best for them and is operating big data in the cloud really a viable option?
Cloud – for better or worse
Choosing the right big data architecture is no longer a one size fits all method as one single solution is unlikely to cater for all the requirements of business needs, across the entire organization.
Typically, enterprise data analytics is deployed through a combination of proprietary on-premise data warehousing solutions and BI platforms. At the same time modern Hadoop, NoSQL, MPP and Streaming technologies are not longer uncommonness, but rather a standard step towards accommodating big data requirements. Fuelled by this novelty and an all but infinitely scalable cloud resource, the benefits of processing and analyzing big data in the cloud has become an important part of the wider industry discussion.
When I talk to companies about implementing big data projects, their most common concerns include:
- Their data users experience serious performance, data quality, and usability issues
- They have an existing data infrastructure and licenses that are rigid, so it is, therefore, difficult to update the legacy system
- There is a skills gap in the organization where many departments don’t actually know how to gain access to the data or pull insights from it
- The investment in time and money needed to build a new system from scratch is unappealing
On this basis, you can see why cloud-based solutions are increasingly found to be an attractive path to modern big data analytics.
[easy-tweet tweet=”Cloud-based solutions are increasingly found to be an attractive path to modern big data analytics.” hashtags=”tech, cloud, bigdata, “]
Instant infrastructure
One of the single most valuable benefits of choosing a cloud big data solution is flexibility. A cloud-based solution can allow a single network administrator to support hundreds or even thousands of users and open the doors for new business roles to gain access to the data. In addition to this, it enables the creation of a highly available and elastic infrastructure. This enables businesses to focus more on the core business and solves technical problems in a completely different way.
With 63 percent of the organizations viewing big data analysis as a necessity to remain competitive in increasingly data-based economies, the ability to establish big data infrastructure as quickly as possible is a major pull factor for cloud-based solutions. By combing cloud services for data storage, processing and archiving companies such as AWS provide the opportunity for rapid prototyping and deployment of infrastructure that companies would otherwise have to build up themselves from scratch.
Cost reduction
Organizations are increasingly migrating business functions to the cloud whether it’s security, unified communications or data archiving and a key driving factor behind this is the potential reduction in capital expenditure (CapEx). Similarly, migrating big data analytics to the cloud, even if it is in small pockets, can offer major financial advantages.
Performing big data analytics on-premise requires companies to invest in and maintain large data centers, which can incur a sizable initial investment and create a long-term drain on IT budgets. This limits the accessibility of big data to companies of a certain size with enough revenue to maintain this investment. When businesses move to the cloud, this responsibility shifts to the cloud services provider which makes this solution appealing for smaller start-up companies as well as larger organizations that require a scalable solution that can be altered to meet demand.
We don’t suggest that businesses abandon their own, in-house big data centers completely but instead reduce their reliance on these resources, maintaining small, efficient data centers while transferring the majority of their big data analytics workloads to hosted environments. This cuts the cost of purchasing equipment, cooling machines, and security, while also allowing them to keep the most sensitive data on-premise.
The ability to take a hybrid approach to big data analytics is vital for some industries whereas others are happy to go “all in”. Although the financial services industry understands the value of big data and its ability to provide business critical insight, there still remains hesitation to migrate to a multi-tenant public cloud provider due to the security of sensitive data. SoftServe’s recent Big Data Snapshot indicates that 75 percent of financial institutions rank security as their top big data concern meaning banks and financial institutions are likely to continue to take this hybrid approach.
What’s next?
Machine learning is adopting at a rapid pace, and businesses are rushing to take big data analytics one step further to not just benefit from business intelligence dashboards but also a new level of efficiency by automating manual analysis or enabling new user experiences. Businesses already understand the value of machine learning – despite the term only entering the Gartner Hype Cycle last year, 62 percent or organization already plan to implement it within the next two years.
[easy-tweet tweet=”The Cloud has super data gravity meaning that the more workload it ingests the cheaper it becomes” hashtags=”tech, bigdata, cloud, business”]
The Cloud has super data gravity meaning that the more workload it ingests the cheaper it becomes for everyone. Similar to traditional big data analytics solutions, processing machine learning in the cloud has appealing cost saving benefits. In addition to this, machine learning workloads could be variable making it difficult for businesses to invest in and forecast on-premise equipment, therefore an elastic and scalable solution within a hosted environment is a much better fit for those businesses considering the technology.
Ultimately the success of cloud-based big data analytics is dependent on a number of key factors and perhaps the most significant of these is the quality and reliability of the solution provider. The right vendor should combine robust, extensive expertise in both the big data and cloud computing sectors, and for businesses that still can’t decide between the multitudes of solutions on offer it’s advisable to consult an expert who can provide invaluable guidance through the decision stage and implementation of your big data project.
Serge Haziyev, VP Technology Services Group at SoftServe
Serhiy (Serge) Haziyev is a VP Technology Services Group at SoftServe. He has more than 18 years' experience in designing, evaluating and modernizing large scale software architectures in various technology domains including BI, Big Data, Clouds, SOA and Carrier-grade telecommunication services for both Fortune 100 and startups.
He is a co-author of the architectural poker game currently used by leading institutions to teach students to architect Big Data solutions. Also, Serge is a co-author of Big Data chapter in the SEI Series book Designing Software Architectures: A Practical Approach.
He frequently speaks at professional and scientific conferences across the globe (such as SEI SATURN, IEEE ICSE, WICSA and HICSS) where he conducts tutorials and provides practical inputs on emerging technologies.