Hadoop Maturity Survey: Key Findings

Results of the largest worldwide survey to date of Hadoop users were released recently by AtScale, which was assisted in this project by Cloudera, MapR and Tableau. The answers from more than 2,100 participants provide insights into the current and future use of Hadoop and where and how it brings business value to the companies investing in this foundational big data technology.

“Enterprises today have reached a new stage where they are worried about providing access to Hadoop to business users,” says Bruno Aziza, AtScale’s Chief Marketing Officer. “Self-service is the next big thing for Hadoop and the companies that understand how to deliver on the promise of democratizing access to the Data Lake will win.”

So what’s up with Hadoop?

Focusing on revenues, providing self-service, and an executive mandate are key to getting value from Hadoop

Hadoop DriversCompanies that deploy Hadoop to drive revenues are about 30% more likely to achieve value from Hadoop than those who are looking to cut costs. Those that provide business users with self-service access to Hadoop are nearly 50% more likely to realize a tangible value. The presence of an “executive mandate” (i.e., telling people they are going to do Hadoop rather than just “sponsoring” it) boosts an enterprise’s chances of deriving value from Hadoop by 20%.

Hadoop_Self-Service

Hadoop_Exec Mandate

The more you invest in Hadoop, the likelier you are to get value out of it

72% of the enterprises surveyed that have deployed over 500 nodes are getting value out of Hadoop, but only 33% of those with up to 10 nodes say they got value out of their investment. 49% of all survey participants say they are already deriving tangible value from Hadoop and 94% think they will continue or start to get value from Hadoop.

Hadoop is moving to the mainstream of enterprise IT

In its early days, Hadoop was talked about (or dismissed) as “replacement for tape,” a better-performing data archiving solution. As its capabilities and ecosystem have evolved, it is now maturing into a mainstream enterprise data management tool.

76% of current Hadoop users plan on doing more with it over the next 3 months and almost half of those who haven’t yet deployed Hadoop say they will do so in the next 12 months. While 26% describe Hadoop as “experimental,” 47% regard it as “strategic” and 15% say it is “game changing.”

Hadoop_Strategic

Business intelligence is rising again

When big data and data science took over the airwaves five years ago, Hadoop rose as a technology of choice for two specialized workloads: The traditional Extract-Transform-Load (ETL, the process by which data is taken from different sources, formatted, and uploaded to a data warehouse or database) for the folks in IT dealing with unstructured data, and the new data science process used by the new breed of data scientists to find new insights in the data, somewhat overshadowing the business analysts performing traditional business intelligence tasks.

This is reflected in where Hadoop has been deployed so far. 74% of the companies participating in the survey that have already deployed Hadoop are using it for ETL, 62% for data science and 65% for business intelligence.

In another indication of the mainstreaming of Hadoop, however, business intelligence may rise again to take advantage of the new new thing.  The picture is somewhat different for those that have not yet deployed Hadoop, with 69% intending to use it for business intelligence, 56% for data science and only 51% for ETL.

Hadoop_Tipping Point

Online companies are Hadoop-savvy, financial firms not so much

As could be expected, the companies that make their living on that vast repository of unstructured data, the World Wide Web, are ahead of other sectors in Hadoop maturity.  Surprisingly, the financial services sector which is typically the first to invest in and learn new enterprise IT tools and applications did not do well in terms of Hadoop maturity in this survey. It may well be that financial services companies hold themselves to very high standards or that the survey captured many small financial services firms that do not have the same resources as their larger brethren.

Hadoop_Industries

As ones and zeros are eating the world and enterprises worldwide are analyzing private and public data to find new sources of revenues and new ways to streamline operations, time-to-access (for data) and time-to-value (for analysis) are shrinking rapidly. Hadoop and similar big data tools are graduating into an enterprise IT environment where the underlying technologies merge to collectively provide more data and better analytical tools to more employees.

Originally published on Forbes.com