NewVantage Partners Survey: Big Data Now Mainstream, Term Still Widely Disliked

For the third year in a row, NewVantage Partners has conducted a survey of Fortune 1000 senior business and technology executives regarding their companies’ investments in big data. This annual survey (see my summary of the 2013 report here) provides a unique glimpse into the big data initiatives of large enterprises and how fast they have adopted the set of technologies, processes, and skills associated with it.

Big data has gone mainstream with 67% of executives reporting that they have big data initiatives running in production, up from 32% last year. A remarkably rapid adoption rate, since only two years ago 85% of the respondents had a big data initiative “planned or in progress.” While a third of respondents in 2012 had big data investments of $1 million or more, 45% reported big data spending of greater than $10 million this year, and 74% estimated that their companies will spend that amount in 2017.  82% of the executives surveyed said big data is already integrated into the mainstream of their organizations while only 12% said it is managed as a separate environment.

Who leads this new mainstream corporate activity which 82% of the executives surveyed view as being highly important or mission-critical within their organizations? 37% of executives reported that the primary executive sponsor is the CEO, COO, or business President, stressing the importance of sponsorship from the top of the organization. And who does the executive sponsor turn to for big data accountability? Leading the charge in many situations is the Chief Data officer (CDO) with 43% of executives reporting that their organization has defined and established such a role. This is a sharp jump from previous years–in 2012 only 19% of executives reported that their organization had established a CDO function, growing only slightly to 26% in 2013.

The business function that is the primary driver of investment in big data for 36% of the respondents is sales and marketing. But other functions also lead big data initiatives, including those focused on internal activities: Risk, security, regulatory, and compliance (29%); R&D and new products (21%); IT and operations (10%); and finance, HR and administrative (4%).

What are these diverse functions expecting from their investments in big data?  Three quarters of executives cited greater insight and learning, the ability to answer business questions faster, make informed decisions faster, and accelerate speed-to-market as the primary business driver for their big data initiatives.

“It should be noted,” the NewVantage Partners report state, “that the level of skepticism [about big data] has dropped significantly since we first conducted this survey in 2012.” Indeed, most of the executives surveyed (74%) now believe that big data warrants “serious attention” and only 3% call it “same old stuff.” But the level of comfort with the term itself is still very low. Most respondents said that the term is not helpful in focusing a serious discussion about the value and benefits of big data to their organizations and 83% of executives claim to dislike the term big data (53%), or find it to be overstated (30%).

As Tom Davenport writes in his introduction to the report: “Everything is sunny in the world of big data… except, its name! Perhaps never has there been such enthusiasm about a business topic, and less satisfaction with the name of it… [Big data is] clearly not just hype, and it’s not just experimental. Companies are getting real value from it in production applications. Now if we could only figure out what to call it!”

It is clear, however, that the executives answering the survey, while not happy with the term itself, knew quite well what they were talking about when they referred to their big data investments. For them, it is a new type of activity associated with new programs and new roles and responsibilities, and it is centered on new business drivers (see above), but also a diverse set of needs related to the rise in the volume and variety of data. When asked about the most important technical driver for their big data investments, they answered as follows: Integrate more varieties of data (22%); apply new big data approaches (17%); integrate larger volumes of data (16%); more effectively integrate legacy data (13%); and enable greater agility (12%).

Regardless of the mainstream coverage big data gets which is mostly about the new technologies (e.g., Hadoop) it is associated with, its ascendance to the mainstream of IT investments by large enterprises has much more to do with the success of these investments in answering specific business needs and demonstrating the value of improved data analysis.

[Originally published on Forbes.com]