The (Almost) Mainstreaming of Big Data


Source: NewVantage Partners

Two recent surveys shed new light on the state of big data, showing growing adoption but persistent data management challenges.

Management consulting firm NewVantage Partners (NPV) will release next week the results of its 4th annual Big Data Executive Survey (see my coverage of the previous rounds here and here). Survey respondents were Fortune 1000 senior business and technology executives. IT industry association CompTIA has released last month its Big Data Insights and Opportunities report, based on an online survey of 402 business and IT professionals.

Here are 6 observations, based on these two reports, about what is happening with big data in practice:

Big data adoption is rising but it’s still not in production everywhere

NVP found that 62.5% of firms have at least one instance of big data in production, up from 31.4% in 2013 and 48.2% in 2014. Only 5.4% of firms now report that they have no big data initiatives planned or underway.  51% of the companies participating in the CompTIA survey said they have some form of big data initiative in place, up from 42% in 2013.

The reputation of big data is getting stronger

69.6% of firms view big data as very important or critical to their business success, up from 54.5% in 2014 (NVP).  72% of companies that have launched some form of big data initiative say that their results have exceeded expectations (CompTIA).

Big data drivers: Insights wanted, speed required

Developing greater insights about the business and customers (37%) and speed (faster time-to-answer, faster time-to-decision, and faster time-to-market, at 29.7%), were cited as the biggest drivers of big data investments (NPV). 63% rely on data for day-to-day operations, 60% use data to better understand customers, and 59% rely on data to measure business objectives (CompTIA).

The challenge of the variety of data sources trumps volume and velocity

Firms continue to report that variety is the primary technical driver behind big data investments (40.0%), with volume (14.5%) and velocity (3.6%) lagging well behind (NVP). 45% of companies say that a high degree of their data is fragmented  and another 42% say their data fragmentation is moderate (CompTIA).

Organizational challenges: finding the right talent and leadership

42% of companies don’t have real-time analytics skills and 41% are lacking relational databases skills. As a result of poor data management and usage, 29% suffer from inefficient decision-making and inability to reach new customer segments, while 38% suffer from wasted time. Only 31% of companies are exactly where they want to be in managing and using data (CompTIA).  54% of firms have appointed a Chief Data Officer (CDO), up from 12% in 2012. And 20% report that the Chief Data Officer is the executive with primary ownership responsibility for big data initiatives (NVP).

For big data initiatives to succeed, the business and technology sides of the organization must cooperate

33.9% of firms identified business and technology cooperation as the most critical factor in business adoption, up from 23.4% in 2013, and leading all other factors by a wide margin. Strong business sponsorship was cited as the most critical factor by 23.2% of firms. By contrast, technology leadership (5.4%) and technology selection (0.0%) drew negligible responses (NPV).


Originally published on