2014 Big Data Predictions from IDC and IIA

Crystal_BallBoth IDC and The International Institute of Analytics (IIA) discussed their big data and analytics predictions for 2014 in separate webcasts last week. Here is my summary of their predictions plus a few nuggets from other sources.

IDC predicts that the market for big data will reach $16.1 billion in 2014, growing 6 times faster than the overall IT market. IDC includes in this figure Infrastructure (servers, storage, etc., the largest and fastest growing segment at 45% of the market), services (29%) and software (24%). IDC commented that the benefits of big data are not always clear today (indeed, BNY Mellon recently asked its 50,000 employees “for ideas about how to harness the power of Big Data”). IIA predicted that companies will want to see demonstrable value in 2014 and will focus on embedding big data analytics in business processes to drive process improvement.

The much-discussed shortage of analytics and data science talent led IIA to make three separate but related predictions. One prediction is that the adoption of analytics-as-a-service will accelerate with “ready-made analytics in the cloud” offering an attractive option for quickly testing big data analytics or scaling up existing programs. Similarly, Capgemini predicts (in an email to me) “smaller, nimble analytics,” as a result of the rise of machine-to-machine data, “making cloud the de facto solution.” And IDC predicts that cloud infrastructure will be the fastest-growing sub-segment of the big data market, with a 2013-2017 CAGR of close to 50%.

Another IIA prediction related to the dearth of talent is the increasing attention paid by companies to organizing in teams the analysts and data scientists they currently have on board, either embedded in the business units or in a center of excellence. The focus will be on making these teams more effective by establishing and sharing best practices and by “operationalizing and managing models,” with the rest of the world getting closer to the proficiency level of the financial industry in this regard (in other words, keeping up with the quants? hopefully, also learning from the financial industry’s failures in this regard—see financial crisis, 2008 edition).

As for the prospects for alleviating the talent shortage, IIA commented that there are now well over 100 programs at universities in the US where analytics and data science “are in focus” (see my list of graduate programs here). IDC, for its part, cautioned that these programs “will bear fruit only in four to five years,” referring obviously to the newly-established data science programs. IDC agrees with IIA that companies providing big data analytics services will fill the gap in the meantime and predicts that the big data professional services market will exceed $4.5 billion in 2014. The number of vendors providing such services will triple over the next three years, according to IDC, and these firms will “aggressively acquire scarce big data talent,” making it scarcer.

A very interesting dimension to the dearth of talent raised by IDC is the shortage of IT professionals capable of dealing with the new big data requirements. 33% of respondents to an IDC and Computerworld survey earlier this year noted as one of their big data challenges the “lack of sufficiently skilled big data and analytics IT staff” (“lack of sufficient number of staff with appropriate analytics skills” was selected by 45% of respondents).

Also interesting was IDC’s expansion of the services part of the market to include “value added content providers.” These include “traditional vendors” such as Thompson, LexisNexis, and Experian; “new wave vendors” such as DataSift, Gnip, and LinkedIn; “company and personal information vendors” such as Acxiom, Equifax, and Tarsus; and “search engine/aggregators” such as Yahoo, Google, and Salesforce/Data.com. IDC believes that this market segment will be “challenged by lack of business model clarity and standards.”

A related prediction from IDC is that VC investment will shift to the top layers of the big data software stack, from information management to the “analytics & discovery” and “applications” layers. New types of applications (“use cases”), such as personalized medicine, will emerge out of what IDC predicts will be the blurring of the boundaries between high-performance computing (previously limited to scientific/engineering applications) and “enterprise big data” (i.e., mainstream applications managed by an IT department). IIA sees other new horizons for the application of big data, predicting that companies in a variety of industries will increasingly use analytics on the data they have accumulated to develop new products and services. GE has been the poster boy for this emerging trend, called “Analytics 3.0” by IIA, or “the digitization of everything” by me (you decide).

Another application, security, will become the next big front for big data, IDC predicts, as security infrastructure will increasingly take on big data-like attributes. Big data will be used to correlate log data and identify malicious activity in real time, allowing companies to react quickly, rather than after the event. Gartner begs to differ, however, predicting that “big data technology in security contexts will stay immature, expensive and difficult to manage for most organizations as targeted attacks become more stealthy and complex to identify in progress. … The noise about big data for security has grown deafening in the industry, but the reality lags far, far behind.”

In a somewhat far-out prediction, IIA talked about facial recognition and wearable device data that will be incorporated into predictive analytics. One of the examples given was “pet stores could use facial recognition to greet dogs as well as customers.” IDC was a bit closer to 2014 (or was it?) when it predicted that the “proliferation of sensor, mobile, wearable, and embedded devices (Internet of Things) will become a significant driver of the big data market,” stressing the need for investment in “Data-in-Motion” and “real-time analysis of geo-dispersed incoming data streams,” primarily in the cloud (that you don’t need wearables or geo-whatever to satisfy your obsession with quantifying your life, was recently demonstrated by the resident data scientist at MarkITx who crunched his lunches to come up with a happiness-per-gram metric).

Both IDC and IIA got a bit more into the technologies behind big data analytics, with IDC predicting the co-habitation for the foreseeable future (my words) of traditional database technology (RDBMS) with the newer Hadoop ecosystem and NoSQL databases, concluding that “in the short term,” information management will become more complex for most organizations (see shortage of qualified IT staff above); and IIA predicting that “the adoption of data visualization will accelerate in both the high and low ends of the complexity spectrum [for analytics].” Humans, however, don’t comprehend things in more than two dimensions or, at most, three dimensions, so IIA advised tempering our enthusiasm “a bit” (this came from self-described Tom “Curmudgeon” Davenport so you may want to consider how much tempering you want to do; as for me, I always opt for being “uber-curmudgeon”).

Last but certainly not least, both IDC and IIA talked about automation in the context of big data. IDC predicts that “decision and automation solutions, utilizing a mix of cognitive computing, rules management, analytics, biometrics, rich media recognition software and commercialized high-performance computing infrastructure [phew!], will proliferate.” Some of these solutions, IDC says (warns?), “will begin to replace or significantly impact knowledge worker roles.”  IIA predicts that “we will see a continued move to machine learning and automation to keep pace with speed and volume of data” and that “as they strive to operationalize analytics but encounter challenges with the over-automation of decisions, companies will focus more on the optimal mix between human and machine capability and judgment.” If you take humans too much out of the equation, their decision making will atrophy, warned IIA, asking “If you don’t have experts, who will train the next generation of [machine learning] software?” From the IIA’s lips, to the NSA’s ears, I say. (Well, we can assume these words were collected and stored by the omnipresent sleuths the second they were uttered; the question is: do they understand what the words mean?)

One prediction that didn’t make the official list of IIA’s predictions, but Davenport nevertheless managed to include in the webcast, was that “companies will need to hire lawyers to verify that they actually own the data.” Indeed, the nagging issues—that I think will be even more prominent in 2014—of privacy and governance were largely missing from the IDC and IIA discussions (Capgemini, in contrast, contributed this: “A heightened focus on governance will improve analytic results… Governance will need to be a driver in shaping the ROI story for Big Data in 2014”).  Also missing were discussions of “open data” and the increased use of big data by the public sector (outside of the NSA) to name just a few pertinent big data trends not on their list of predictions. But of course, the challenge is to select the nine or ten most important ones and we have lots to chew on with IDC’s and IIA’s lists.

Listeners to the IIA webcast were given the opportunity to vote on which predictions they thought would come true:


Participants in the IIA webcast included Sarah Gates, Tom Davenport, Bob Morison, Bill Franks, Greta Roberts, Omer Sohail and Sanjeev Kumar; IDC’s webcast was delivered by Dan Vesset and Ashish Nadkarni; Capgemini’s predictions were attributed to SVP for Business Information Management Scott Schlesinger.

[Originally published on Forbes.com]