Big Data and Intuition: Status Update

BD_accenture0914Two new reports on big data and big decisions were released last week by Accenture and PwC. Both reports shed new light on the impact of big data on enterprises today, and how it is changing the process of decision making by senior executives.

An Accenture worldwide survey of senior technology and business executives has found that companies that have completed at least one big data project are happy with the results. 60% of executives said their companies have successfully completed a big data implementation, 36% haven’t pursued a big data project yet and 4% were currently pursuing but hadn’t finished their first big data project. 92% of executives from companies that have completed a big data implementation are satisfied with the results and 89% rated big data as “very important” or “extremely important” to their businesses’ digital transformation.

Other highlights include:

  • Executives see tangible business outcomes from big data in finding new sources of revenue (56%), enhancing the customer experience (51%), new product and service development (50%), and winning and keeping customers (47%).
  • Challenges in implementing big data are security (51%); budget (47%); lack of big data implementation talent (41%); lack of talent for big data and analytics on an ongoing basis (37%); and integration with existing systems (35%).

In welcome news to the graduates of the numerous data science and business analytics programs, nearly all (91%) companies expect to increase their data science expertise, the majority within the next year. 54% of executives said their companies have already developed internal technical training opportunities for their employees and most organizations also tap outside expertise. Only 5% of respondents said their company used only internal resources for their big data implementations.

The report also reveals that “many companies have different definitions of big data.” Indeed, in answer to the question “Which of the following do you consider part of big data?” responses varied from “Large data files” (65%), to “Advanced analytics or analysis” (60%) to “Data from visualization tools (50%).”

Regardless of how they define big data, 89% believe big data will revolutionize business operations in the same way the Internet did. 85% feel big data will dramatically change the way they do business and 79% agree that “companies that do not embrace big data will lose their competitive position and may even face extinction.” More than anything, in my opinion, these answers reflect the persuasive powers of a buzzword, inherent in its dual, attention-getting role: The threat of “disruption” and the promise of a “revolution” (two words used to great effect by the report).

The PwC report, titled “Gut & Gigabytes: Capitalising on the art & science in decision making,” is based on an Economist Intelligence Unit (EIU) worldwide survey of 1,135 senior executives. It clearly defines big data as “the recent wave of electronic information produced in greater volume by a growing number of sources (i.e. not just data collected by a particular organisation in the course of normal business).”

Intuition has got a bad rap in the age of big data and the most interesting finding of this report is that 30% of executives admit that intuition is what they most relied on when they made their last big decision. An additional 28% relied on other people’s intuition (“advice or experience of others internally”). Only 30% said that “data and analysis (internal or external)” is what they relied on for their last big decision and another 9% relied on “financial indicators.”

The reliance on intuition and experience is based on… experience. 46% of executives said that relying on data analysis has been detrimental to their business in the past. They are concerned about quality, accuracy and completeness of data and find it difficult to access useful data.

Still, 64% of the executives surveyed said that big data has changed decision-making in their organizations and 25% expect it will do so over the next two years. And 49% of executives agree that data analysis is undermining the credibility of intuition or experience, compared with 21% who disagree. Says the report: “In reality, however, experience and intuition, and data and analysis, are not mutually exclusive. The challenge for business is how best to marry the two. A ‘gut instinct’ nowadays is likely to be based on increasingly large amounts of data, while even the largest data set cannot be relied upon to make an effective big decision without human involvement.”

[Originally published on Forebs.com]