Big Data Landscape 2016

big_data_landscape_2016.png

Matt Turck:

VC investment in the space remains vibrant and the first few week of weeks of 2016 saw a flurry of announcements of big founding rounds for late stage Big Data startups: DataDog ($94M), BloomReach ($56M), Qubole ($30M), PlaceIQ ($25M), etc.  Big Data startups received $6.64B in venture capital investment in 2015, 11% of total tech VC. M&A activity has remained moderate (we noted 35 acquisitions since our last landscape)…

With continued influx of entrepreneurial activity and money in the space, reasonably few exits, and increasingly active tech giants (Amazon, Google and IBM in particular), the number of companies in the space keeps increasing…

In terms of fundamental trend, the action (meaning innovation, launch of new products and companies) has been gradually moving left to right, from the infrastructure layer (essentially the world of developers/engineers) to the analytics layer (the world of data scientists and analysts)  to the application layer (the world of business users and consumers) where “Big Data native applications” have been emerging rapidly…

The big trend over the last few months in Big Data analytics has been the increasing focus on artificial intelligence (in its various forms and flavors) to help analyze massive amounts of data and derive predictive insights.

The recent resurrection of AI is very much a child of Big Data.  The algorithms behind deep learning (the area of AI that gets the most attention these days) were for the most part created decades ago, but it wasn’t until they could be applied to massive amounts of data cheaply and quickly enough that they lived up to their full potential…

In turn, AI is now helping Big Data deliver on its promise.  The increasing focus on AI/machine learning in analytics corresponds to the logical next step of the evolution of Big Data: now that I have all this data, what insights am I going to extract from it? Of course, that’s where data scientists come in – from the beginning their role has been to implement machine learning and otherwise come up with models to make sense of the data.  But increasingly, machine intelligence is assisting data scientists – just by crunching the data, emerging products can extract mathematical formulas… or automatically build and recommend the data science model that’s most likely to yield the best results… A crop of new AI companies provide products that automate the identification of complex entities such as images or provide powerful predictive analytics…

As unsupervised learning based products spread and improve, it will be interesting to see how their relationship with data scientists evolve – friend or foe?  AI is certainly not going to replace data scientists any time soon, but expect to see increasing automation of the simpler tasks that data scientists perform routinely, and big productivity gains as a result…

…AI has made a powerful appearance at the application level as well.   For example, in the cat and mouse game that is security, AI is being leveraged extensively to get a leg up on hackers and identify and combat cyberattacks in real time.  “Artificially intelligent” hedge funds are starting to appear.  A whole AI-driven digital assistant industry has appeared over the last year, automating tasks from scheduling meetings (watch Dennis Mortensen, CEO of x.ai here) to shopping to bringing you just about everything.  The degree to which those solutions rely on AI varies greatly, ranging from near 100% automation to “human in the loop” situations where human capabilities are augmented by AI – nonetheless, the trend is clear…

In many ways, we’re still in the early innings of the Big Data phenomenon.  While it’s taken a few years, building the infrastructure to store and process massive amounts of data was just the first phase.  AI/machine learning is now precipitating a trend towards the emergence of the application layer of Big Data.   The combination of Big Data and AI will drive incredible innovation across pretty much every industry.  From that perspective, the Big Data opportunity is probably even bigger than people thought.

As Big Data continues to mature, however, the term itself will probably disappear, or become so dated that nobody will use it anymore.  It is the ironic fate of successful enabling technologies that they become widespread, then ubiquitous, and eventually invisible.