Google’s RankBrain Outranks the Best Brains in the Industry

Bloomberg recently broke the news that Google is “turning its lucrative Web search over to AI machines.” Google revealed to the reporter that for the past few months, a very large fraction of the millions of search queries Google responds to every second have been “interpreted by an artificial intelligence system, nicknamed RankBrain.”

The company that has tried hard to automate its mission to organize the world’s information was happy to report that its machines have again triumphed over humans. When Google search engineers “were asked to eyeball some pages and guess which they thought Google’s search engine technology would rank on top,” RankBrain had an 80% success rate compared to “the humans [who] guessed correctly 70 percent of the time.”

There you have it. Google’s AI machine RankBrain, after only a few months on the job, already outranks the best brains in the industry, the elite engineers that Google typically hires.

Or maybe not. Is RankBrain really “smarter than your average engineer” and already “living up to its AI hype,” as the Bloomberg article informs us, or is this all just, well, hype?

Desperate to find out how far our future machine overlords are already ahead of the best and the brightest (certainly not “average”), I asked Google to shed more light on the test, e.g., how do they determine the “success rate”?

Here’s the answer I got from a Google spokesperson:

“That test was fairly informal, but it was some of our top search engineers looking at search queries and potential search results and guessing which would be favored by users. (We don’t have more detail to share on how that’s determined; our evaluations are a pretty complex process).”

I guess both RankBrain and Google search engineers were given possible search results to a given query and RankBrain outperformed humans in guessing which are the “better” results, according to some undisclosed criteria.

I don’t know about you, but my TinyBrain is still confused. Wouldn’t Google search engine, with or without RankBrain, outperform any human being, including the smartest people on earth, in terms of “guessing” which search results “would be favored by users”? Haven’t they been mining the entire corpus of human knowledge for more than fifteen years and, by definition, have produced a search engine that “understands” relevance more than any individual human being?

The key to the competition, I guess, is that the “search queries” used in it were not just any search queries but complex queries containing words that have different meaning in different context. It’s the kind of queries that will stump most human beings and it’s quite surprising that Google engineers scored 70% on search queries that presumably require deep domain knowledge in all human endeavors, in addition to search expertise.

The only example of a complex query given in the Bloomberg article is “What’s the title of the consumer at the highest level of a food chain?” The word “consumer” in this context is a scientific term for something that consumes food and the label (the “title”) at highest level of the food chain is “predator.”

This explanation comes from search guru Danny Sullivan who has come to the rescue of perplexed humans like me, providing a detailed RankBrain FAQ, up to the limits imposed by Google’s legitimate reluctance to fully share its secrets. Sullivan: “From emailing with Google, I gather RankBrain is mainly used as a way to interpret the searches that people submit to find pages that might not have the exact words that were searched for.”

Sullivan points out that a lot of work done by humans is behind Google’s outstanding search results (e.g., creating a synonym list or a database with connections between “entities”—places, people, ideas, objects, etc.). But Google needs now to respond to some 450 million new queries per day, queries that have never been entered before into its search engine.

RankBrain “can see patterns between seemingly unconnected complex searches to understand how they’re actually similar to each other,” writes Sullivan. In addition, “RankBrain might be able to better summarize what a page is about than Google’s existing systems have done.”

Finding out the “unknown unknowns,” discovering previously unknown (to humans) links between words and concepts is the marriage of search technology with the hottest trend in big data analysis—deep learning. The real news about RankBrain is that it is the first time Google applied deep learning, the latest incarnation of “neural networks” and a specific type of machine learning, to its most prized asset—its search engine.

Google has been doing machine learning since its inception. The first published paper listed in the AI and  machine learning section of its research page is from 2001, and, to use just one example, Gmail is so good at detecting spam because of machine learning). But Goggle hasn’t applied machine learning to search. That there has been internal opposition to doing so we learn from a summary of a 2008 conversation between Anand Rajaraman and Peter Norvig, co-author of the most popular AI textbook and leader of Google search R&D since 2001. Here’s the most relevant excerpt:

The big surprise is that Google still uses the manually-crafted formula for its search results. They haven’t cut over to the machine learned model yet. Peter suggests two reasons for this. The first is hubris: the human experts who created the algorithm believe they can do better than a machine-learned model. The second reason is more interesting. Google’s search team worries that machine-learned models may be susceptible to catastrophic errors on searches that look very different from the training data. They believe the manually crafted model is less susceptible to such catastrophic errors on unforeseen query types.

This was written three years after Microsoft has applied machine learning to its search technology. But now, Google got over its hubris. 450 million unforeseen query types per day are probably too much for “manually crafted models” and google has decided that a “deep learning” system such as RankBrain provides good enough protection against “catastrophic errors.”

Deep learning has taken the computer science community by storm since it was used to win an image recognition competition in 2012, performing better than traditional approaches to teaching computers to identify images.

With deep learning, the computer “learns” by putting together the pieces of a puzzle (e.g., an image of a cat), moving up a hierarchy created by the computer scientist, from simple concepts to more complex ones. (see here and here for overviews of deep learning). Decades ago this idea got the unfortunate name “neural networks” under the misguided (and hype-generating) notion that the computer networks were “mimicking the brain” (what they were mimicking were speculations about how neurons work in the human brain). The hype did not produce the promised results but starting about ten years ago, with the availability of greater computer power and much larger sets of data and more sophisticated algorithms, neural networks have been reincarnated as deep learning.

In 2012, Google engineers made their first deep learning splash when they announced that Google computers have detected the image of a cat after processing zillions of unlabeled still frames from YouTube videos.

In their post on this deep learning experiment, Jeff Dean, a Google Fellow, and Andrew Ng, A Stanford professor on leave at Google at the time, wrote:

“And this isn’t just about images—we’re actively working with other groups within Google on applying this artificial neural network approach to other areas such as speech recognition and natural language modeling.”

And in 2013, Google engineers announced an open source toolkit called word2vec “that aims to learn the meaning behind words.” They wrote: “Now we apply neural networks to understanding words by having them ‘read’ vast quantities of text on the web. We’re scaling this approach to datasets thousands of times larger than what has been possible before, and we’ve seen a dramatic improvement of performance — but we think it could be even better.”

2013 was also the year Google hired Geoffrey Hinton of the University of Toronto, “widely known as the godfather of neural networks,” according to Wired.  But the two other widely known members of the (self-labeled) “deep learning conspiracy” went to Google’s competitors: Yann LeCun to Facebook (leading a new AI research lab) and Yoshua Bengio to IBM (teaching Watson a few deep learning tricks).

Then there’s Apple, Yelp, Twitter and others—all of Google’s competitors are rushing to adopt deep learning.

This creates a serious competition for talent, for all the graduate students who three or four years ago switched the topic of their dissertations to something related to deep learning and all the others who have joined recently this “computers can learn on their own” movement. Hence the need to tell the world via Bloomberg that Google is in the game and for Google’s CEO to insist on its latest earnings call that “machine learning is a core transformative way by which we are rethinking everything we are doing.”

But beyond PR and prestige, future profits could be the most important incentive for Google to add deep learning to its search technology. It’s not only reducing costs by reducing the need to rely on humans and their “manually crafted models.” It’s also search quality, the reason Google has become the dominant search engine and a verb.

A Search Engine Land columnist, Kristine Schachinger, sheds further light on RankBrain in the context of search quality and Google’s shift in 2013 (the “Hummingbird” overhaul of their search algorithms) from providing search results based on words (strings of letters) to search results based on its knowledge of “things” (entities, facts):

Google has become really excellent at telling you all about the weather, the movie, the restaurant and what the score of last night’s game happened to be. It can give you definitions and related terms and even act like a digital encyclopedia. It is great at pulling back data points based around entity understanding.

Therein lies the rub. Things Google returns well are known and have known, mapped or inferred relationships. However, if the item is not easily mapped or the items are not mapped to each other, Google has difficulty in understanding the query…

While Google has been experimenting with RankBrain, they have lost market share — not a lot, but still, their US numbers are down. In fact, Google has lost approximately three percent of share since Hummingbird launched, so it seems these results were not received as more relevant or improved (and in some cases, you could say they are worse)…

Google might have to decide whether it is an answer engine or a search engine, or maybe it will separate these and do both.

I will go even further and speculate that Google is seeing the end of search as we know it (and they perfected), the possibility that in the future we will not enter search queries into search boxes but will rely on “knowledge navigators” (to use the term Apple coined in 1986), going beyond the current answer engines to communicating with us, providing relevant information and news, and anticipating our needs by linking things in our past, present, and future.

Now, is it possible that with Facebook’s investment in AI and deep learning, it will be the first to provide us with a futuristic knowledge navigator? What will happen to Google’s advertising revenues if the social network will consist not only of people but also deep learning machines?

Given its past performance and the competitive people running it (and its parent company), it’s obvious that RankBrain is just one of the many investments Google is making in “disrupting itself before others do” (I’m pretty sure that’s how they talk about it). Google will continue to provide outstanding, free, advertising-supported service to its users, no matter what form this service will take in the future.

Or maybe not. Being a devoted and admiring Google search user, I was a bit skeptical when I read Schachinger’s words quoted above that Google’s search results “were not received as more relevant or improved (and in some cases, you could say they are worse).”  But one very surprising search result I recently got from Google, led me to think that, indeed, sometimes when you invest in the future, you sacrifice the present.

I Googled the address “75 Amherst Street, Cambridge, MA 02139.” What I got (a number of times, over three days) at the top of the search results was a map of 75 Amherst Alley, Cambridge, MA 02139.

There is such a place, but I have never heard about it or ever been there. What’s more, 75 Amherst Street is the home of MIT’s Media Lab, so this is not only a very simple query but also one that probably has been entered into Google numerous times (the Media Lab’s contact page appears as the second result, just under the erroneous map).

Time to invest in more humans working diligently on “manually crafted models”?