Why The Future Is Not What It Used To Be

by alex-xs

by alex-xs

The IEEE Computer Society published in March a report titled “What Will Our World Look Like in 2022?” It identified 23 technology areas that we can expect to disrupt the state-of-the-art. These range from medical robotics to big data and analytics to photonics to 3D integrated circuits to quantum computing.

The unifying theme for all these technologies is “seamless intelligence,” where everything is connected through ubiquitous networks and interfaces. “We project that by 2022,” the authors of the report say, “society will advance so that intelligence becomes seamless and ubiquitous for those who can afford and use state-of-the-art information technology.”

The IEEE report is a bit different from similar attempts at predicting the future because it comes from technologists, some in academia but others who work at corporate research labs, and is based in part on a survey of members of the IEEE Computer Society. Typically, predictions are the stock-in-trade of think tanks and research firms. Last year, for example, the McKinsey Global Institute published “Disruptive technologies: Advances that will transform life, business, and the global economy,” identifying 12 technologies that “could drive truly massive economic transformations and disruptions in the coming years.” Earlier this year, Juniper Research published “The World in 2020: A Technology Vision,” identifying 11 key technologies that it believes “will become the most disruptive by 2020.”

Beyond the use of the word “disruptive,” there are other commonalities between the three reports. Robotics and drones, 3D printing, the Internet of Things and wearables, self-driving cars, and cloud computing appear in all or at least two of the reports. But, for the most part, there is not a whole lot of agreement on the disruptive technologies of the future. Photonics, real-time translation, and renewable energy, for example, appear in only one of the reports.

The IEEE report opens with the famous Yogi Berra quote: “It’s tough to make predictions, especially about the future.” In the rest of this post, I will discuss three reasons why.

  1. Innovations that have made a strong impact on us in recent times obscure more important recent innovations.

The first item on The New York Times’ list of greatest inventions of the 19th century, published in 1899, was friction matches, introduced in their modern form in 1827. “For somebody to whom the electric light was as recent an innovation as the VCR is to us, the instant availability of fire on demand had indeed been one of the greatest advances of the century,” wrote Frederick Schwartz in 2000 in Invention & Technology. Which invention of the last 100 years or even 10 years is overshadowing an even more important invention of recent years?

  1. The road taken is no less important than the end result.

Another difficulty in predicting what the world will look like in just 5 or 7 years from now, is that some predictions eventually become a reality but they still miss altogether exactly how we are going to get there. Often, this is the most important (and practical) part of the prediction. To paraphrase Lewis Carroll, if you know where you are going, it matters a lot which road you are taking.

Many commentators writing this month about the 50th anniversary of Gordon Moore’s article charting a future course for the semiconductor industry (what became to be known as “Moore’s Law”), mentioned his predictions regarding home computers and “personal portable communications equipment.” But they ignored Moore’s prediction that “the biggest potential lies in the production of large systems. In telephone communications, integrated circuits in digital filters will separate channels on multiplex equipment. Integrated circuits will also switch telephone circuits and perform data processing.”

Moore was right that integrated circuits will have an impact on large systems but failed to see that “the biggest potential” of the constant and predictable miniaturization he forecasted will be in smaller and smaller devices, in ubiquitous computing. In 1965, it was difficult to see that centralized systems will be replaced by distributed, anywhere computing. Which is why Moore added to his use of the term “home computers”—“or at least terminals connected to a central computer.”

  1. We extrapolate from the present and ignore or misunderstand non-technological factors.

Many predictions are what the forecasters want the future to be or simply an extension of what they are familiar and comfortable with. I have in my files a great example of the genre, a report published in 1976 by the Long Range Planning Service of the Stanford Research Institute (SRI), titled “Office of the Future.”

The author of the report was a Senior Industrial Economist at SRI’s Electronics Industries Research Group, and a “recognized authority on the subject of business automation.” His bio blurb indicates that he “also worked closely with two of the Institute’s engineering laboratories in developing his thinking for this study. The Augmentation Research Center has been putting the office of the future to practical test for almost ten years… Several Information Science Laboratory personnel have been working with state-of-the-art equipment and systems that are the forerunners of tomorrow’s products. The author was able to tap this expertise to gain a balanced picture of the problems and opportunities facing office automation.”

And what was the result of all this research and analysis? The manager of 1985, the report predicted, will not have a personal secretary. Instead he (decidedly not she) will be assisted, along with other managers, by a centralized pool of assistants (decidedly and exclusively, according to the report, of the female persuasion). He will contact the “administrative support center” whenever he needs to dictate a memo to a “word processing specialist,” find a document (helped by an “information storage/retrieval specialist”), or rely on an “administrative support specialist” to help him make decisions.

Of particular interest is the report’s discussion of the sociological factors driving the transition to the “office of the future.” Forecasters often leave out of their analysis the annoying and uncooperative (with their forecast) motivations and aspirations of the humans involved. But this report does consider sociological factors, in addition to organizational, economic, and technological trends. And it’s worth quoting at length what it says on the subject:

“The major sociological factor contributing to change in the business office is ‘women’s liberation.’ Working women are demanding and receiving increased responsibility, fulfillment, and opportunities for advancement. The secretarial position as it exists today is under fire because it usually lacks responsibility and advancement potential. The normal (and intellectually unchallenging) requirements of taking dictation, typing, filing, photocopying, and telephone handling leave little time for the secretary to take on new and more demanding tasks. The responsibility level of many secretaries remains fixed throughout their working careers. These factors can negatively affect the secretary’s motivation and hence productivity. In the automated office of the future, repetitious and dull work is expected to be handled by personnel with minimal education and training. Secretaries will, in effect, become administrative specialists, relieving the manager they support of a considerable volume of work.”

Regardless of the women’s liberation movement of his day, the author could not see beyond the creation of a 2-tier system in which some women would continue to perform dull and unchallenging tasks, while other women would be “liberated” into a fulfilling new job category of “administrative support specialist.”  In this 1976 forecast, there are no women managers.

But this is not the only sociological factor the report missed. The most interesting sociological revolution of the office in the 1980s – and one missing from most (all?) accounts of the PC revolution – is what managers (male and female) did with their new word processing, communicating, calculating machine. They took over some of the “dull” secretarial tasks that no self-respecting manager would deign to perform before the 1980s.

This was the real revolution: The typing of memos (later emails), the filing of documents, the recording, tabulating, and calculating. In short, a large part of the management of office information, previously exclusively in the hands of secretaries, became in the 1980s (and progressively more so in the 1990s and beyond) an integral part of managerial work.

This was very difficult, maybe impossible, to predict. It was a question of status. No manager would type before the 1980s because it was perceived as work that was not commensurate with his status. Many managers started to type in the 1980s because now they could do it with a new “cool” tool, the PC, which conferred on them the leading-edge, high-status image of this new technology. What mattered was that you were important enough to have one of these cool things, not that you performed with it tasks that were considered beneath you just a few years before.

What was easier to predict was the advent of the PC itself. And the SRI report missed this one, too, even though it was aware of the technological trajectory: “Computer technology that in 1955 cost $1 million, was only marginally reliable, and filled a room, is now available for under $25,000 and the size of a desk. By 1985, the same computer capability will cost less than $1000 and fit into a briefcase.”

But the author of the report (just like Gordon Moore in 1965) could only see a continuation of the centralized computing of his day. The report’s 1985 fictional manager views documents on his “video display terminal” and the centralized (and specialized) word processing system of 1976 continues to rule the office ten years later.

This was a failure to predict how the computer that will “fit into a briefcase” will become personal, i.e., will take the place of the “video display terminal” and then augment it as a personal information management tool. And the report also failed to predict the ensuing organizational development in which distributed computing replaced or was added to centralized computing.

Yes, predicting is hard to do. But compare forecasters and “analysts” with another human subspecies: Entrepreneurs.  Entrepreneurs don’t predict the future; they make it happen.

A year before the SRI report was published, in January 1975, Popular Electronics published a cover story on the first do-it-yourself PC or what they called  “first minicomputer kit,” the Altair 8800. Paul Allen and Bill Gates, Steve Jobs and Steve Wozniak, founded their companies around the time the SRI report was published not because they read reports about the office of the future. They simply imagined it.

Update: Gordon Moore quoted in VentureBeat “Once I made a successful prediction, I avoided making another.” and ““I wish I had seen the applications earlier. To me the development of the Internet was a surprise. I didn’t realize it would open up a new world of opportunities.”

Originally published on Forbes.com

Posted in Predictions | 1 Comment

ABI Research: 7.7 Million Autonomous Truck Fleets to Ship by 2025


ADAS=Advanced Driver Assistance Systems

Truck platoons are the most imminently anticipated application of highly automated driving in commercial vehicles. A fusion of forward-looking radar and V2V communication enable fleets of trucks to safely maneuver with a short distance between vehicles. The reduction in aerodynamic drag for following vehicles, and buildup of pressure behind the lead vehicle yields impressive fuel efficiencies, with various tests reporting convoy savings of between 5% and 10%. “With most fleet operators attributing some 30 to 40% of their operating costs to fuel expenditure, the savings presented by platooning are significant,” comments James Hodgson, Research Analyst, ABI Research.

As technology progresses and regulations adapt to accommodate greater vehicle automation, further benefits to fleet operators will come in the shape of labor productivity gains and better asset utilization. Currently, solutions from pioneers such as Peloton Technology require active intervention from the following driver to keep the vehicle within the lane of travel, but in the future the driver of the lead vehicle could be in sole control of all vehicles in the convoy; allowing following drivers to rest, or eliminating the need for them altogether.

Free ADAS and Active Safety Webinar on May 21, 2015 at 11 am ET for a deeper look at ABI Research’s commercial trucking coverage and the convergence of ADAS, telematics, autonomous driving, and big data,

These findings are part of ADAS and Autonomous Driving Technology in Trucks and Commercial Vehicles, a report from ABI Research’s Automotive Safety and Autonomous Driving and Commercial Vehicle Telematics Market Research.

Posted in Internet of Things | Leave a comment

2 New Surveys About the Market for Data Scientists

Two new surveys tell us a lot about both the supply and demand sides of the hot market for data scientists, “the sexiest job of the 21st Century.”

On the demand side—the challenges of recruiting, training, and integrating data scientists—we have the MIT Sloan Management Review and SAS fifth annual survey of 2,719 business executives, managers and analytics professionals worldwide. On the supply side—the talent available and what salaries it commands—we have the second annual Burtch Works Study, surveying 371 data scientists in the U.S. (see also the video presentation at the end of this post).

The median salary of a junior level data scientist is $91,000, but those managing a team of ten or more data scientists earn base salaries of well over $250,000, according to Burtch Works. Supply is still tight and top managers enjoyed over the last year an eight percent increase in base salary and median bonuses over $56,000. When changing jobs, data scientists see a 16 percent increase in their median base salary.

Who are these data scientists that are so much in demand? The vast majority have at least a master’s degree and probably a Ph.D., and one in three are foreign born. But with a younger generation of data scientists, freshly minted from more than 100 graduate programs worldwide, the median years of experience dropped from 9 in 2014 to 6 in 2015.

As data science is increasingly adopted by all companies in all industries, the proportion of data scientists employed by startups—the firms that have dominated the application of big data analytics— declined from 29 percent in 2014 to 14 percent in 2015.

It is the mainstreaming of data science and the specific challenges of acquiring and benefiting from this still-scarce talent pool that is the focus of the MIT Sloan Management Review survey. Four in ten (43%) companies report their lack of appropriate analytical skills as a key challenge but only one in five organizations has changed its approach to attracting and retaining analytics talent.

As a result of the scarcity of data scientists, 63 percent of the companies surveyed are providing formal or on-the-job training in-house. “One big plus of developing analytics skills among current employees,” says the report, “is that they already know the business.” These companies are also doing more to train existing managers to become more analytical (49%) and train their new data scientists to better understand their business (34%). Still, half of the survey respondents cited turning analytical insights into business actions as one of their top analytics challenges.

To better manage these challenges, the study recommends giving preference to people with analytical skills when hiring and promoting, developing analytical skills through formal in-house training, and integrating new talent with more traditional data workers.

“Infusing new analytics talent without proper support and guidance can alienate traditional data workers and undermine everyone’s contributions,” says the report. Yet only 27% of companies report that they successfully integrate new analytics talent with more traditional data workers. So even after managing to find (and pay for) the data science talent, there is no guarantee for the desired results, either because of the lack of understanding of the business by the new recruits, resistance from current employees engaged in data preparation and analysis, or failure to translate new insights into meaningful action.

Many companies have responded to these challenges by creating new roles and responsibilities and devising new organizational structures. The report points out that the range of analytics skills, roles and titles within organizations has broadened in recent years. What’s more, new executive roles, such as chief data officers, chief analytics officers and chief medical information officers, have emerged to ensure that analytical insights can be applied to strategic business issues.

Whether the work is centralized or decentralized, data science and analytics should be perceived and managed by companies as a professional function with its own clear career path and well-defined roles. Tom Davenport asked in a recent essay: “When was the last time you saw a job posting for a ‘light quant’ or an ‘analytical translator’? But almost every organization would be more successful with analytics and big data if it employed some of these folks.”

Davenport defines a “light quant” as someone who knows something about analytical and data management methods, and a lot about specific business problems, and can connect the two. An “analytical translator” is someone who is extremely skilled at communicating the results of quantitative analyses.

Data science is a team sport that requires the right blending of people with different skills, expertise, and experiences. Data science itself is an emerging discipline, drawing people with diverse educational backgrounds and work experiences. Typical of the requirements for a graduate degree is what we find in a recent announcement from the University of Wisconsin’s first system-wide online master’s degree in data science: “The Master of Science in Data Science program is intended for students with a bachelor’s degree in math, statistics, analytics, computer science, or marketing; or three to five years of professional experience as a business intelligence analyst, data analyst, financial analyst, information technology analyst, database administrator, computer programmer, statistician, or other related position.”

As with any team sport, there are stars that are paid more than the average player. According to Glassdoor (HT: Illinois Institute of Technology Master of Data Science program), the average salary for data scientists is a bit more than what Burtch Works reported, at over $118,000 per year. (By the way, Glassdoor reports the average salary for statistician is $75,000 and $92,000 for a senior statistician).

It’s possible that the Glassdoor numbers include more of what Burtch Works calls “elite data scientists.” Do we know who is in the elite of top data science players? The closest we get to identify the MVP of data science is the Kaggle ranking of the data scientists participating in its competitions. Currently, Owen Zhang is number one. Zhang says on his profile that “the answer is 42” and his bio section tells us that he is “trying to find the right question to ask.” He lists his skills as “Excessive Effort, Luck, and Other People’s Code.”

Zhang is currently the Chief Product Officer at DataRobot, a startup helping other data scientists build better predictive models in the cloud. He is also yet another example of how experience and skills still matter today more than formal data science education. His educational background? Master of Applied Science in Electrical Engineering from the University of Toronto.

Originally published on Forbes.com

This Burtch Works webinar provides highlights from the 40+ pages of compensation and demographic data in the report, which is available for free download here: http://goo.gl/RQX1xd

Posted in Data Science, Data Science Careers | Leave a comment

Data Science on Cloud Foundry (Video)

Data Scientists frequently need to create applications that enable interactive data exploration, deliver predictive analytics APIs or simply publish results. Cloud Foundry provides an ideal platform for data scientists by making it easy to quickly deploy data driven apps backed by a variety of data stores.

Posted in Data Science | Leave a comment

Top 10 Most-Funded Big Data Startups April 2015

These are the top ten most-funded startups in the big data space:

Cloudera                                             $1040 Million                       Hadoop-based software, services and training

Palantir Technologies                      $950                                        Analytics applications

Domo                                                   $450                                        Cloud-based business intelligence

MongoDB                                           $311                                          Document-oriented database

InsideSales.com                               $199                                         Cloud-based predictive analytics

Mu Sigma                                            $195                                         Data-Science-as-a-Service

DataStax                                              $190                                        Apache Cassandra-based platform

Dataminr                                             $180                                        Social media analytics

MapR                                                    $174                                          Hadoop-based software, services and training

Birst                                                       $156                                          Cloud-based business intelligence

This list is based on my own research, augmenting the very helpful but sometimes unreliable, CrunchBase database. The funding includes VC contributions as well as investments by established companies.

In the two years that I’ve been compiling and tracking this list, it has changed considerably. Cloudera was the most-funded big data startup on the March 2013 list and is still the most-funded big data startup today. But Cloudera, Mu Sigma, MongoDB, and DataStax are the only companies on today’s list that were on it two years ago. And, reflecting the recent increase in the size and scope of venture capital investments, one of the companies that is new to the list, Dataminr, has raised $130 million in just one round of funding last month, a sum that would have put it as number 2 on the 2013 list.

Another company new on the list, Insidesales.com, has raised “only” $60 million last month, $40 million less than in its previous round. It may well be yet another case of company founders and exiting investors agreeing to accept funds from investors eager to invest (or up their previous investment)  in a promising company. One of the original investors in Insidesales.com was Josh James who co-founded Omniture, a web analtyics company that was sold in 2009 to Adobe for $1.8 billion. James went on to found Domo in 2010 and raised an impressive $125 million in February 2014 because investors were eager to bet on whatever he was betting on.

Earlier this month, Domo announced that it has raised a new $200 million funding round at a $2 billion valuation.  James told Re/Code that “the company will be ready to go public in six months.” When it does so, it will join Hortonworks, a startup that used to be on this list and went public in December 2014 after raising $248 million.

Originally published on Forbes.com

Posted in Big Data Landscape, startups | Leave a comment

Big Data Market 2011-2026, From $7.6 to $84.69 Billion


Wikibon: For the calendar year 2014, the Big Data market – as measured by revenue associated with the sale of Big Data-related hardware, software and professional services – reached $27.36 billion, up from $19.6 billion in 2013. While growing significantly faster than other enterprise IT markets, the Big Data market’s overall growth rate slowed year-over-year from 60% in 2013 to 40% in 2014. This is to be expected in an emerging but quickly maturing market such as Big Data, and Wikibon does not believe this slightly slower growth rate indicates any structural market issues.

Posted in Big data market | Leave a comment

3 Recent Books on Data Mining, Data Science and Big Data Analytics

Data-MiningNow that most of the hype around big data has died down, overtaken by the buzz over the Internet of Things, we are sometimes treated to serious discussions of the state-of-the-art (or science, for that matter) in data analysis. If you are planning a career as a data scientist or you are a business executive trying to understand what the data scientists are telling you, three recent books provide excellent and accessible overviews:

The Analytics Revolution: How to Improve Your Business By Making Analytics Operational In The Big Data Era by Bill Franks

Data Mining For Dummies by Meta S. Brown

Data Science For Dummies by Lillian Pierson

Bill Franks is the Chief Analytics Officer for Teradata, and his specialty is translating complex analytics into terms that business users can understand. The Analystics Revolution follows Franks’ Taming the Big Data Tidal Wave, which was listed on Tom Peters’ 2014 list of “Must Read” books.

“With all the hype around big data, it is easy to assume that nothing of interest was happening in the past if you don’t know better from experience” says Franks. The over-excitement about big data caused many organizations to re-create solutions that already exist and build new groups dedicated to big data analysis, separate from their traditional analytics functions. As a correction, Franks advocates “a new, integrated, and evolved analytics paradigm,” combining traditional analytics on traditional data with big data analytics on big data.

The focus of this new approach–and the book–is Operational Analytics. It takes us from the descriptive and predictive analytics of traditional and big data analytics to prescriptive analytics. It pays close attention to the numerous decisions and actions, mostly tactical, taking place every day in your business. Most important, it places great emphasis on the process of analytics, on embedding it everywhere, and on automating the required response to events and changing conditions.

“Of course,” says Franks, “it takes human intervention to decide that an operational analytics process is needed and to build the process.”  But once the process is designed and turned on, the process accesses data, performs analysis, makes decisions, and then actually causes actions to occur. And humans are crucial to the success of this new brand of automated analytics, not only at the design phase, but also in the on-going monitoring and tweaking of the process.

An example of operational analytics is the development of an improved maintenance schedule using sensor data. There will be no value in the Internet of Things without an automated process for data analysis and action based on that analysis. “As traditional manufacturers suddenly find themselves embedding sensors, collecting data, and producing analytics for their customers, industry lines blur. Not only are new competencies needed, but the reason customers choose a product may have less to do with traditional selection criteria than with the data and analytics offered with the product,” says Franks.

The practical advice Franks provides in the book ranges from how to set up an analytics organization to developing and maintaining a corporate culture dedicated to discovery (finding new insights in the data and quickly acting on them) to implementing operational analytics. The Analytics Revolution is an excellent guide to the new business world of blurred industry lines and innovative data products.

If you are ready to move on from understanding the why of analytics today and how to think about it in a broad business and organizational context to a more specific understanding of the how of analyzing data, Data Mining for Dummies by Meta Brown should be your first step. The book was written for “average business people,” showing them that you don’t need to be a data scientist and “you don’t need to be an expert in statistics, a scientist, or a computer programmer to be a data miner.”

Brown is a consultant, speaker and writer with hands-on experience in business analytics. She’s the creator of the Storytelling for Data Analysts and Storytelling for Tech workshops. In Data Mining for Dummies, Brown tells the story of what data miners do.

It starts with a description of a day in the life of a data miner and goes on to discuss in clear, easy-to-understand prose all the key data mining concepts, how to plan and organize for data mining, getting data from internal, public and commercial sources, how to prepare data for exploration and predictive modeling, building predictive models, and selecting software and dealing with vendors. Data Mining for Dummies is an excellent step-by-step guide to understanding data mining and how to become a data miner.

If you are ready to move on from understanding data mining and being a data miner to more advanced tools and applications for data analysis, Data Science for Dummies by Lillian Pierson should be your first step. The book was written for readers with some technical and math skills and experience, but it aims to provide a general introduction to one and all: “Although data science may be a new topic for many, it’s a skill that any individual who wants to stay relevant in her career field and industry needs to know.”

Pierson is a data scientist and environmental engineer and the founder of Data-Mania, a start-up that focuses mainly on web analytics, data-driven growth services, data journalism, and data science training services. “Data scientists,” she explains, “use coding, quantitative methods (mathematical, statistical, and machine learning), and highly specialized [domain] expertise in their study area to derive solutions to complex business and scientific problems.

Data Science for Dummies is an excellent practical introduction to the fundamentals of data science.  It provides a guided tour of the data science landscape today, from data engineering and processing tools such as Hadoop and MapReduce to supervised and unsupervised machine learning, statistics and mathematical modeling, using open-source applications such as Python and the R statistical programming language, finding resources for publicly-available data, and data visualization techniques for showcasing the results of your analysis. Stressing the importance of domain expertise for data scientists, Pierson provides detailed examples of applying data science in specific domains such as journalism, environmental intelligence, and e-commerce.

“A lot of times,” says Pierson, “data scientists get caught up analyzing the bark of the trees that they simply forget to look for their way out of the forest.” The three books reviewed here provide a handy map to the maze of data analysis and a safe conduct pass for business executives, IT staff, and students, ensuring that they successfully get in and out of the data forest. Remember, as ones and zeros eat the world, data is the new product and operational analytics, data mining, and data science is the new process of innovation.

Posted in Big Data Analytics, Data Science | Leave a comment