Logz.io Combines Open Source, Cloud, Big Data and Machine Learning for DevOps and SRE

cognitive insights screenshot

Logz.io Cognitive Insights

90% of startups fail. Those that survive and thrive either capitalize on a new technology or provide a timely response to a new market development. Logz.io does both, combining the power of at least 4 technology trends—open source, cloud computing, big data analytics, and machine learning—while addressing a new group of influencers in IT purchasing—DevOps staff and Site Reliability Engineers (SREs).

Logz tells its customers what’s going on with their software applications. It offers an enhanced version of the open source ELK stack which combines an enterprise search engine with log analytics and visualization tools. On top of ELK, it has developed Cognitive Insights, an artificial intelligence platform that detects overlooked and critical events and provides the user with actionable data about context, severity, relevance, and recommended next steps.

“Log data is very noisy, very susceptible to false positives,” says Tomer Levy, co-founder and CEO of Logz. To figure out when and where its customers should pay attention, Logz combine in one very large database in the cloud (i.e., Amazon Web Services cloud) data from 3 sources: What it finds by searching the relevant online discussion forums where software developers talk about specific issues and their solutions; what it learns from the experience of its customers; and what it learns from how its customers have reacted to specific events.

In other words, Logz crowdsources expert knowledge from the best engineers in the world and then applies it in real time to offer recommendations to its customers as critical events happen. “We take communal knowledge and run it on your data,” says Levy. Without the type of assistance they get from Logz, software engineers can rely only on their own limited knowledge and experience and typically just guess at what needs to be done.

With today’s complex, massive, and distributed IT infrastructures, it is close to impossible to find the correlations and relations between specific occurrences of the same piece of information in the vast amounts of log data. At this scale, you need artificial intelligence to observe, learn and provide recommendations based on the experiences of hundreds of software development teams and hundreds of thousands of contributors to discussion forums.

Open software and open discussions have fueled the rise of “digital natives” such as Google, Facebook, and Amazon. These companies, in turn, have contributed greatly to all the technological developments Logz is capitalizing on—developing in-house new software applications to allow them to manage IT operations at unprecedented scale and then sharing these applications with the world as open software; building a massive, distributed IT infrastructure, and in some cases, providing it as a cloud service to other organizations; collecting, analyzing and running their companies on big data; and using machine learning to analyze the data and advance every aspect of their business. In the process, they have also created new IT-related roles and changed how IT solutions are acquired.

The IT world used to be neatly divided between software developers responsible for developing applications and operations staff responsible for running and maintaining the applications in production. But this division of labor did not work for the online life of digital natives. Specifically, early in Google’s corporate life, its new practice of constantly revising its software and continuously introducing new features, while at the same time maintaining high quality user experience, required a new approach. The solution was the creation of a new team, Site Reliability Engineering, applying the disciplines and practices of software engineering to managing IT operations and running production environments.

The marriage of software engineering with IT operations in the form of the new role of the Site Reliability Engineer spread quickly to other high-tech companies. But about ten years ago, the concept was taken a step further in the form of the new DevOps movement. As all businesses are becoming digital businesses, they are also adopting the practice of Agile software development, continuously revising their software applications. They also need to answer the new requirements of employees, customers, and partners expecting an always-on, secure, and reliable IT infrastructure.

Today, IT operations has become part of software development. For applications to be resilient and fault tolerant, they need to include constant monitoring, adaptability to large swings in load, and Web performance optimization. The DevOps team takes care of these new requirements, now at a growing number of organizations, of all sizes and in all industries. And they have an increasingly significant role in evaluating and recommending tools that ensure the new IT infrastructure is always humming, always performing.

“DevOps is about accountability,” says Levy, “not just writing the code but also overseeing how the code is running in production.” The DevOps team is where business users now go to complain if they think there is a problem with how the application is working. This new development in the IT market has created an opportunity for Logz (and many other startups) to address a new influential group that can give them an entry into the business.

“We launched the product in October 2015, we have raised $24 million to date, and now there are more than 1000 companies in 80 different countries that use Logz every day,” says Levy. While he describes the product as a “DevOps solution,” there are a number of different uses for it in addition to application monitoring, including business intelligence, compliance, and security, the latter now accounting for 20% of revenues. For example, Rent-A-Center has reported that Cognitive Insights helped it detect potential security threats before they impact its customers. And Logz helped Dyn quickly respond to and mitigate the unprecedented DDoS attacks it recently experienced, attacks that temporarily shut down or slowed many popular websites.

Other brand names Logz.io counts among its customers include British Airways, CNN, Electronic Arts, and Oracle. Sitting at the confluence of the latest technology trends and being propelled by a rising group of IT influencers is a proven recipe for startup success.

Originally published on Forbes.com