Mining the Past to Predict the Future

Big Data is much more than mining our Web clicks to predict our next purchasing decision. For example, can we develop a “historically conscious” smarter grid using Big Data tools? In the June issue of IEEE Computer,  Cynthia Rudin, Rebecca J. Passonneau, and Axinia Radeva of Columbia University’s Center for Computational Learning Systems team up with Steve Ierome and Delfina F. Isaac of Consolidated Edison of New York to report on how they explored this question.Specifically, they “sought to determine whether Con Edison data regarding past failures on the city’s low-voltage grid—manhole fires, explosions,  smoking manholes, and burnouts—could be used to predict, and thus prevent, future events.” New York City has the world’s oldest grid and the data in Con Edison’s databases goes back to the 1880s. Processing the text in trouble tickets, reconstructing each manhole’s decade-long event history from a variety of data sources, and using machine learning algorithms, they were able to create a “meaningful event prediction model, targeted to predicting failures on individual manholes.”

Their conclusion potentially applies to your next Big Data project: “…researchers can repurpose extremely raw historical data for use in prediction. Databanks similar to Con Edison’s are commonly not repurposed, left instead to become ‘data tombs.’ But researchers often can analyze and exploit such data to make important contributions—in this case, to devise a better procedure for electrical grid inspection and repair that could improve public safety and energy reliability. The challenge is navigating an ocean of possible data processing tasks, some more rewarding than others, to achieve a more accurate predictive model.”

The article, titled “21st-Century Data Miners Meet 19th-Century Electrical Cable, is currently available only to subscribers or for purchase.