Microsoft's Big Data-driven improvement efforts flounder

Microsoft Research and a researcher from North Carolina State University have studied Redmond's internal use of Big Data tools and found that the company needs to change in order to put data to work. And so, probably, does yours.

The paper, The Bones of the System: A Case Study of Logging and Telemetry at Microsoft (PDF) considers data produced by Microsoft software and consumed by different teams across the company. Specifically, the study looks at what's come to be called “event data” inside Microsoft. That term covers both log files and the dashboards and telemetry that Microsoft uses to present insights derived from automated processing of log files.

Microsoft's built the latter tools, the paper says, because only developers tend to look at log files and only do so for troubleshooting. Event data, by contrast, is hoped to make it possible for a wider team to benefit from data, and therefore make better decisions about how to develop software and how to develop future products.

The researchers therefore cast a wide net, interviewing and surveying developers, program managers, operations engineers, data scientists and others. Across all those disciplines, people had trouble putting data to work. Some didn't trust the data they were sent, sometimes because they didn't trust filtering mechanisms that reduce data sets to workable sizes. Some teams found that only certain expert team members could make sense of the data, creating bottlenecks. Others complained that even with a groomed feed of event data, expertise wrangling technical tools to create role-specific feeds was substantial and had high risk of failure.

Teams didn't share information well with one interviewee complaining of “a lack of knowledge of what event data is added to the system, and who is consuming it.”

None of these issues, the paper concludes, are a sign Big Data's is failing. Rather, Microsoft knows how to collect and package data, but doesn't yet know how to package it in ways that help its people to use it well.

“As more companies transition to a data-driven culture,” the paper therefore concludes, “we expect that researchers will need to develop new processes and tools to meet these new challenges.”

There's no sure-fire recipe for success with Big Data? And not even mighty Microsoft has cracked it at this early stage of the game? Knock us down with a vulture feature. ®