Big Data about to bottom out, says Gartner

Big Data hype has peaked and adopters are about to enter Gartner’s dread trough of disillusionment, says one of the firm’s analysts, Svetlana Sicular.

Hype about Big Data is certainly prevalent: here at Vulture South the term is often thrown around by vendors who in past years were content to describe their data-crunching products as offering ‘business intelligence’ or ‘business analytics’ tools. Big Data has even been suggested to us as applicable to small business, despite such organisations seldom possessing the infrastructure or expertise to put it to work.

Sicular says even enthusiasm for the tool most-often associated with Big Data – Hadoop – is waning, as despite good efforts they “… do not realize that they are ahead of others and think that someone else is successful while they are struggling.”

Lots are struggling, she says, because “they are disappointed with a difficulty of figuring out reliable solutions.” Sentiment analysis, a customer-mood-detecting technique often touted as a way to monetise user-generated content, is proving tough as vendors are yet to meet users needs.

“Difficulties are also abundant when organizations work on new ideas,” Sicular writes, especially when organisations try to link unstructured data sources.

Sicular offers the following as one case study of a Big Data user beginning to feel disillusioned:

“Several days ago, a financial industry client told me that framing a right question to express a game-changing idea is extremely challenging: first, selecting a question from multiple candidates; second, breaking it down to many sub-questions; and, third, answering even one of them reliably.”

We've been waiting for an excuse
to use this photo for a while now

“Formulating a right question is always hard, but with big data, it is an order of magnitude harder, because you are blazing the trail (not grazing on the green field),” she adds.

Another reason for a loss of confidence in Big Data is that it does not deal in absolute but instead produces what Sicular describes as “a proof of your hypothesis with a certain degree of confidence” rather than a concrete answer. That kind of result means users need to be satisfied with what she calls “satisficing” solutions, “the first solution that appears good enough”.

Sicular therefore expects Big Data will get bad press for a while. Starting …. now! ®