Big data makes research more difficult say data scientists

Given the amount of hype surrounding it you could be forgiven for thinking that big data is the answer to most business and scientific problems.

But a new survey by database specialists Paradigm4 reveals that almost three-quarters of data scientists believe that big data has made their research harder. The reason for this is not, as you might expect, the volume of data but the variety.

The results, from a survey of 111 US data scientists, show that 40 percent say they struggle managing new types and sources of data and 36 percent say that getting answers from big data takes too long.

Hadoop comes in for criticism too with 76 percent saying it's too slow, takes too much effort to program or has other limitations. So much so that 35 percent of scientists who have tried using Hadoop or Spark say they have now stopped using it.

Problems cited include 39 percent saying it takes too much effort to program and 37 percent saying it's too slow for interactive ad-hoc queries. 30 percent also say it's too slow for performing real-time analytics.

It seems that despite all this, however, big data still has a key role to play. 91 percent of those surveyed say they’re using or plan to use complex analytics on their big data within two years.

If you want further insight into the trials of being a data scientist, you can get hold of the full survey results on the Paradigm4 website or there's a summary of the findings in handy infographic form below.

Image Credit: aslysun / Shutterstock