If You Thought Big Data Was Complicated, You Must Not Have Met IoT

Those working in IT who find big data projects to be overly complicated might want to brace themselves for Internet of Things (IoT) projects. This type of project requires velocity and volume that can easily put big data to shame, and the need for enabling real-time decision-making and analysis is tremendous. Companies will need the ability to process thousands – if not millions – of events every second on their devices.

Real-time storage solutions have been developed in order to deal with large volumes of data, and these have certainly made this type of project somewhat easier. However, it’s important to understand that IoT projects and big data ones, despite having several things in common, do not have identical needs. One of the big differences is that while big data projects typically don’t need to make use of the data they’ve collected right away, time is a vital component of IoT projects.

“Big Data On Steroids”

In fact, John Gantz, a researcher for IDC, has said that he believes IoT solutions will demand decisions within just one minute of detecting a particular situation. Moreover, there are a lot of factors to consider in terms of velocity, not the least of which is the fact that the data culled from devices is typically in a raw format that is going to need some handling before it’s usable. This raw data must quickly be organized and then transformed into whatever derived values are applicable in that particular case. Then, it needs to be enriched with data from enterprise sources in many cases. The need to constantly retrieve this type of information could be the biggest challenge of all.

It’s a very complicated process indeed, and the incredible challenges it poses have has led Software AG’s IoT Director, Bart Schouw, to refer to IoT as “big data on steroids.”

Register for EF6