Big data, IoT, Business Intelligence and small data (T-SQL Tuesday #95)
Big data is both a buzzword, or phrase, and a booming area of technology. Technical professionals and companies alike are investing a lot in big data and I want to hear your thoughts on the topic. Your post can be about; how big data affects the industry and our careers, how the cloud is enhancing our ability to work with big data, how you deal with big data in SQL Server on-premises, NoSQL, development challenges and strategies for working with internet of things data, or anything else you come up with.
Well, here is my story...
Five years ago, I started to investigate the hype around Big Data. Back then I was working in a Business Intelligence (BI) company, and every now and then we had customers asking questions like “is this Big Data thing for us?” And so, we researched tooling and methodologies and thought of business cases for our customers within Big Data. Almost every time, we had plenty of ideas for them, but we were also facing the dilemma of their “Small data” not being used.
In many cases, our customers said, “That’s very interesting, but…” and named so many reporting problems, that were easy to solve with standard BI methods and tools. Therefore, we started to include a story about BI maturity in our presentations on Big Data, where we argued that a company really should consider if they got “bang for the buck” on the lower levels, before investing too much in projects on a higher level:
Driving this was partly the fact that methods and tools on lower levels were well understood and mature, whereas on the higher levels, core technologies existed, but tools and best practices were very immature (we called it Cowboy Land or the Wild West). This was also inspired by articles on “Small data” in Harvard Business Review. I still think this is a relevant discussion, although the tooling around Big Data and Machine Learning have matured quite a bit.
At the time in Rehfeld R&D, we experimented with making Effektor a metadata repository for a Hadoop data warehouse, where instead of generating tables and ETL processes in the different data warehouse layers, the synchronization engine in the product would generate the Hive objects on top of Hadoop tables. We never made more than an overall spec and a prototype, but the experiment gave us some insight into the technologies around Hadoop.
Around that time, Phillips released the Hue lightbulbs, and our COO bought us two packs to play with. The idea was to create a physical BI dashboard, where lightbulbs would display KPIs, and change color according to its value and the KPI threshold. I still think that was a brilliant idea, and I would love to see more use of consumer electronics in enterprise BI.
At the Directions conference this year, I attended a session on Internet of Things (IoT) and Dynamics NAV. The presenters talked a lot about how to get started with IoT (using sensor data to improve business processes, especially in manufacturing), and I totally believe this is important. But I also sat there thinking that there must be a market out there to go the other way: instead of consuming and aggregating data from sensors (a data pull model), how about pushing small data to IoT devices? Since a lot of electronics devices can be controlled using WIFI, we should utilize that to embed effects such as light, sounds, temperature, smell? into our business applications.
That’s my 25 cents for this month’s T-SQL Tuesday topic. Stay tuned.
Read more about Phillips Hue here:
Read more about T-SQL Tuesday #095 here: https://www.sqlhammer.com/tsql2sday-95-invitation-big-data/