As follow up to my blog Cyber Insecurity in our new World of IoT, from my attendance at the Executive Women’s Forum Summit on Unintended Consequences: Internet of Things (IoT) and Big Data , I’d like focus on the growing Information Governance Obesity Challenge creating a Data Bloat for information risk and IT systems in today’s Big Data world.
Now, when I hear the words “Obesity Challenge”, my first reaction was about the state of the weight of the average American. Americans rank #3 in the world for obesity for world populations, and the weight of the average American woman today is roughly 164 pounds, the same weight as the average American man in the 1960’s. While we may be taller as a species now, our diet has changed. In that same decade, homes had a dial-up phone, rabbit ears for Television, radio was king, and no one could have anticipated fifty years later what our data appetite would be for information and devices.
With the explosion of the internet, digital media, the internet of things, data is proliferating faster than ever. The technology appetite has accelerated for Big Data not only by generations, but by the technology industry itself. In fact, it is estimated that Big Data drove over $34 Billion in IT spending. It is not just about the connections between systems, but how the technology itself is driving new opportunities for data usage, personalization.
Gartner recently outlined the concept of Dark Data. Dark Data is all the data in an organization that is not a part of day to day operations. Gartner compared Dark Data to all the furniture and junk in your basement, or all the hidden food in the pantry that just continues to grow, taking up space.
According to Wiki’s definition, Big Data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications. This leads to key challenges in managing Big Data within an information governance program. I’ll try and keep the Data Bloat theme going as we need to factor into our information governance appetite, diet, and calorie plan how we collect and use information
What’s Your Organization’s Data Appetite Strategy?
- Capture: How much data do you collect, consume on a daily basis?
- Curation: Do you have a keep it forever mentality?
- Storage: Do you know where your data is? Do you have dark data?
- Search: How much wasted time do you spend searching for meaningful data?
- Sharing: Do you have a matrix or math vectors for where the data is going and to whom?
- Analysis: What is the expiration date for data value?
- Visualization: What type of scale do you need to quantify the volume of data?
Due to our drive for convenience, choice, personalization, we have become a nation of data hoarders using a keep it “just in case” storage mentality. We justify our approach since today storage is cheap but litigation can be very costly with a keeping data forever approach in an eDiscovery situation. In fact the value of data degrades over time, for meaningful data. It is the rare business model, like monitoring and measuring weather patterns since the beginning of time that require permanent data storage.
Buying patterns from 15 years ago are not as valuable as measuring last holiday season to this as we shift to an online economy. Even from a consumer media perspective, data migration from VHS to DVD has been shifted to the web with streaming.
Organizations have the same challenges of data stacks starting with physical files, to unstructured data, to databases. Building web tracking and online collection of data, builds big data lakes to surf and fish to find factoids, creating an almost Cyber-Fishbowl of monitoring.
Three steps to implement your Data Bloat Diet Plan
- Implement the 80% rule – go after the data calories that give you the most impact.
- Eliminate “Convenience Copies” of data that add inches to your Data stack.
- Create a plan to find your Dark Data.
Don’t be too IT-Centric – business systems and business processes are creating more data. At the end of the day it’s about a balanced diet. Collecting, Consuming, and Storing data for future use needs a calorie plan and a combination of exercise to eliminate unwanted pounds. Dark vegetables have the most benefits, and figuring out what data is not valued in your data to day operations can reduce your IT and business data bloat.