Big data is a broad term for data sets so large or complex that traditional data processing applications are just no longer adequate.
Already back in 2001, Gartner analyst Doug Laney defined data growth challenges and opportunities as being three-dimensional. Since then more qualities have been added:
Volume, the size of the data determines the value and potential of the data under consideration, and whether it can actually be considered “Big Data” – a term related to size.
Velocity, how fast the data is generated and processed to meet demands and the challenges
Variety, the range of data types and sources.
Veracity, the quality of captured data can vary greatly. Accurate analysis depends on the veracity of source data
Variability, the inconsistency data can show at times hampering the process of effectively handling and managing
Complexity, data management can be very complex, especially when large volumes of data come from multiple sources. Data must be linked, connected, and correlated so users can grasp the information the data is supposed to convey.
10 roadblocks to implementing Big Data analytics
Big Data and business analytics for most enterprises are still developmental. The road to an effective Big Data operation is fraught with challenges. Here are 9 of the major obstacles (among many, many more) we believe have to be tackled by companies considering implementing Big Data analytics.
Traditional servers in enterprise data centers are not designed for processing Big Data. At very least analytics servers, and in some case high performance computing (HPC) servers and applications will be needed. This means making strong case for new IT investment.
- IT know-how
Big Data requires a different strategy from online transactional data for both storage and processing. Big Data processors run several processing threads in parallel and do not proceed sequentially, like online transactions. Storage strategy must also change, starting with tiering of storage that places the most sought-after data on faster storage devices, such as cache/SSDs, and less frequently accessed data on slower hard disks. Storage staff must have the latest in storage management training.
- Data cleanup
Cleaning up data to ensure that incomplete, inaccurate, and duplicate data is removed should be the first step of any Big Data project.
- The storage bulge
The amount of data under management in enterprises has grown five times over the past four years. Benefit will only arise if enterprises can sort through the data, deciding what is important, and either archiving or getting rid of the rest.
- Don’t underestimate the data integration challenges
Deriving value from Big Data usually is dependent on processing unstructured information—video feeds from shop floors, telematics sensors in vehicles, GPS sensors in mobile devices, speech to text files and a host of other bits and pieces of information that are not readily processed.
- Vendor role clarification
Because so many organizations are inexperienced with Big Data, many vendors offer turnkey solutions complete with prefab analytics reports. Eventually, though, you’ll want to start developing your own internal expertise.
- Business and IT alignment
Business goals and IT Big Data strategy should be tightly aligned before any IT investments are made. Know what you are going after before you invest in Big Data and analytics.
- Focus on data management.
The IT department, specifically data architects, need to determine where the data and apps will reside. In one on-premise system or together in a cloud implementation? The traditional Business Intelligence approach of 10 years ago—trying to have everything in one data warehouse—frequently failed.
- Data visualization is important for Big Data users
Front line professionals and others who are expected to be able to take action based on Big Data insights need an easily digestible delivery mechanism.
And one more for luck:
- Big Data access via mobile devices
The latest generation of touch-enabled smartphones and tablets are driving a huge change in the way companies operate and communicate internally and with their partners and customers. IT managers must address their demands for access to manipulate Big Data information and insights via their mobile devices.