Big data-- it's the heavyweight champion of tech hype in the past few years. But suddenly it seems to have gone quiet. Is it dead? One could be forgiven for thinking so. But no, it isn't dead. It's just passed the top of the hype cycle. And that's actually a good thing, because although "big data" the buzzword may be dead, big data analytics in the real world is getting ready to roll up its sleeves.
A quick recap: Every year, Gartner puts out its closely watched "Hype Cycle of Emerging Technologies" report. In the last few years I have watched big data rise to the top of the hype, reaching "the peak of inflated expectations" in 2013. But it had fallen into the trough of disillusionment in 2014. So in 2015, big data would have been expected to move into the slope of enlightenment. Surprisingly, Gartner too believed it would take five to ten years for big data to reach the plateau of productivity and maturity.
But something amazing happened last summer. Gartner issued its annual hype cycle for 2015 with one clear absentee -- big data. Where did it go? According to Betsy Burton, VP distinguished analyst who heads up the research and the Gartner report, a decision was taken to retire big data from the hype cycle as it is "no longer an emerging technology" but has become "prevalent in our lives," she says.
Though I think it's safe to say that big data has been embraced by the real world, now is the time for communications service providers -- spanning wireless and pay-TV -- to bring the benefits of big data analytics to their customers, in the form of an improved customer experience on a range of levels. To elaborate, let's break down big data into three layers that I call "the hard work," "the toys" and "the money."
The money layer
The money layer refers to the analytics -- or the sexy part of big data. It includes the previously unthought-of insights derived by sifting through mass data sets and developing algorithms that drive business value. These insights enable service providers to know (even before the customer) when the customer is about to contact the call center, and act proactively to resolve the situation, thus eliminating the need for a costly phone call and improving the customer experience.
The toys layer
The toys layer refers to new open source technologies like Hadoop and Spark which harness the cost efficiencies of distributed computing. Companies that provide these technologies like Hortonworks and Cloudera are thriving as service provider IT departments rush out to buy these shiny toys.
Up until now, all the chatter that drove big data to number one in the hype hit parade has really focused on these two areas. But the third layer -- the hard work -- has been overlooked, as hard work often is. It needs to be understood before big data can divorce itself from the hype and get down to work.
The hard work
This hard work will take many forms. Service providers need to transform themselves into data-driven businesses to take complete advantage of all the value that can be derived from big data. For example, smartphones can significantly lower lending costs in the developing world, because the apps they run generate huge amounts of data -- texts, emails, GPS coordinates, social-media posts, retail receipts and so on -- indicating thousands of subtle behavioral patterns that correlate with repayment or default. But with most organizations that are just getting started, they are discovering that the path towards this goal is surrounded by challenges.
They will need to evolve their data management environments from traditional, appliance-based data warehouses to "data lakes" which are hybrid landscapes of mass-volume, low-cost storage that can house structured, non-structured, third-party and in-house data. This helps them to create rich and massive data sets that can reveal valuable insights.
But the real hard work lies in the next step: extracting and cleansing relevant data from the source systems, such as billing, networks and CRM, and hydrating the newfound data lakes -- which requires both time and money -- and the data needs to be fresh, relevant and analytics-ready.
So, keeping the "hard work" mantra in mind, here's what we can expect for big data analytics -- the technology, not the hype -- in 2016:
Real-time: As competition intensifies to provide the best customer experience, service providers require real-time data to help provide the digital experience that today's consumers demand -- interactions that are more personalized, more contextual and more relevant. People want more social, mobile and online service and expect faster responses. This is where real-time or streaming analytics can help, by providing the most up-to-date information to the service provider, or even directly to the customer. With new technologies such as Spark and other streaming analytics this is now possible, enabling service providers to provide value at the main pain-point of customer experience –the direct customer touch-point.
IoT: The Internet of Things (IoT) is gaining momentum. Although many providers still struggle finding the optimal strategy around IoT, it's definitely becoming a reality. The nature of IoT -- involving data feeds from large numbers of sensors of various types, each of which needs to be monitored and analyzed -- presents an inherent need for automation and analytics for extracting value out of the internet-of-everything.
Skilled resources: While the promise of big data and analytics is real, we still see that lack of skilled resources is a major obstacle for successful big data analytics implementations. Data scientists -- particularly those with experience in telco data -- continue to be in strong demand, as shown by the fact that data scientist salaries are the fastest-growing category. IT operators today understand that analytics require intimate understanding of the telecom domain: While many data analysis algorithms are quite common, telecommunications data and business processes have unique characteristics. As a result, a successful analytics implementation requires expertise in both data analysis and the specific business processes of telecom service providers.
— Matt Roberts, marketing director for big data and strategic innovations, Amdocs The New IP