Tuesday, February 11, 2014

What's Next for Big Data?

Big Value.  Broad Usefulness.

The time for putting all data to work is now.  The cost of collecting and using all data is plummeting. Meanwhile, the tools to access and work with data are becoming simpler. New, big, multi-structured data sets combined with the significant architectural advances of cloud computing has created advantages accessible to any organization.  Most profoundly, the next few years will usher in a new class of application, one that uses all available data and is built on the superior economics of cloud-based computing. This class of applications will help make big data small and will be purpose-built to solve an enormous array of real business problems. This is what’s next for Big Data.

Cost reduction and improving economics are key ingredients to pervasive use of data.  Open source projects are fueling the rise in usefulness of Big Data and creating disruption on a grand scale.  Apache Hadoop and Cassandra are some of the best known examples, but represent just tips of the iceberg. A recent IDG survey showed that 70% of enterprises already have or are planning to implement a Big Data project and about 30% of respondents reported using an open source big data framework, such as Hadoop. Our own Big Data survey described a doubling of Big Data projects (versus our same survey last year) as well as a marked reduction (47% decrease) in the confusion about the definition and usefulness of Big Data. 

I’ve written previously about the amazing advancements in sensor technologies, storage resources and compute cycles – which create a perfect, low cost foundation for developing and delivering new applications that harness huge data sets to new effect. Two of my favorite examples of this new generation of applications in action are Clockwork and TrackX (formerly Fluensee).

Clockwork has created a modern, cloud-based Enterprise Asset Management system that is transforming the management of capital-intensive assets for any size of organization.  Clockwork recently reported that it has saved its customers more than $3 billion in the past year by combining sensors and sophisticated software to fundamentally improve the maintenance, repair and operations of their assets.

TrackX creates software and technology to effectively track and manage physical assets using RFID, barcode, GPS and sensor technologies with its rapidly-deployed, cloud-based software.  A key ingredient in TrackX solutions is analytics, because its mission is to provide insight on all asset-related data. TrackX takes all asset attributes and measurements and creates an analytical picture that provides customers with actionable intelligence to better optimize time, labor and asset productivity.  This insight is cleverly designed-in its application software, helping anyone in the organization to become more analytically-astute.

So, What’s Next for Big Data?

The two large growth areas for analytics are: 
1. making data scientists far more capable by giving them more useful and cost-effective tools than ever before, including Jaspersoft’s recent innovations enabling rapid blending of big and small data through data virtualization;
2. making everyone else in the organization more capably analytical by providing them with just the right amount of analytics at the right place and the right time (no more, no less) within the applications and processes they use every day. As a result, they can make better data-driven decisions than ever before.

The second growth area above will have the broadest, most transformational effect on an organization because it can involve everyone, regardless of level or title, making decisions on a daily basis. IDG’s recent survey found that 42% of the respondents plan to invest in these applications in 2014, making this one of the top five areas for investment. 

This investment level should provide us with great hope, because, although we’re in the very first phase of a revolution in data and analytics, the evidence is now clear.  Dramatically lower costs for capturing and using all the data, combined with a new breed of cloud-based analytical applications, will power huge gains in value and usefulness for data in any organization.