Philip Carnelley
Research Director, European Software Group
Big Data is not just about large data volumes. In some cases, Big Data also brings issues of velocity – the speed at which data is generated and must be dealt with: whether to store it: where to store it: and whether to modify before storage. This is a problem that that financial capital markets have had for a long time, with millions of trades being conducted around the world every hour that need to be processed and stored.
Part of the solution lies in a class of products called ‘streaming analytics’ solutions, which can perform real-time analytics on large volumes of incoming data. Until lately, this has remained a specialist niche. However, evolving requirements such as web clickstream analytics and the rapidly increasing number of intelligent, connected devices (the “Internet of Things”) mean that data velocity is an issue more and more companies will have to deal with.
IDC forecasts that by 2020 there will be over 20 billion intelligent devices (and about 5 billion smartphones), together generating over 50 Petabytes of data per day. Thus IDC believes the number of organizations wishing to use streaming analytics is set to increase rapidly. Potential use cases include improving customer experience, real-time pricing, fraud detection and predictive maintenance of machinery.
Current offerings broadly fall into 2 camps: established offerings from large enterprise software companies like IBM, Microsoft and Oracle; or, new and open source solutions linked to the Apache Hadoop project, with curious names like Flink, Samza, Spark and Storm, and with somewhat overlapping functionality.
Organizations who feel they need to pilot one of these solutions face a dilemma. They can license a potentially expensive, but battle-hardened solution with a support and services ecosystem, or they can test the water with an open source offering, new and fast evolving, with a potential lack of support, and of experienced staff.
A new option which could potentially overcome this dilemma comes from Software AG, which up to now has fallen squarely in the first camp. About 3 years ago it picked up one of the better-known names in this space, called Apama. This started life as a project at Cambridge University in the UK, was commercialized and eventually ended up in the Software AG portfolio. Software AG is now offering a “Community Edition” of Apama that is free to download, without registration. This free edition is also deployable in the cloud. One thing about Apama that may have particular appeal is that it can run on edge devices as well as the core – for example on a Raspberry Pi.
IDC believes that edge processing will turn out to be a key requirement for IoT solutions, where an important architectural approach to dealing with data volume and velocity is to process and refine data before it enters the central processing hub, by transmitting aggregate rather than raw data, removing potentially erroneous data, etc.
Software AG’s move is not pure altruism, of course. Like all freemium offerings, the company is hoping that organizations who try the free version (which has some restrictions) will upgrade to the full enterprise version. But we applaud any move that can help to expand the usability and applicability of this increasingly important technology for the wider market.
If you want to know more about this particular topic or are interested in European Software, please contact Philip Carnelley.