The explosion of sensors, devices, servers and applications over the past few years has led to an exponential increase in the volume of unstructured machine data. Furthermore, the rise of the machine has been happening in parallel with a thirst for real-time applications. Organizations are simply wanting to extract greater value from their real-time Big Data assets.

However, applications based on the traditional store-first, process-second data management architectures have been proven not to scale for real-time Big Data applications. Even Hadoop-based systems are unable to offer the combination of latency and throughput requirements for real-time applications in industries such as Telecoms, Internet of Things and cybersecurity. Stream processing is the answer.

Stream processing for Smart Services

So where is the level of performance of stream processing most useful? For example, the Internet of Things has raised the bar for real-time applications, but there are a range of industries that are benefiting from the ability to deliver real-time smart services based on stream processors.

• Smart Cities with real-time traffic analytics, congestion prediction and traveller travel time apps.
• Oil & Gas, with the real-time predictive analytics and automated actions to avert potential equipment failures.
• Security intelligence for fraud detection and cybersecurity alerts. For example, detecting Smart Grid consumption issues, SIM card misuse and SCADA intrusion attacks.
• Industrial automation, offering real-time analytics and predictive actions for patterns of manufacturing plant issues and quality problems.
• For Telecoms, real-time call rating, fraud detection and QoS monitoring from CDR and network performance data.
• Cloud infrastructure and web clickstream analysis for IT Operations.

The characteristics of a stream processor

The terms ‘stream processing’ and ‘streaming analytics’ are often applied to any solution that processes time-series data, regardless of the latency and throughput requirements. Hadoop-based solutions for example often claim ‘real-time Big Data’, yet the process of ETL, data storage and the continuous re-querying of complete data sets, introduces latency that can be measured in hours.

In order to deliver millisecond latency from data arriving a millions of records per second, at true stream processor must exhibit the following characteristics:

• In-memory, continuous processing of real-time time-series machine data streams over time and record-based windows.
• Highly optimized architecture for the combined requirements of near zero latency from Big Data volume.
• Scalability through efficient distributed execution over multiple cores (scale up) and multiple servers (scale-out).
• Designed for resilient 24×7 operations, supporting dynamic updates for adding new machine data sources, new enterprise integrations and applying application updates, all without downtime.
• Real-time integration with enterprise storage platforms for stream persistence, restreaming and joining of trend data and KPIs with arriving data streams.
• Real-time process automation through continuous integration with operational systems.

A stream processing platform supports all these key business requirements – low latency, high throughput, ease of platform management for operational systems, seamless enterprise integration and real-time operational process automation.

The mechanics of a stream processor

The high level architecture of a stream processor is closely related to the types of dataflow architectures exhibited by signal processing hardware systems, where arriving data streams are clocked through a network of processing nodes, and where each node performs some action or transformation on the data as it flows through. Software stream processing behaves in exactly the same way, except instead of logic gates, each node in the system is a continuously executing and independent query that performs operations on data streams such as filtering, aggregation and analytics.

The conceptual architecture for a stream processor is shown below, essentially a network of concurrently executing continuous queries that operate on the data streams as they flow through.

• Each node is an independent continuous query, that is, a query that never ends, processing data as they arrive.
• All nodes execute concurrently, subscribing or consuming one or more input data streams, and generating one or more output streams. Essentially a living, organic network of communicating and chained data management operators.
• Underneath, the stream processing engine is responsible for the real-time scheduling, query optimization, and the runtime execution management of the overall system timing and the movement (or clocking) of data through the system.

Stream Processing Dataflow Architecture

Achieving real-time performance

Stream processors are capable of generating results in a few milliseconds from data arriving at millions of records per second. Compare this to Hadoop and RDBMS platforms which can process data at a few thousand records per second at most, and with latency measured in hours. Real-time performance is achieved through a range of architectural features, including:

• Continuous ETL combined with in-memory processing without having to first store the data. This eliminates the two main performance bottlenecks of traditional storage-based solutions – delays due to ETL / data load processes and physical storage updates.
• Continuous queries execute forever over the arriving data, removing the performance impact of storage-based platforms which must continuously re-execute queries as new data arrive.
• Data streams are processed over time or row-based windows, where data need only be retained for the duration of each window. This enables very efficient management of system memory – millions of records per second can be processed in a relatively small amount of main memory.
• The results of each query are updated incrementally as every new record arrives – compare this with a storage-based solution where the query must execute across all data for each repeated re-execution of the query.
• Dataflow stream processing product architecture is also optimized for low latency with high throughput, making efficient use of multiple cores and scaling across multiple servers and data centers.

SQL for stream processing

SQL is the now the standard for Big Data management. Just as Big Data vendors such as Cloudera utilize SQL for processing unstructured stored data in Hadoop, SQL is equally applicable to the Big Data stream processing of machine data. SQL is declarative, which means applications can be written in just a few lines of high level queries, it is reusable and portable across data management platforms, and most importantly, declarative means that queries can be automatically optimized and distributed by the underlying streaming data management platform.

Standard SQL (as defined in the latest SQL:2011 standard) also contains all the facilities to perform stream processing. Mapping SQL:2011 onto the conceptual dataflow architecture illustrated above for a stream processor:

• Each processing node in the dataflow network is an autonomous but concurrently executing continuous SQL query. For example, the ability to SELECT FROM a stream, and use operators such as WHERE (filtering), GROUP BY (aggregation), PARTITION BY, UNION and JOIN over a time-based WINDOW.
• SQL Views provide the mechanism for communication between the independent processing nodes. In static database SQL, a View is a virtual table that holds the result set of a SQL query. In streaming SQL, a View is a virtual stream that holds the result set of a streaming query, essentially a buffer to which any other processing node (SQL query) can subscribe. Views (and buffer management) are the backbone of a stream processing system, enabling queries to be chained together into processing pipelines.
• Data is clocked through the system by the underlying runtime execution environment. The movement of data is calculated from the timestamps on each data record, and the length (duration) of all time-based windows present in the application.
• Streaming SQL provides two mechanisms for determining how data flows between processing nodes. The most common approach is to chain independent queries together using Views. In the example, the first query reads the ny_orders stream and outputs orders from New York as they arrive, and a concurrently executing query in the stream processing pipeline is reading from the ny_orders stream and filters out all orders with a value less than 10000.

/* Processing Node 1 */
CREATE VIEW ny_orders AS
SELECT STREAM * FROM orders
WHERE city = 'New York';

/* Chained processing Node 2 */
CREATE VIEW high_value_ny_orders AS
SELECT STREAM * FROM ny_orders
WHERE value > 10000;

• There are also occasions when a developer may require greater control over the movement of data through the application and the execution of data processing pipelines. A PUMP can be used to SELECT data from any stream (a View) and to feed data explicitly into an output stream, to which any other query can subscribe. For example:

CREATE PUMP "WeatherDataPump" STARTED AS
INSERT INTO "WeatherStream" 
( "City", "Temperature", "Sensor_ID" )
SELECT STREAM
  "City", "Temperature", "Sensor_ID" 
FROM "RawInputDataStream";

In summary, SQL for stream processing brings all the benefits of SQL as the leading data management language, coupled with the power of the underlying SQLstream s-Server stream processor. Streaming SQL supports continuous queries, with Views providing the communication buffers between autonomous nodes in the application architecture, with time being the controlling factor – data timestamps and processing time windows. The result is an enterprise-class stream processor capable of near-zero latency from millions of records per second, which means real-time enterprise integration and the ability to automate operational processes directly.