Overview

In this chapter, we examine visualization and debugging as well as the relationship of these services and the infrastructure provided by a SPS. Visualization and debugging tools help developers and analysts to inspect and to understand the current state of an application and the data flow between its components, thus mitigating the cognitive and software engineering challenges associated with developing, optimizing, deploying, and managing SPAs, particularly the large-scale distributed ones.

On the one hand, visualization techniques are important at development time, where the ability to picture the application layout and its live data flows can aid in refining its design.

On the other hand, debugging techniques and tools, which are sometimes integrated with visualization tools, are important because the continuous and critical nature of some SPAs requires the ability to effectively diagnose and address problems before and after they reach a production stage, where disruptions can have serious consequences.

This chapter starts with a discussion of software visualization techniques for SPAs (Section 6.2), including the mechanisms to produce effective visual representations of an application's data flow graph topology, its performance metrics, and its live status.

Debugging is intimately related to visualization. Hence, the second half of this chapter focuses on the different types of debugging tasks used in stream processing (Section 6.3).

Visualization

Comprehensive visualization infrastructure is a fundamental tool to support the development, understanding, debugging, and optimization of SPAs.

Overview

The world has become information-driven, with many facets of business and government being fully automated and their systems being instrumented and interconnected. On the one hand, private and public organizations have been investing heavily in deploying sensors and infrastructure to collect readings from these sensors, on a continuous basis. On the other hand, the need to monitor and act on information from the sensors in the field to drive rapid decisions, to tweak production processes, to tweak logistics choices, and, ultimately, to better monitor and manage physical systems, is now fundamental to many organizations.

The emergence of stream processing was driven by increasingly stringent data management, processing, and analysis needs from business and scientific applications, coupled with the confluence of two major technological and scientific shifts: first, the advances in software and hardware technologies for database, data management, and distributed systems, and, second, the advances in supporting techniques in signal processing, statistics, data mining, and in optimization theory.

In Section 1.2, we will look more deeply into the data processing requirements that led to the design of stream processing systems and applications. In Section 1.3, we will trace the roots of the theoretical and engineering underpinnings that enabled these applications, as well as the middleware supporting them. While providing this historical perspective, we will illustrate how stream processing uses and extends these fundamental building blocks.

Stream processing has emerged from the confluence of advances in data management, parallel and distributed computing, signal processing, statistics, data mining, and optimization theory.

Stream processing is an intuitive computing paradigm where data is consumed as it is generated, computation is performed at wire speed, and results are immediately produced, all within a continuous cycle. The rise of this computing paradigm was the result of the need to support a new class of applications. These analytic-centric applications are focused on extracting intelligence from large quantities of continuously generated data, to provide faster, online, and real-time results. These applications span multiple domains, including environment and infrastructure monitoring, manufacturing, finance, healthcare, telecommunications, physical and cyber security, and, finally, large-scale scientific and experimental research.

In this book, we have discussed the emergence of stream processing and the three pillars that sustain it: the programming paradigm, the software infrastructure, and the analytics, which together enable the development of large-scale high-performance SPAs.

In this chapter, we start with a quick recap of the book (Section 13.1), then look at the existing challenges and open problems in stream processing (Section 13.2), and end with a discussion on how this technology may evolve in the coming years (Section 13.3).

Book summary

In the two introductory chapters (Chapters 1 and 2) of the book, we traced the origins of stream processing as well as provided an overview of its technical fundamentals, and a description of the technological landscape in the area of continuous data processing.

Stream processing is a paradigm built to support natural and intuitive ways of designing, expressing, and implementing continuous online high-speed data processing. If we look at systems that manage the critical infrastructure that makes modern life possible, each of their components must be able to sense what is happening externally, by processing continuous inputs, and to respond by continuously producing results and actions. This pattern is very intuitive and is not very dissimilar from how the human body works, constantly sensing and responding to external stimuli. For this reason, stream processing is a natural way to analyze information as well as to interconnect the different components that make such processing fast and scalable.

We wrote this book as a comprehensive reference for students, developers, and researchers to allow them to design and implement their applications using the stream processing paradigm. In many domains, employing this paradigm yields results that better match the needs of certain types of applications, primarily along three dimensions.

First, many applications naturally adhere to a sense-and-respond pattern. Hence, engineering these types of applications is simpler, as both the programming model and the supporting stream processing systems provide abstractions and constructs that match the needs associated with continuously sensing, processing, predicting, and reacting.

Second, the stream processing paradigm naturally supports extensibility and scalability requirements. This allows stream processing applications to better cope with high data volumes, handle fluctuations in the workload and resources, and also readjust to time-varying data and processing characteristics.

Overview

In this chapter, we switch the focus from a conceptual description of the SPS architecture to the specifics of one such system, InfoSphere Streams. The concepts, entities, services, and interfaces described in Chapter 7 will now be made concrete by studying the engineering foundations of Streams. We start with a brief recount of Streams' research roots and historical context in Section 8.2. In Section 8.3 we discuss user interaction with Streams’ application runtime environment.

We then describe the principal components of Streams in Section 8.4. We focus on how these components interact to form a cohesive runtime environment to support users and applications sharing a Streams instance. In the second half of this chapter we focus on services (Section 8.5), delving into the internals of Streams' architectural components, providing a broader discussion of their service APIs and their steady state runtime life cycles. We discuss Streams with a top-down description of its application runtime environment, starting with the overall architecture, followed by the specific services provided by the environment.

Finally, we discuss the facets of the architecture that are devoted to supporting application development and tuning.

Background and history

InfoSphere Streams can trace its roots to the System S middleware, which was developed between 2003 and 2009 at IBM Research [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]. The architectural foundations and programming language model in Streams are based on counterparts in System S.

Tractability is the study of computational tasks with the goal of identifying which problem classes are tractable or, in other words, efficiently solvable. The class of tractable problems is traditionally assumed to be solvable in polynomial time by a deterministic Turing machine and is denoted by P.The class contains many natural tasks such as sorting a set of numbers, linear programming (the decision version), determining if a number is prime, and finding a maximum weight matching. Many interesting problems, however, lie in another class that generalizes P and is known as NP: the class of languages decidable in polynomial time on a non-deterministic Turing machine. We trivially have that P isasubset of NP (many researchers also believe that it is a strict subset). It is believed that many problems in the class NP are, in the worst case, intractable and do not admit efficient inference. Problems such as maximum stable set, the traveling salesman problem and graph coloring are known to be NP-hard (at least as hard as the hardest problems in NP). It is, therefore, widely suspected that there are no polynomial-time algorithms for NP-hard problems.