Overview

In the preceding chapters we described the stream processing programming model and the system architecture that supports it. In this chapter we will describe the principles of stream processing application design, and provide patterns to illustrate effective and efficient ways in which these principles can be put into practice.

We look at look at functional design patterns [1] and principles that describe effective ways to accomplish stream processing tasks, as well as non-functional ones [1] that address cross-cutting concerns such as scalability, performance, and fault tolerance.

This chapter is organized as follows. Section 9.2 describes functional design patterns and principles, covering the topics of edge adaptation, flow manipulation, and dynamic adaptation. Section 9.3 describes non-functional design patterns and principles, covering the topics of application composition, parallelization, optimization, and fault tolerance.

Functional design patterns and principles

We start by examining functional design patterns and principles, covering edge adaptation, flow manipulation, and dynamic adaptation.

9.2.1 Edge adaptation

SPAs consume data from external sources available in various different formats and accessible by employing different protocols. Similarly, results produced by streaming applications are often consumed by external systems in various formats and through different protocols. We term the process of interacting with external systems to receive and send data as edge adaptation.