In order to compile flow statistics, each router maintains a table of records indexed by flow key, e.g. 5-tuple of the flow. A flow is said to be active at a given time if there exists a record for its key. When a packet arrives at the router, the router determines if a flow is active for that key. If not, it instantiates a new record for that key. The statistics for the flow are updated for the packet, typically including counters for packets and bytes and arrival times of the first and most recent packet of the flow. Due to the fact that the router does not have knowledge of application-level flow structure, it must terminate the flow according to some criteria. The most commonly used criteria are the following: (i) inter-packet timeout, e.g. the time since the last packet observed for the flow exceeds some threshold; (ii) protocol syntax, e.g. observation of a FIN or RST packet of the TCP flow; (iii) aging, e.g. flows are terminated after a given elapsed time since the arrival of the first packet of the flow; (iv) memory management, e.g. flows might be terminated at any point in time to release memory. When a flow is terminated, its statistics are flushed for export and the associated memory is released for use by new flows.

Before embarking on the exploration of techniques to assist operators in the management and design of IP networks, this chapter lays the foundation of the terms and concepts that will be used in the rest of the book. We describe the Internet architecture and the elements and protocols guiding its behavior. We then outline issues associated with the design, management, optimization and security of such a complex infrastructure, topics that will be the focal points in the following chapters.

What is the Internet?

The Internet is a diverse collection of independent networks, interlinked to provide its users with the appearance of a single, uniform network. Two factors shield the user from the complex realities that lie behind the illusion of seamlessness: (i) the use of a standard set of protocols to communicate across networks and (ii) the efforts of the companies and organizations that operate the Internet's different networks to keep its elements interconnected.

The networks that comprise the Internet share a common architecture (how the components of the networks inter-relate) and software protocols (standards governing the exchange of data), which enable communication within and among the constituent networks. The nature of these two abstract elements – architecture and protocols – is driven by the set of fundamental design principles adopted by the early builders of the Internet. It is important to distinguish between the public Internet and the Internet's core technology (standard protocols and routers), which are frequently called “IP technology.”

Number theory and algebra play an increasingly significant role in computing and communications, as evidenced by the striking applications of these subjects to such fields as cryptography and coding theory. My goal in writing this book was to provide an introduction to number theory and algebra, with an emphasis on algorithms and applications, that would be accessible to a broad audience. In particular, I wanted to write a book that would be appropriate for typical students in computer science or mathematics who have some amount of general mathematical experience, but without presuming too much specific mathematical knowledge.

Prerequisites. The mathematical prerequisites are minimal: no particular mathematical concepts beyond what is taught in a typical undergraduate calculus sequence are assumed.

The computer science prerequisites are also quite minimal: it is assumed that the reader is proficient in programming, and has had some exposure to the analysis of algorithms, essentially at the level of an undergraduate course on algorithms and data structures.

Even though it is mathematically quite self contained, the text does presuppose that the reader is comfortable with mathematical formalism and also has some experience in reading and writing mathematical proofs. Readers may have gained such experience in computer science courses such as algorithms, automata or complexity theory, or some type of “discrete mathematics for computer science students” course. They also may have gained such experience in undergraduate mathematics courses, such as abstract or linear algebra.

Classifying traffic flows according to the application that generates them is an important task for (a) effective network planning and design and (b) monitoring the trends of the applications in operational networks. However, an accurate method that can reliably identify the generating application of a flow is still to be developed. In this chapter and the next, we look into the problem of traffic classification; the ultimate goal is to provide network operators with algorithms that will provide a meaningful classification per application, and, if this is infeasible, with useful insight into the traffic behavior. The latter may facilitate the detection of abnormalities in the traffic, malicious behavior or the identification of novel applications.

State of the art and context

Currently, application classification practices rely to a large extent on the use of transport-layer port numbers. While this practice may have been effective in the early days of the Internet, port numbers currently provide limited information. Often, applications and users are not cooperative and, intentionally or not, use inconsistent ports. Thus, “reliable” traffic classification requires packet-payload examination, which is scarcely an option due to: (a) hardware and complexity limitations, (b) privacy and legal issues, (c) payload encryption by the applications.

Taking into account empirical application trends and the increasing use of encryption, we conjecture that traffic classifiers of the future will need to classify traffic “in the dark.”

This chapter introduces the notion of an abelian group. This is an abstraction that models many different algebraic structures, and yet despite the level of generality, a number of very useful results can be easily obtained.

Definitions, basic properties, and examples

Definition 6.1. An abelian group is a set G together with a binary operation * on G such that:

1. (i) for all a, b, c ∈ G, a * (b * c) = (a * b) c (i.e., * is associative);

2. (ii) there exists e ∈ G (called the identity element) such that for all a ∈ G, a * e = a = e * a;

3. (iii) for all a ∈ G there exists a′ ∈ G (called the inverse of a) such that a * a′ = e = a′ * a;

4. (iv) for all a, b ∈ G, a * b = b * a (i.e., * is commutative).

While there is a more general notion of a group, which may be defined simply by dropping property (iv) in Definition 6.1, we shall not need this notion in this text. The restriction to abelian groups helps to simplify the discussion significantly. Because we will only be dealing with abelian groups, we may occasionally simply say “group” instead of “abelian group.”

In this chapter, we review standard asymptotic notation, introduce the formal computational model that we shall use throughout the rest of the text, and discuss basic algorithms for computing with large integers.

Asymptotic notation

We review some standard notation for relating the rate of growth of functions. This notation will be useful in discussing the running times of algorithms, and in a number of other contexts as well.

Let f and g be real-valued functions. We shall assume that each is defined on the set of non-negative integers, or, alternatively, that each is defined on the set of non-negative reals. Actually, as we are only concerned about the behavior of f(x) and g(x) as x → ∞, we only require that f(x) and g(x) are defined for all sufficiently large x (the phrase “for all sufficiently large x” means “for some x0 and all x ≥ x0”). We further assume that g is eventually positive, meaning that g(x) > 0 for all sufficiently large x. Then

• f = O(g) means that |f(x)| ≤ cg(x) for some positive constant c and all sufficiently large x (read, “f is big-O of g”),

• f = Ω(g) means that f(x) ≥ cg(x) for some positive constant c and all sufficiently large x (read, “f is big-Omega of g”),

• f = Θ(g) means that cg(x) ≤ f(x) ≤ dg(x) for some positive constants c and d and all sufficiently large x (read, “f is big-Theta of g”),

• […]

It is sometimes useful to endow our algorithms with the ability to generate random numbers. In fact, we have already seen two examples of how such probabilistic algorithms may be useful:

• at the end of §3.4, we saw how a probabilistic algorithm might be used to build a simple and efficient primality test; however, this test might incorrectly assert that a composite number is prime; in the next chapter, we will see how a small modification to this algorithm will ensure that the probability of making such a mistake is extremely small;

• in §4.5, we saw how a probabilistic algorithm could be used to make Fermat's two squares theorem constructive; in this case, the use of randomization never leads to incorrect results, but the running time of the algorithm was only bounded “in expectation.”

We will see a number of other probabilistic algorithms in this text, and it is high time that we place them on a firm theoretical foundation. To simplify matters, we only consider algorithms that generate random bits. Where such random bits actually come from will not be of great concern to us here. In a practical implementation, one would use a pseudo-random bit generator, which should produce bits that “for all practical purposes” are “as good as random.” While there is a well-developed theory of pseudo-random bit generation (some of which builds on the ideas in §8.9), we will not delve into this here.

As networks continue to grow rapidly in size and complexity, it has become increasingly clear that their evolution is closely tied to a detailed understanding of network traffic. Large IP networks are designed with the goal of providing high availability and low delay/loss while keeping operational complexity and cost low. Meeting these goals is a highly challenging task and can only be achieved through a detailed knowledge of the network and its dynamics.

No matter how surprising this may seem, IP network management today is primarily reactive in nature and relies on trial and error when problems arise. Network operators have limited visibility into the traffic that flows on top of their network, the operational state of the network elements and the behavior of the protocols responsible for the routing of traffic and the reliable transmission of packets from end to end. Furthermore, design and planning decisions only partially rely on actual usage patterns. There are a few reasons behind such a phenomenon.

First, the designers of IP networks have traditionally attached less importance to network monitoring and resource accounting than to issues such as distributed management, robustness to failures and support for diverse services and protocols. Thus, IP network elements (routers and end hosts) have not been designed to retain detailed information about the traffic flowing through them, and IP protocols typically do not provide detailed information about the state of the underlying network.

The traffic matrix (TM) of a telecommunications network measures the total amount of traffic entering the network from any ingress point and destined to any egress point. The knowledge captured in the TM constitutes an essential input for optimal network design, traffic engineering and capacity planning. Despite its importance, however, the TM for an IP network is a quantity that has remained elusive to capture via direct measurement. The reasons for this are multiple. First, the computation of the TM requires the collection of flow statistics across the entire edge of the network, which may not be supported by all the network elements. Second, these statistics need to be shipped to a central location for appropriate processing. The shipping costs, coupled with the frequency with which such data would be shipped, translate to communications overhead, while the processing cost at the central location translates to computational overhead. Lastly, given the granularity at which flow statistics are collected with today's technology on a router, the construction of the TM requires explicit information on the state of the routing protocols, as well as the configuration of the network elements. The storage overhead at the central location thus includes routing state and configuration information. It has been widely believed that these overheads would be so significant as to render computation of backbone TMs, through measurement alone, not viable using today's flow monitors.

The convergence of traditional network services to a common IP infrastructure has resulted in a major paradigm shift for many service providers. Service providers are looking for profitable ways to deliver value-added, bundled, or personalized IP services to a greater number of broadband users. As cable operators and Digital Subscriber Line (DSL) providers capitalize on IP networks, they need to create higher-margin, higher-value premium services, such as interactive gaming, Video-on-Demand (VoD), Voice-over-IP (VoIP) and broadband TV (IPTV). The missing element of the current strategy is service differentiation, i.e. the ability to understand at a granular level how subscribers are using the network, identify what applications or services are being consumed, and then intelligently apply network resources to applications and attract subscribers that promise the highest return on investment. Operators need to manage and control subscriber traffic. This can be accomplished by introducing more intelligence into the network infrastructure, which enhances the transport network with application and subscriber awareness. Such unique visibility into the types of bits carried allows the network to identify, classify, guarantee performance and charge for services based on unique application and subscriber criteria. Instead of underwriting the expenses associated with random and unconstrained data capacity, deployment and consumption, this new wave of network intelligence allows operators to consider Quality-of-Service (QoS) constraints while enabling new possibilities for broadband service creation and new revenue-sharing opportunities. The same is true with third-party service providers, who may, in fact, be riding an operator's network undetected.