This chapter serves two purposes: review of previous work and preview of our new results on the parallel routing problem. Our general approach to the problem is also outlined. The routing schemes and their analysis will appear in the next chapter.

Introduction

Perhaps the most important issue in parallel computers is interprocessor communication. Since processors are linked together by a relatively sparse network due to cost considerations, packets have to traverse many links and even be delayed by other packets due to conflicts or full buffers before reaching their destinations. Communication time, therefore, can easily dominate the total execution time, making designing fast communication schemes a primary concern. It is also preferable that the buffer size be a constant independent of the size of the network for the sake of scalability. Finally, as more components mean more faults [326], other things being equal, the issue of fault tolerance without loss of efficiency should be dealt with together.

We review previous work and preview our approach and results in this chapter. In the next chapter, it will be shown how the above-mentioned problems can be tackled simultaneously using information dispersal.

Fast, fault-tolerant communication schemes using constant size buffers on the hypercube and the de Bruijn networks are presented. All our schemes run with probability of successful routing 1 – N−Θ(logN) where N denotes the number of processors. A summary of this chapter's results can be found in Section 4.4.

Parallel Communication Scheme

The notion of symmetric parallel communication scheme (SPCS) captures the essence, and facilitates the analysis, of our parallel routing algorithms.

Definition 5.1In anepoched communication scheme (ECS), packets are sent inepochs, numbered from 0. In any epoch, packets that demand transmission to neighboring nodes in that epoch, as determined by the routing algorithm, are sent along the links as requested. A symmetric parallel communication scheme (SPCS) is an ECS that satisfies the following conditions.

1. Each node initially has h packets.

2. All packets are routed independently by the routing algorithm; that is, in any epoch, a packet crosses an out-going link independently with equal probability.

3. The expected number of packets at each node is h at the end of each epoch.

Such a scheme is called an h-SPCS.

Comment 5.2 Condition 3 implies that no packets can be lost. In all our schemes, however, pieces (that is, packets in the definition of ECS) can be lost due to buffer overflow. This is not a serious problem since we can analyze a scheme as if pieces over the capacity of the buffer were not lost.

We demonstrate in this chapter that, if fsra is used, a constant fraction of the wires in the hypercube network can be disabled simultaneously without disrupting the ongoing computation or degrading the routing performance. This general result can lead to efficient on-line maintenance procedures. This seems to be the first time that the important issue of on-line maintenance is addressed analytically.

Introduction

The fact that hardware deteriorates and the demand that machine be more available to the user make the property of on-line maintenance without performance penalty a desirable design goal. For example, in the Tandem/16 computer system [173], modular design allows some components to be replaced on-line. Periodic maintenance of the hardware is also key to ensuring consistent system performance; without it, one cannot safely say a particular component retains roughly the same failure rate at different times.

In this chapter we address the issue of on-line wire maintenance on the hypercube network with FSRA as the routing algorithm. It is shown that the set of edges can be partitioned into a constant number, 352, of disjoint edge sets of roughly equal sizes such that the probability of unsuccessful routing is exponentially small if any edge set in the proposed partition is disabled (Theorem 8.3). That implies little performance penalty as re-routing is extremely unlikely. The partition is also easily and locally computable.

We survey interconnection networks that link the processors in a parallel computer. Important terms and design issues pertaining to networks are also briefly discussed. This chapter is intended to be concise and sufficiently complete.

Introduction

The interconnection network in a multiprocessor specifies how the processors are tied together. The processors then communicate by sending information through the network. Since the interprocessor delay easily dominates the execution time [56] except where interprocessor communications are rare or only nearly processors exchange information, the choice of the network makes the difference between an efficient system and an inefficient one; clearly, a network where information has to traverse, say, 100 links on the average is less efficient than the one where only ten suffice, other things being equal. Besides the efficiency issue, the interconnection network also sets a limit to the number of faults a parallel computing system can sustain. For example, since a disconnected network that isolates some processors from the others makes joint computation impossible, the network should be able to withstand many faulty links.

This chapter surveys interconnection networks and related issues. In Section 3.2 some of the basic terms for networks are reviewed. A simple and useful graph- theoretical abstraction of networks is introduced in Section 3.3. Several popular networks are then briefly surveyed in Section 3.4, followed by Section 3.5, where some key issues in the choice of networks are summarized.

In the previous chapter we introduced the notion of a sequential process (agreeing that where no confusion would arise, the term process would be sufficient) in an informal way. Now we need to formalise this concept, and to provide some concrete examples of the representation of a process which will enable us to clarify the basic ideas behind systems of concurrent processes. We begin, however, by considering some of the issues concerning the nature of concurrent activity, and describing some of the notations which have been proposed whereby concurrency may be introduced into a program.

Specification of Concurrent Activity

There are a variety of ways of talking about concurrency, some of which have found their way into programming languages in various guises, and some of which are merely notational mechanisms for describing parallel activity.

When discussing the ways in which parallel activity can be specified within a programming language or system, it is useful to consider two orthogonal features of any proposed method. The first is the specification of where in a program a separate process may be started, and where, if at all, it terminates. The specification of where a new process may begin and end provides some synchronisation points within the concurrent program. We assume that it is possible to specify that two processes may execute in parallel, and it is at that point that the two processes are assumed to be synchronised.

Concurrency has been with us for a long time. The idea of different tasks being carried out at the same time, in order to achieve a particular end result more quickly, has been with us from time immemorial. Sometimes the tasks may be regarded as independent of one another. Two gardeners, one planting potatoes and the other cutting the lawn (provided the potatoes are not to be planted on the lawn!) will complete the two tasks in the time it takes to do just one of them. Sometimes the tasks are dependent upon each other, as in a team activity such as is found in a well-run hospital operating theatre. Here, each member of the team has to co-operate fully with the other members, but each member has his/her own well-defined task to carry out.

Concurrency has also been present in computers for almost as long as computers themselves have existed. Early on in the development of the electronic digital computer it was realised that there was an enormous discrepancy in the speeds of operation of electro-mechanical peripheral devices and the purely electronic central processing unit. The logical resolution of this discrepancy was to allow the peripheral device to operate independently of the central processor, making it feasible for the processor to make productive use of the time that the peripheral device is operating, rather than have to wait until a slow operation has been completed.

As long as all of the concurrent processes are proceeding completely independently of each other, we would expect them all to continue at their own speed until they terminate (if they do). If this does not happen, that is if the results of a process are affected by the presence or absence of another supposedly independent process, then we have to investigate the underlying mechanism to find the reason for this problem. For the purposes of the discussion of the concurrent processes themselves, they will have an effect on one another only if they are required to communicate with each other.

We have already seen one example of inter-process communication as manifested in one of the process creation and deletion methods. This entailed the parent process calling the fork operation to initiate the child process, and the subsequent join operation (if there is one) as a way of resynchronising the parent and child. These two operations are very restrictive, allowing as they do the parent and child to synchronise with each other only when the child begins and ends its execution. What is required is a more general mechanism (or mechanisms) by which two processes may communicate with each other, not just at the beginning and end of the lives of the processes. A discussion of several possible methods is the subject of this chapter.

For a number of years, Concurrent Programming was considered only to arise as a component in the study of Operating Systems. To some extent this attitude is understandable, in that matters concerning the scheduling of concurrent activity on a limited number (often one!) of processing units, and the detection/prevention of deadlock, are still regarded as the responsibility of an operating system. Historically, it is also the case that concurrent activity within a computing system was provided exclusively by the operating system for its own purposes, such as supporting multiple concurrent users.

It has become clear in recent years, however, that concurrent programming is a subject of study in its own right, primarily because it is now recognised that the use of parallelism can be beneficial as much to the applications programmer as it is to the systems programmer. It is also now clear that the principles governing the design, and the techniques employed in the implementation of concurrent programs belong more to the study of programming than to the management of the resources of a computer system.

This book is based on a course of lectures given over a number of years initially to third, and more recently to second year undergraduates in Computing Science. True to the origins of the subject, the course began as the first part of a course in operating systems, but was later separated off and has now become part of a course in advanced programming techniques.

We introduce here a simple concurrency kernel, showing how it may be used to provide pseudo-concurrency on a single processor, and also in the chapter we shall show how some of the concurrency constructs introduced in chapters 4 and 5 may be implemented.

We assume that the machine on which the kernel is to be implemented consists of a single processor of the conventional type with a single monolithic address space. A simple Motorola M68000 based system would be an appropriate type of system on which to run the kernel. To make the kernel more useful, it would be possible to consider hardware which provides a virtual address translation capability, in which case the various processes could be provided with distinct address spaces in which to operate. This, however, not only makes the kernel more complicated, but also makes sharing of memory more difficult. For the purposes of the simple kernel which we are considering here, we shall assume a single (shared) address space.

The implementation language will be Pascal, and for simplicity we shall assume that a hypothetical Pascal machine has been implemented on top of the basic hardware. Apart from the ability to run Pascal programs, we shall also require some additional features to support the concurrency kernel to be constructed. It is clear that much of the underlying support, both for Pascal itself and for the concurrency kernel, will have to be provided through a small amount of assembly code.

Before embarking upon a discussion of the concurrency facilities provided by various programming languages, it is necessary to issue an apology and a warning. The set of languages discussed in this chapter is intended to be a representative selection of languages available at the present time, which between them illustrate a number of different approaches to concurrency. It is hoped that no reader will feel offended at not finding his favourite concurrent language in the discussion. Examples are chosen to demonstrate a variety of treatments of controlled access to shared data, usually in the form of a shared data structure with limited access to the operations on the data structure (i.e. variations on the ‘monitor’ theme), and also languages supporting some form of message passing or mailbox handling are considered.

It should be clear that beneath each of the mechanisms to be found in any of the languages discussed in this chapter, some lower level code (or hardware) must be supplied to provide, if not the reality, then the illusion of concurrency. Beneath any successful concurrent language there is a system which manages the concurrency, schedules processes onto processors, provides the fundamental operations to implement process creation and deletion, and to provide communication facilities.

When any new language is designed, the designer attempts to correct many of the inadequacies (as he sees them) of one or more existing languages.

In the previous chapter we considered the basic problem of communication between processes, and reduced it to the problem of preventing two processes from accessing the same shared resources simultaneously. It should be clear that from such mechanisms it is possible to construct a large number of different methods for communication. It is also clear that the relationship between the lowlevel mechanisms described in the previous chapter and the higherlevel constructs is similar to the relationship between a low-level assembly language program and the corresponding program in a highlevel language; the first provides a basic mechanism, which clearly is sufficient, if inconvenient, to implement all the necessary synchronisation. The second expresses the solution to the problem in a much more “problem oriented” fashion.

In this chapter we shall consider a number of high-level constructs which have been proposed which allow a more structured approach to many inter-process communication/synchronisation problems. In the same way that the use of high-level languages allows the programmer to express his algorithms in a more natural way, and hence will make the inclusion of simple logical errors less likely, so the use of high-level synchronisation constructs in concurrent programs will also tend to reduce the incidence of elementary concurrent programming errors.

To take a small example, consider program 4.1, which uses binary semaphores to provide an implementation of the critical section problem (c.f. program 3.12).