There has recently been a marked increase in research activity regarding the structural and function of the brain. Much of this has been generated by the more general advances in biology, particularly at the molecular and microscopic levels, but it is probably fair to say that the stimulation has been due at least as much to recent advances in computer simulation. To accept this view does not mean that one is equating the brain to an electronic computer, of course; far from it, those involved in brain research have long since come to appreciate the considerable differences between the cerebral cortex and traditional computational hardware. But the computer is nevertheless a useful device in brain science, because it permits one to simulate processes which are difficult to monitor experimentally, and perhaps impossible to handle by theoretical analysis.

The articles in this book are written records of talks presented at a meeting held at the Gentofte Hotel, Copenhagen, during the three days August 20–22, 1986. They have been arranged in an order that places more general aspects of the subject towards the beginning, preceding those applications to specific facets of brain science which make up the balance of the book. The final chapters are devoted to a number of ramifications, including the design of experiments, communication and control.

The meeting could not have been held without the financial support generously donated by the Augustinus Foundation, the Carlsberg Foundation, the Mads Clausen (Danfoss) Foundation, the Danish Natural Science Research Council, the Hartmann Foundation, IBM, the Otto Mønsted Foundation, NORDITA, the NOVO Foundation, and SAS.

Introduction

A prominent feature of the brain is the apparent diversity of its structure: the distribution of neurons and the way in which their dendrites and axon fibers differ in various brain centers. The pattern of inputs and outputs of each neuron in the brain most probably differs from that of any other neuron in the system, and this possibility clearly imposes constraints on any attempts at generalization. Yet, since its inception, microscopy of the central nervous system (CNS) has involved a sustained effort to define the laws of spatial arrangement and of connectivity distinguishing specific structures. However, the question which naturally arises from the above is whether these structural features may reflect, and perhaps determine, fundamental differences in the mode of operation of distinct brain structures. Alternatively, the possibility may exist that such structural specializations merely represent anatomical ‘accidents of development’, perhaps reflecting phylogenic origin, but playing a functional role which is no more significant than, for example, that of the appendix or the coccygeal vertebrae in man.

It is difficult to provide an answer to this question from the presently available anatomical and physiological data. Although substantial neurohistological data, on one hand, and neurophysiological information, on the other, are available, meaningful correlation of these two sets of data can only be accomplished in very isolated instances. In general, unlike recording from invertebrates, where the simplicity and viability of the nervous system makes it feasible to observe the elements recorded, physiological studies of the mammalian CNS are performed in a ‘blind’ fashion and it is exceedingly difficult to correlate these studies with the microscopical anatomy of the tissue.

Introduction

Networks of formal neurons provide simple models for distributed, content addressable, fault-tolerant memory. Many numerical and analytical results have been obtained recently, especially on the Hopfield model (Hopfield, 1982). In this model, a network of a large number of fully connected neurons, with symmetric interactions, has a memory capacity which increases linearly with the size of the network – that is, in fact, with the connectivity. When the total number of stored patterns exceeds this capacity, a catastrophic deterioration occurs, as total confusion sets in. Alternative schemes have been proposed (Hopfield, 1982; Nadal et al., 1986; Parisi, 1986) that avoid overloading: new patterns can always be learned, at the expense of more anciently stored ones which get erased – for this reason, these schemes have been called palimpsest. Numerical and analytical results (Nadal et al., 1986; Mézard et al., 1986) detail the behavior of these models, which show striking analogies with the behavior of human short-term memory. We will review the main results, and point out the possible relevance for human working memory.

In Section 13.2 the origin of the catastrophic deterioration in the standard scheme (Hopfield model) is simply explained. In Section 13.3 simple modifications are shown to lead to short-term memory effects. In Section 13.4 properties of these networks are exposed in close analogy with known data of experimental psychology.

Introduction

Neural networks with plasticity (dynamic connection coefficients) can recognize and associate stimulus patterns. Previous studies have shown the usefulness of a simple algorithm called ‘brain-washing’ which leads to networks which can have many eligible neurons with large variations in activity and complex cyclic modes (Clark & Winston, 1984). Methods of modifying connection coefficients are discussed and evaluated.

Successful pattern recognition with quasirandom, rather than topographic, networks would be much more significant and general. There is no doubt that topographic networks could be more efficient in the brain but less adaptable to changing conditions. A quasirandom network could be trained to recognize temporal and spatial stimuli, while a topographic network would be limited to a particular type of stimuli.

Three types of neurons have been incorporated into the network. A group of 10 input (stimulus) neurons (Ni) send µi efferents to neurons in the main network (see Figs. 30.1 and 30.2). Neurons in the main network are interconnected by µa afferent connections and µe efferent connections. In addition a group of output neurons (No) can be included to monitor activity of the main network and to train the network.

Components of a successful, sensible and biologically feasible training algorithm will be discussed.

Physical limitations to training algorithms

A specified neuron obtains the majority of information from afferent and efferent neurons.

Introduction

This article addresses a well-defined issue: does coherent firing of several neurons play a role in the function of the cerebral cortex? Its main purpose is to present the results of a computer simulation of a neural network in which the exact timing of impulses is indeed of paramount importance. And it is demonstrated that such a network would have potentially useful powers of discrimination and recall.

It is reasonably clear that the timing of the arrival of nerve impulses at a given neuron cannot be a matter of total indifference. The voltage across a neural membrane relaxes back towards its resting value, once a stimulus has been removed, so it is easy to envisage situations in which incoming impulses will fail to provoke a response unless they can act in unison by arriving simultaneously, or nearly so, at the somatic region. And there is a considerable corpus of evidence that the timing of incoming impulses is important. In the human auditory system, for example, small temporal offsets between impulse trains in the two cochlear nerves is exploited to locate sound sources, while the relative timing of impulses in the same nerve appears to be essential for the correct functioning of speech discrimination (Sachs, Voigt & Young, 1983). There is also evidence that the timing of sensory stimulation, down at the ten-millisecond level, is critically important for classical conditioning (Sutton & amp; Barto, 1981). And on the clinical side, one sees an extreme example of neuronal synchronization in the case of epilepsy, which apparently arises from mutual excitation between neurons (Traub & Wong, 1982).

A huge amount of physiological research shows the importance of the role of the cerebellum in motor control (Ito, 1984; and many others). It is natural for physiologists to try to understand what the cerebellum function is and when and which corrections or other modifications of cerebral cortical motor programs are introduced by the cerebellum. It seems to us that in recent decades the most interesting hypotheses on cerebellar performance have been proposed by the late David Marr in ‘A theory of cerebellar cortex’ (Marr, 1969). According to Marr, the main operation of the cerebellar circuitry is the switching on of proper motor commands by the current sensory input and the automatic adaptive acquisition of such cerebellar network capability. The location of the cerebellum – in the crest of almost all the ascending and descending nervous tracts – is definitely strategic for such a function. The unique combination of tens of thousands of granular cells (GrCs) and one climbing fibre (CF) at one Purkinje cell seems to be crucial for it.

The kernel of Marr's theory is composed by the postulates of the GrC–PC synaptic modification due to simultaneous excitation of the climbing fibre (CF) and parallel fibres (PF). In other words, Marr supposed that the Purkinje cell memorizes the afferent conditions in which it ought to be active.

Introduction

Rhythmic behaviour is characteristic for the nervous system at different hierarchical levels. Periodic temporal patterns can be generated both by endogenous pacemaker neurons and by multicellular neural networks. At single-cell level it was demonstrated, both experimentally and theoretically, that periodic membrane potentials could bifurcate to more complex oscillatory behaviour (ultimately identified by chaos) in response to drug treatment (Holden, Winlow & Haydon, 1982; Chay, 1984). Even the alteration of periodically synchronized oscillation and chaotic behaviour has been found in periodically forced oscillators of squid giant axons (Aihara, Matsumoto & Ichikawa, 1985).

The appearance of quasi-periodicity and chaos has been associated with abnormal neural phenomena not only at single neural level but as well at macroscopic scale connecting chaotic EEG dynamics to epileptic seizure (Babloyantz, Salazar & Nicholis, 1985). At intermediate level, chaotic behaviour was found in a model of the central dopaminergic neuronal system, and was associated with schizophrenics (King, Barchas & Huberman, 1984).

‘Normal’ and ‘abnormal’ dynamic behaviour, also at intermediate, namely synaptic level, has recently been investigated (Érdi & Barna, 1986; Éedi & Barna, 1987). Preliminary numerical calculations suggested that the regular periodic operation of synaptic level rhythmic generator of cholinergic system requires a fine-tuned neurochemical control system. Even mild impairment of the metabolism might imply ‘abnormal’ dynamic synaptic activity.

Memory disorders associated with Alzheimer's disease, partially due to disturbance of the control system of acetylcholine (ACh) synthesis, can be accompanied by change of dynamic patterns of firing frequency (Wurtman, Hefti & Melamed, 1981).

Introduction

The belief that complex macroscopic phenomena of everyday experience are consequences of cooperative effects and large-scale correlations among enormous numbers of primitive microscopic objects subject to short-range interactions, is the starting point of all mathematical models on which our interpretation of nature is based. The models used are basically of two types: continuous and discrete models.

The first have been until now the most current models of natural systems; their mathematical formulation in terms of differential equations allows analytic approaches that permit exact or approximate solutions. The power of these models can be appreciated if one thinks that complex macroscopic phenomena, such as phase transitions, approach to equilibrium and so on, can be explained in terms of them when infinite (thermodynamic) limits are taken.

More recently, the great development of numeric computation has shown that discrete models can also be good candidates to explain complex phenomena, especially those connected with irreversibility, such as chaos, evolution of macroscopic systems from disordered to more ordered states and, in general, self-organizing systems. As a consequence, the interest in discrete models has vastly increased. Among these, cellular automata, C.A. for short, have received particular attention; we recapitulate their definition:

A discrete lattice of sites, the situation of which is described at time t by integers whose values depend on those of the sites at the previous time t - τ (τ is a fixed finite time delay).

Introduction

Neural networks of spin glass type reveal remarkable properties of a content-addressable memory (Hopfield, 1982; Amit et al, 1985; Kinzel, 1985a). They are able to retrieve the full information of a learned pattern from an initial state which contains only partial information. Recently much effort has been devoted to the modeling of networks based on Hebb's learning rule (Cooper et al., 1979). These networks are the Hopfield model and its modifications. All have in common a local learning rule which allows the storage of orthogonal patterns without errors. The learning rule is local if the change of the synaptic coefficient depends only on the states of the two interconnected neurons and possibly on the local field of the postsynaptic one. This property seems to be essential from a biological point of view. However, the storing capability of these networks is strongly limited by the fact that they are not able to store correlated patterns without errors (Kinzel, 1985b).

On the other hand a storing procedure for correlated patterns is available (Personnaz et al, 1985; Kanter & Sompolinsky, 1986). But it involves matrix inversions which are not equivalent to a local learning mechanism. It is the purpose of this paper to present a new local learning rule for neural networks which are able to store both correlated and uncorrelated patterns. Moreover, this learning rule enables the network to fulfil two further important properties of natural networks: the learning process does not reverse the signs of the synaptic coefficients and leads to a network with unsymmetric bonds even if it starts from a symmetric one.

Introduction

The mechanisms of the complex functions attributed mostly to the cerebral cortex are hidden in the collective behaviour of a vast neural network that cannot practically be described in detail or in general. Cyclic modes of activity which emerge spontaneously in the dynamics of neural networks may underly possible mechanisms of short-term memory and associative thinking. The transitions from seemingly random activity patterns to cyclic activity have been examined in isolated networks with pseudorandomly chosen synapses and in networks with very simple architectures.

The basic computer model (Clark, Rafelski & Winston, 1985) envisions a collection of neurons, linked by a network of axons and dendrites that synapse onto one another. The synaptic interactions are modeled by a connection matrix V. The net algebraic strength of the connections from neuron j to neuron i, represented by the matrix element Vij can be positive (excitatory), negative (inhibitory) or zero (no connection). In the present study, the Vij were chosen randomly, but in accord with certain specified gross network parameters, viz.

N = net size = number of neurons in net,

m = connection density = probability that a given j → i link exists,

h = fraction of inhibitory neurons.

No more than one connection (‘synapse’) was allowed from any source neuron j to a given target neuron i.

The neurons update their states synchronously, corresponding to the assumption of a universal time delay δ for direct signal transmission.

The retina

Mathematical model of amphibian retina

The retina is composed of a variety of cell types including the photoreceptors, horizontal cells, bipolar cells, amacrine cells, and retina ganglion cells (for a review see Grüsser & Grüsser-Cornehls, 1976). Only the retina ganglion cells (RGCs) send axons to the brain of the animals. Therefore any visual information the brain may rely on is mediated by cells of this type.

According to their response properties the RGCs are usually divided into four classes, which here for simplicity are called R1, R2, R3, and R4. We have restricted our attention to the classes R2 and R3 because these form the majority (about 93%) of cells projecting to the tectum opticum, which is that area in the brain where recognition of prey objects is supposed to be centered.

The recognition process starts in the retina. The overall operation of the ganglion cells (R2, R3) and their precursors (photoreceptors, etc.) on some arbitrary visual scene can be decomposed into the following more primitive operational components:

(1) Let x(s, t) be any distribution of light in the visual field, where x denotes light intensity (we do not consider colored scenes), s = (s1, s2) some point in the visual field, and t is time.

(2) These ganglion cells do not respond to stationary, but only to transitory, illumination.

Introduction

In dealing with the problem of trying to describe mathematically neural tissue populations, known as neural nets, two approaches can be taken, the global and the microscopic. The global approach gives a phenomenological description of neural tissue populations. The microscopic, derived through appropriate simplifications, gives properties of the net from the properties of its constituents, the neurons, the connections and the synapses. While phenomenological theories appear to be easier to build and, overall, yield more results that are in agreement with experiments, it is impossible to build a realistic model of the brain without knowing the detailed functioning, interactions and interrelations of its constituents.

For the construction of an actual neural net machine, theories based on the properties of the fundamental constituents of the net will be more applicable to discovering the laws that govern the secrets of biological information processing. For instance, it is more relevant to simulate the activities of the brain from the underlying fundamental laws of nature. Examples taken from physics itself clarifies this point.

From a philosophical point of view it is more appealing to derive the laws of thermodynamics from the statistical behaviour of the particles that make up the system than to introduce thermodynamics as an independent branch of physics.

Analogously, it is more significant to derive the properties of superconductivity from the actual properties of the electrons and lattice than from phenomenological reasoning.

Introduction

Many features of the functional architecture of the mammalian visual system have been experimentally identified during the past 25 years. Among the most striking of these features is the presence of layers of orientation-selective cells – cells whose response to an edge or bar in the appropriate portion of visual field is sensitive to the local orientation of the input. These cells are organized in bands or ‘columns’ of cells of the same or similar orientation preference. The preferred orientation varies roughly monotonically, but with frequent breaks and reversals, as one traverses the cell layer. Orientation-selective cells, organized in this fashion, are found in cat, monkey, and other mammalian systems. In macaque monkey, they are present at birth.

I have found that several salient features of mammalian visual system architecture – including orientation-selective cells and columns – emerge in a multilayered network of cells whose connections develop, one layer at a time, according to a synaptic modification rule of Hebb type. The theoretical base is biologically plausible, none of the assumptions is specific to visual processing, no orientation preferences are specified to the system at any stage, and the features emerge even in the absence of environmental input.

The development of this system is discussed in detail, and references to experimental work are provided, in a series of three papers (Linsker, 1986a,b,c).

Introduction

The fundamental aim of simulation of neural nets is a better understanding of the functioning of the nervous system. Because of the complexities, simplifying assumptions have to be introduced from the beginning. In many frequently used models these assumptions are:

Each model neuron can be in one of few discrete states (e.g. ‘on’, ‘off’, ‘refractory’).

The totality of interactions between neurons is treated summarily by specifying few parameters, often just one ‘synaptic strength’, which is determined randomly, or by a deterministic algorithm, or by a combination of both.

The information processing by a neuron consists of comparing the value of a certain parameter, which is determined by incoming signals from other neurons, with some threshold. Essentially, when the value of this parameter exceeds the threshold, the neuron transits to another one of its possible states. A typical example for this parameter is the membrane potential at the axon hillock, which is determined by summing the action potentials impinging on this neuron during a certain time and whose value determines whether the neuron ‘fires’ or not.

The hypothesis that individual neurons can be described in a simple way makes the simulation of networks containing many model neurons possible. On the other hand, there are important parts of nervous systems, where model neurons simplified to the extent described above have little in common with reality.

Motivation

A neuron's electrophysiological response to a disturbance is within the millisecond range and consists of action potentials which are produced when the threshold potential is being crossed. This response can be measured with microelectrodes and related to the stimulus. With neuroanatomical responses, this is a completely different task. Morphogenetic changes, such as reactive synaptogenesis or degeneration, take hours to days, so they cannot be directly observed by microelectrodes or the like. Besides, it would be very hard to relate the measured functional changes of electrical signals to changes in the synaptical ‘hardware’. Therefore, animals have to be taken at different stages of time and the development of the central nervous system (with or without disturbances) has to be concluded from changes of their individual morphology. Moreover, to get reliable data, several animals have to be sacrificed at each time step. Sometimes, even that may not be enough, as the following example may show.

Measuring the development of the number of synapses in rat cortex from postnatal day 2 to adulthood (Balcar, Dammasch & Wolff, 1983), we expected to fit the data with a sigmoid curve corresponding to logistic growth, but the material appeared to be systematically disturbed. A population-kinetic model (Wagner & Wolff, subm.) developed at that time, could explain the disturbance as a temporary overshoot of free synaptic offers that were not distinguishable from bound synaptic elements.

In deciding on a chapter for this book I had to choose between my interests in bringing ideas from biology to computational hardware, and in trying to bring computational ideas towards the biological wetware. I chose the direction of biological wetware, but it is instructive to begin with a few words about VLSI ‘neural’ networks.

There have been sizeable associative memories built out of the VLSI kinds of hardware. A first effort succeeded in building a 22 neuron circuit (1). A 54-neuron circuit with its 3000 connections has been made at Bell Laboratories (2) and a 512-neuron circuit with its 256000 intersections has been fabricated, but not yet made to work (3). An interesting and significant feature which emerges from looking at these chips is the degree to which, although the idea of an associative memory is very simple, a large fraction of the devices and area on the chip are doing things which are essential to the overall function of the chip, but which seem peripheral to the idea of associative memory. I will describe a similar facet in the biological case, in going from the abstract idea of an associative task toward a complete neural system. Much paraphernalia must be added and many changes made in order that an elementary associative memory can perform a biological task.

Introduction

The detection of image regions and their borders is one of the basic requirements for further (object domain-) image processing in a generalpurpose technical pattern recognition system and, very likely, also in the visual system. It is a pre-requisite for object separation (figure-ground discrimination, and separation of adjoining and intersecting objects), which in turn is necessary for the generation of invariances for object recognition (Reitboeck & Altmann, 1984).

Texture is a powerful feature for region definition. Objects and background usually have different textures; camouflage works by breaking this rule. For texture characterization, Fourier (power) spectra are frequently used in computer pattern recognition. Although the signal transfer properties of visual channels can be described in the spatial (and temporal) frequency domain, there has been no conclusive evidence that pattern processing in the primary visual areas would be in terms of local Fourier spectra.

In the following we propose a model for texture characterization in the visual system, based on region labeling in the time domain via correlated neural events. The model is consistent with several basic operational principles of the visual system, and its texture separation capacity is in very good agreement with the pre-attentive texture separation of humans.

Texture region definition via temporal correlations

When we look at a scene, we can literally generate a ‘matched filter’ and use it to direct our attention to a specific object region.

Is there enough motivation for a solid-state physics approach to the brain?

One of the salient features of the brain networks is that anatomical sections of a few millimetres width, taken from different parts of the cortex, look roughly similar by their texture. This observation might motivate a theoretical approach in which principles of solid-state physics are applied to the analysis of the collective states of neural networks. Such a step, however, should be made with extreme care. I am first trying to point out a few characteristics of the neural tissue which are similar to those of, or distinguish it from non-living solids.

Similarities. There is a dense feedback connectivity between neighbouring neural units, which corresponds to interaction forces between atoms or molecules in solids.

Generation and propagation of brain waves seem to support a continuous-medium view of the tissue.

Differences. In addition to local, random feedback there exist plenty of directed connectivities, ‘projections’, between different neural areas. As a whole, the brain is a complex self-controlling system in which global feedback control actions often override the local ‘collective’ effects.

Although the geometric structure of the neural tissue looks uniform, there exist plenty of specific biochemical and physiological differences between cells and connections. For instance, there exist some 40 different types of neural connection, distinguished by the particular chemical transmitter substances involved in signal transmission, and these chemicals vary from one part of the brain to another.