|
Connectionist Computation |
|
Gualtiero Piccinini |
Abstract—The following three theses are inconsistent:
(1) (Paradigmatic) connectionist systems perform computations.
(2) Performing computations requires executing programs.
(3) Connectionist systems do not execute programs.
Many authors embrace (2). This leads them to a dilemma: either connectionist systems execute programs or they don’t compute. Accordingly, some authors attempt to deny (1), while others attempt to deny (3). But as I will argue, there are compelling reasons to accept both (1) and (3). So, we should replace (2) with a more satisfactory account of computation. Once we do, we can see more clearly what is peculiar to connectionist computation.
|
C |
onnectionist systems are sets of connected signal-processing units with some input units, some output units, and perhaps some hidden units. Each unit receives input signals and delivers output signals as a function of both its input and its current state. As a result of their units’ activities and their organization, connectionist systems turn the input received by their input units into the output produced by their output units. Connectionist systems are routinely used by psychologists and neuroscientists to model cognitive and neural functions. They have many other applications, ranging from recognizing signatures to modeling industrial processes.
Theories of mind based on connectionist systems are discussed as
an alternative to classical or “symbolic” computational theories of mind
According to some authors, connectionist systems compute in a different way than classical computing systems. (Classical systems include, for instance, Turing machines and ordinary computers.) To give an account of such a difference, these authors need an account of computation that distinguishes between classical and connectionist computation. According to other authors, however, connectionist systems are different from classical models in that they do not perform computations at all—they process their signals by non-computational means. To underwrite their view, these authors need a criterion for demarcating what computes from what doesn’t.
Unfortunately, there is no consensus on how to determine whether connectionist systems compute and in what sense they do. Answering these questions requires a clear and adequate account of computation. In this paper, I will recruit some recent philosophical work on computation to shed light on these questions. This paves the way for an improved understanding of the difference between connectionist and classical computational theories of mind.
The problem of connectionist computation may be formulated in terms of the following theses:
(1) (Paradigmatic) connectionist systems perform computation.
(2) Performing computations requires executing programs.
(3) Connectionist systems do not execute programs.
The above theses are mutually inconsistent. Many authors embrace thesis (2)
This view leads to a dilemma: either connectionist systems execute programs or they don’t compute. Accordingly, some authors attempt to deny (1), while others attempt to deny (3). But as I will argue, there are compelling reasons to accept both (1) and (3).
I’m prepared to accept that some
connectionist systems don’t perform computations. For instance, some connectionist systems manipulate
continuous variables
But it is difficult to deny that many paradigmatic examples of
connectionist systems perform computations in roughly the same sense in which
Turing machines and digital computers do.
The term ‘connectionism’ was first used in its present sense—meaning,
roughly, the modeling of psychological processes by artificial neural networks—by
Frank Rosenblatt in the late 1950s.
Roseblatt’s machines, and subsequent extensions and refinements thereof,
can be studied by the same formalisms and techniques that are employed to study
the performance of other paradigmatic computing systems; their computing power
can be defined and evaluated by the same measures
The term ‘connectionist
system’ is typically defined so as to encompass more than machines in
Rosenblatt’s tradition. Rosenblatt was
building on previous neural network models. He openly acknowledged that “the neuron model
employed
It just so happens that a McCulloch and Pitts neuron is functionally equivalent to a logic gate. A collection of (connected, synchronous, ordered) logic gates with input and output lines is a Boolean circuit. Boolean circuits are the components of standard digital computers. Because of this, there is a straightforward sense in which digital computers are connectionist systems. (This fact is not always appreciated by those who debate the merits of connectionist versus classical computational theories of mind. In that debate, connectionist systems are often pitched without further qualifications against classical systems such as digital computers, when in fact digital computers are just one (quite special) kind of connectionist system.) Unless precautions are taken to ensure that Boolean circuits and collections thereof are somehow excluded from consideration, denying that connectionist systems perform computations is tantamount to denying that computers compute.
Digital computers execute programs, and the execution of programs is an important aspect of the way they compute. Insofar as digital computers qualify as connectionist systems, at least some connectionist systems execute programs. Not so for more standard examples of connectionist systems, such as machines in Rosenblatt’s tradition. The latter machines process their input solely in virtue of the properties of and relations between their units. No separate program needs to be written, stored, or executed.
But according to an influential account, executing programs is a
necessary condition for computing (see (2) above). Thus, someone might wish to deny that
connectionist systems—or at least, paradigmatic examples of connectionist
systems—perform computations. Here is
something close to an outright denial: “so long as we view cognition as
computing in any sense, we must view it as computing over symbols. No connectionist device, however complex,
will do, nor will any analog computer”
Rejecting (1) (the idea that connectionist systems perform
computation) to save (2) (the idea that computing requires executing programs) may
sound like a reasonable position to take.
Unfortunately, this position doesn’t fit with the observation we just
made—that the input-output mappings produced by paradigmatic connectionist
systems may be characterized by the same formalisms employed by computability
theorists to characterize classical computing systems. To put it bluntly, the study of the
computational properties of paradigmatic connectionist systems is just one
branch of computability and computational complexity theory among others (see
Perhaps we could save (2) by denying (3) and concluding that connectionist systems do execute programs after all. Except for one thing: what’s the evidence that connectionist systems execute programs? Did we somehow miss the presence of memory components in which connectionist systems store their programs? Of course not. The argument that connectionist systems execute programs proceeds by weakening the notion of program execution to the point that, contrary to what you might have supposed, executing programs does not require writing, storing, or physically manipulating a concrete program in any way—at least in the sense in which programs are manipulated by ordinary computers.
Here is a recent example:
“programs … are just specifications of functional dependencies, and … a
system executes a program if the system preserves such dependencies in the
course of its state changes”
This is not just an ad hoc proposal. Something analogous to this weakened notion
of program execution predates the rise of neo-connectionism in the mid-1980s,
which brought to the attention of philosophers the question of whether and in
what sense connectionist systems perform computations. A similar notion was often employed during
discussions of classical computing systems:
“programs aren’t causes but abstract objects or play-by-play accounts”
This proposal may be put in the following terms: all that program execution requires is acting in accordance with a program. Acting in accordance with a program, in turn, may be explicated in at least two ways.
In the ordinary sense, ‘acting in accordance with a program’ means
performing the operations specified by the program in the specified order. For instance, a program for multiplication
might require the computation of partial products followed by their
addition. A system that acts in
accordance with such a program must, at a minimum, generate outputs
corresponding to partial products and then generate an output corresponding to
their sum. (When Cummins writes ‘program
execution’, he appears to mean acting in accordance with a program in roughly this
sense
This notion of acting in accordance with a program does not solve the present problem. Typical connectionist systems do not generate successive outputs corresponding to separate computational operations defined over their inputs. Rather, they turn their inputs into their outputs in one step. Thus, they do not act in accordance with a program in the ordinary sense.
More recently, Roth
Consider again a connectionist system that performs multiplications. The system acts in accordance with a partial products program if and only if the system’s connection weights are derived from a partial product program, even though the system produces its output in one step. By contrast, if the connection weights are derived from a different multiplication program—such as a program for multiplying by successive additions—then the system doesn’t act in accordance with a partial products program (but rather, a successive additions program).
Roth’s notion of acting in accordance with a program appears to provide an ingenious solution to the present conundrum. If we accept that executing a program amounts to acting in accordance with one, we can now use Roth’s notion of acting in accordance with a program to conclude that at least those connectionist systems that act in accordance with a program in Roth’s sense do perform computations. This suggestion is not as helpful as it seems.
First, how can we derive connection weights from programs? Roth refers to a technical report by P. Smolensky, G. Legendre, and Y. Miyata. These authors describe a technique for defining the connection weights of certain connectionist systems from certain computer programs, in such a way that the resulting connectionist systems are functionally equivalent to the programs. Smolensky et al.’s technique applies only to a special class of connectionist systems. What about all the others, including most of the paradigmatic ones? Roth’s notion of program execution does not appear to apply to them. If we wish to say that they compute, as per Section II, we need an alternative account of connectionist computation.
Second, it remains to be seen whether, after the connection weights are defined with Smolensky et al.’s technique, the best thing to say is that the program is executed by the connectionist system (as Roth puts it). It seems more appropriate to say that the connectionist system computes the function defined by the program without executing the program. The latter, in fact, appears to be Smolensky, Legendre, and Miyata’s view.
Finally, suppose you want to say that a connectionist system executes a program based on Roth’s proposal. Which programming language is it written in? In ordinary computers, this is a question with a well-defined answer. But it appears that the connectionist systems described by Roth act in accordance with any program, written in any programming language, that specifies the relevant operations in the relevant order. This makes it hard to pick one of these programs as the one the system is executing. Or does it execute them all?
We need not press these questions further. For even if we grant Roth his notion of acting in accordance with a program, we should resist his conclusion that the connectionist systems he describes execute programs in the relevant sense. If acting in accordance with a program is sufficient for executing programs, then at least the systems defined using Smolensky et al.’s technique may well execute programs. This might work as a new meaning of ‘program execution’. But in the sense of ‘program execution’ employed in computer science, acting in accordance with a program is hardly sufficient for program execution.
In computer science, a program is a list of instructions implemented in a concrete string of digits; ‘executing a program’ means responding to each instruction by performing the relevant operation on the relevant data. As I’m using the term, digits are discrete physical entities or stable states, which may be stored in memory components and transmitted from component to component.
The modern notion of program execution originates as a way to characterize a special property of modern computers (and some other machines, as we shall see in the next section). Computers do much more than acting in accordance with a program. They can act in accordance with any number of programs. The reason for that is that they can store physical instantiations of the programs, and it is those physical instantiations that, together with input data, drive computers’ processes.
Programs in this sense are much more than descriptions of a system’s behavior or some feature of its connection weights. They are strings of digits that computers can respond to and manipulate in the same way they respond to and manipulate data. They can be written, tested, debugged, downloaded, installed, and, of course, executed. It is the execution of programs in this sense that explains the behavior of ordinary computers. This is also the notion of program execution that contributed to inspire the analogy between minds and computers, on which computational theories of mind are based.
If we want to honestly assess whether connectionist systems execute programs in the relevant sense, and use this assessment to compare different computational theories of mind, we should use the standard notion of program execution that is used in computer science. And in this sense of ‘program execution’, paradigmatic connectionist systems do not execute programs.
Our hands are tied. We have found that connectionist systems perform computations even though they don’t execute programs. We must conclude, contra (2), that computation doesn’t require program execution. This is not a bad thing. In fact, several independent considerations point to the same conclusion.
First, nothing in the notion of computation studied by computer scientists and computability theorists entails that computation must be performed by executing a program. The original mathematical notion of computation is that of a process that accords with the steps of an algorithm or effective procedure—there is no further requirement that the algorithm be implemented by a program, or that the program be executed.
Second, the notion of program-controlled machines, of which program-controlled computers are a species, did not even originate in the field of computing. It originated in the textile industry. The first technology for programming machines—punched cards and the mechanisms for executing them—was developed in the 18th Century to control the weaving of patterns in mechanical looms. In 1801, Joseph Marie Jacquard developed an improved version of this technology for his Jacquard Loom. Only later did Charles Babbage borrow Jacquard’s idea to control his Analytical Engine. The Analytical Engine, which was never actually constructed, was the first (hypothetical) program-controlled computer.
Third, there is a familiar distinction between computing systems that execute programs and computing systems that don’t. Computation, including mechanical computation, is much older than modern, program-controlled computers. Finite state automata, ordinary calculators, and many other computing devices don’t execute programs in the sense in which full-blown computers do. Quite aside from paradigmatic connectionist systems, there are plenty of systems that compute without executing programs.
Finally, the standard explanation of how computers execute programs appeals to components that compute without executing programs. Computers execute programs in virtue of possessing the relevant kind of processing units. Such processing units contain a control and a datapath. When an instruction reaches the processing unit, the control and the datapath perform two different functions.
The control receives one part of the instruction—the part that encodes a command (e.g., sum, multiply, etc.). Then, the control determines which operation must be performed on the data and sends an appropriate signal to the datapath. The datapath receives the command from the control plus the remaining part of each instruction—the part that encodes the data. After that, the datapath performs the relevant operation on the data and sends the result out.
The control and the datapath are computing components—in the language of computability theory, they are finite state automata. But neither of them alone executes any program—it is only their organized effort, in cooperation with memory components, which allows the computer to execute programs. Thus, program execution itself requires components that compute without executing programs.
In recent years, I have articulated and defended a general
account of computation that naturally accommodates both computing systems that
execute programs and computing systems that don’t
All paradigmatic computing systems, such as Turing machines, finite state automata, and digital computers, perform computations in this sense. All paradigmatic examples of connectionist systems perform computations in this sense too. Briefly, when the properties of paradigmatic connectionist systems are characterized computationally, the inputs to their input units and the outputs from their output units are relevant only when they are in one of a finite number of (equivalence classes of) states at certain times. Hence, connectionist systems’ inputs and outputs—though not necessarily the inputs and outputs of their hidden units—are digits in the present sense.
Even when the value of a (paradigmatic) connectionist system’s inputs or outputs can vary continuously, there are discontinuities in the functional significance of the values. Only values in certain neighborhoods—typically, there are two neighborhoods, which may be labeled ‘0’ and ‘1’—are counted as determining what the whole system’s input or output is; there are gaps between neighborhoods; and all values within each functionally significant neighborhood are counted as functionally equivalent. Hence, the systems’ inputs and outputs still constitute digits.
Furthermore, depending on the kind of system, either the spatial ordering of the input units or the temporal sequence of the input units’ inputs constitute a string of digits, and either the spatial ordering of the output units or the temporal sequence of the output units’ outputs constitute a string of digits. Such strings are the entities in terms of which the computation power of the system is defined. Finally, the way the system’s outputs are (or ought to be) functionally related to its inputs (plus, perhaps, its internal states) constitute a rule defined over the digits. Such a rule defines the computation performed (or approximated) by the system.
There is at least one important difference between the computations performed by classical computing systems and those performed by paradigmatic connectionist systems (such as machines in Rosenblatt’s tradition). Unlike the former, the latter do not act in accordance with a step-by-step procedure, or algorithm. Classical computation proceeds by performing one operation at a time, where an operation is a change of one string of digits into another. Since strings are discrete entities, changing one string into another is a discrete operation. Thus, a classical computational dynamics is by definition discrete. By contrast, paradigmatic connectionist systems proceed by letting their units affect each other’s activation values according to continuous dynamical relations. A connectionist system’s dynamics is the analogue of an ordinary computer’s digital logic. Whereas digital logic determines discrete dynamics, the typical dynamic of a connectionist system is continuous.
Unlike classical computing systems, (paradigmatic) connectionist systems may be trained. Training is the adjustment of the system’s dynamics to fit a desired input-output mapping. Thus, the dynamics of connectionist systems may change over time independently of which inputs they get. In other words, a connectionist system may evolve so as to yield different outputs in response to the same inputs at different times. I’m not suggesting that the system’s dynamics is the only useful level of description for connectionist systems. I’m just saying that there is no way to understand the causal mechanism behind the system’s input-output mapping and its evolution over time without understanding the dynamical relations between the system’s units and the way they can change.
If all of this is right, why do connectionists speak about connectionist algorithms, as they often do? The term ‘connectionist algorithm’ is used in several ways. The present account allows us to distinguish and elucidate all of them.
Sometimes, ‘connectionist algorithm’ is used for the procedure that trains a connectionist system, that is, the procedure for adjusting the dynamical relations between units. This is typically an algorithm in the classical sense from computability theory—a list of instructions for manipulating strings of discrete entities. A connectionist algorithm in this sense is not what drives any connectionist computation: connectionist computations are driven by the state of and dynamical relations between their units. Thus, from the fact that a system is trained using an algorithm, it doesn’t follow that the system itself computes by following an algorithm.
But ‘connectionist algorithm’ is also used for the process by which a connectionist system produces its output. Based on what we have seen, this is not an algorithm in the classical sense. Paradigmatic connectionist systems do not have the kind of discrete dynamics defined over strings of digits that can be accurately described by algorithms in the classical sense. Therefore, this second usage of ‘connectionist algorithm’ is equivocal. But given how common it has become, it may be considered a new sense of the term ‘algorithm’—not to be confused with the classical sense.
Finally, the process by which a series of connectionist systems, each one feeding into the next, generates a final output through intermediate connectionist processes may be called a ‘connectionist algorithm’. This may well be an algorithm in the classical sense—or something close—provided that each intermediate process is definable as a transformation of strings of digits.
In sum, the idea that executing programs is necessary for computing leads to the dilemma that either connectionist systems execute programs, or they don’t compute. On the first horn, we must disconnect the notion of program execution from the special type of mechanism that gave rise to the idea of (computational) program execution in the first place. On the second horn, we must say that systems that compute by the lights of computability theory do not, in fact, compute. Fortunately, we need not impale ourselves on either horn. It is more reasonable to take a broader view of computation, according to which computation doesn’t require program execution. This is a good result, with plenty of independent motivation. After abandoning the mistaken notion that computation requires program execution, we gain a better understanding of connectionist computation.
Acknowledgment
Thanks to Carl Craver, Martin Roth, and Anna-Mari Rusanen for helpful comments.
References
[1]
C. Macdonald and
G. Macdonald, eds., Connectionism:
Debates on Psychological Explanation, Volume Two.
[2]
J. A. Fodor, The Language of Thought.
[3]
Z. W. Pylyshyn, Computation and Cognition.
[4]
R. Cummins, Meaning and Mental Representation.
[5]
M. Roth “Program
Execution in Connectionist Networks,” Mind
and Language, vol. 20, no. 4, pp. 448-467, 2005.
[6]
T. Chen and H.
Chen, “Approximations of Continuous Functionals by Neural Networks with
Application to Dynamic Systems,” IEEE
Transactions on Neural Networks, vol. 4, no. 6, pp. 910-918, 1993.
[7]
M. L. Minsky and
S. A. Papert, Perceptrons: An
Introduction to Computational Geometry, expanded edition.
[8]
F. Rosenblatt, Principles of Neurodynamics: Perceptrons and
the Theory of Brain Mechanisms. Washington, D. C.: Spartan, 1962.
[9]
C. R. Gallistel
and J. Gibbon, The Symbolic Foundations
of Conditioned Behavior.
[10] J. Šíma and P. Orponen, “General-purpose Computation
with Neural Networks: A Survey of Complexity Theoretic Results,” Neural Computation, vol. 15, pp. 2727-2778,
2003.
[11] R. Cummins, The
Nature of Psychological Explanation.
[12] R. Cummins, “Programs in the Explanation of Behavior,” Philosophy of Science, vol. 44, pp.
269-287, 1977.
[13] R. Cummins and G. Schwarz, “Connectionism, Computation,
and Cognition,” in Connectionism and the
Philosophy of Mind, T. Horgan and J. Tienson, eds.,
[14] G. Piccinini, “Computational Modeling vs. Computational
Explanation: Is Everything a Turing Machine, and Does It Matter to the
Philosophy of Mind?” Australasian Journal
of Philosophy, vol. 85, no. 1, pp. 93-115, 2007.
[15] G. Piccinini, “Computation without Representation,” Philosophical Studies, to be published,
2007.
[16] G. Piccinini, “Computing Mechanisms,” Philosophy of Science, to be published.
Manuscript received December 20, 2006. This work was
supported in part by a grant from the
G. Piccinini is with the University of Missouri – St. Louis, St. Louis, MO 63121-4400 USA (phone: 314-516-6160; fax: 314-516-4610; e-mail: piccininig@umsl.edu).