Warren McCulloch and Walter Pitts in 1943 showed that simple networks of simulated neurons could evaluate any computable function and claimed that if supplemented by a tape and means of writing/altering symbols on it, could function as a Universal Turing Machine. Each unit of the network (a McCulloch-Pitts neuron) is a binary unit (2 states - on/off) receiving excitatory and inhibitory inputs (+ or -1) from other units or from outside the network. The state of a network emerges over several cycles. In each cycle, inhibitory input blocks firing. Excitatory inputs are summed and if exceeding threshold lead to firing - simplified model of a neuron, as also of an electric relay. Further associated neurons with propositions and their activation states with truth values.
Donald O. Hebb (1949) published The Organization of Behavior: A Neuropsychological Theory ``Stimulus and response - and what happens in the brain in the interval between them.'' (Emphasised anti-localisation approaches of the Gestalt theorists and his own mentor, Lashley see sec. )
``When an axon of cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both cells such that A's efficiency, as one of the cells firing B, is increased'' (p. 62). This idea laid the conceptual foundation for connectionism.
Hebb postulated that a particular stimulation leads to the slow development of a ``cell assembly'' - sensory neurons capable of acting together to stimulate other specific motor assemblies. These assemblies were created by an interaction in which repeated of firing of one cell due to another strengthens the connection between them.
Cell assemblies simulated on the digital computer included inhibitory connections and formulation of the ``learning rules'' employed in later connectionist modelling.
Apart from these models at architectural level there were more complex models, eg. of perception which used spacial maps of sensory inputs and motor outputs directing say eye movements.
Frank Rosenblatt (1962) constructed perceptrons (two-layer networks). Input device (retina) consisting of binary units with weighted connections to associator units which would become active when their combined input exceeded a threshold. The associator units in turn activated the response units. During each cycle the activity of the response unit was compared to the desired response. Connections which gave the correct response were strenthened and those giving the wrong one were weakened. Rosenblatt proved that if an appropriate set of connection weights existed, this procedure would find it. His work involved both mathematical modeling as well as experiments. His Mark I Perceptron had a 20X20 grid of photoelectric cells connected to electrical networks with variable connections.
A few years later Minsky and Papert (Perceptrons, 1969) proved that single-level perceptron networks could not calculate the simplest functions involved in pattern recognition, eg. they could not tell connected patterns from disconnected ones. Minsky and Papert's book sharply decreased interest in connectionism. Most cognitively-oriented computational work in the decade following the publication of Minsky and Papert's paper was based on symbolic processing, in the spirit of Newell and Simon's program. (Divided funds, practical limitations of hardware and available computer power, inability to deal with high-level cognition, aura of speculation and mystical holism.)
Revival of interest in neural networks
Though ignored by cognitive scientists, work in connectionist networks continued in the background, till in the mid-1980s interest in connectionism saw a dramatic revival. The factors:
'80s NETtalk (Sejnowski) could convert letters into phonemes and thus learn to read (learning took it from babbling to baby-like talk to clear talk).