Cellular Neural Networks Reported by:

Hamada Ahmed Khadragy

Abstract:
A novel class of information processing system called cellular neural network is proposed. Like a neural network, it is a largescale non linear analog circuit which processes signals in real time. Like cellular automata, it is made of a massive aggregate of regularly spaced circuit clones called cells, which communicate with each other directly only through its nearest neighbors. Each cell is made of a linear capacitor, a nonlinear voltage-controlled current source, and a few resistive linear circuit elements. Cellular neural networks share the best features of both worlds; its continuous time feature allows real-time signal processing found wanting in the digital domain and its local interconnection feature makes it tailor made for VLSI implementation. Cellular neural networks are uniquely suited for high speed parallel signal processing.[1]

Introduction:
After the rebirth of the interest for artificial Neural Networks in 1982 [1], many different implementations for this concept have been developed. The differences between these implementations affect the overall characteristics of the network and thus the applicability for a certain purpose like, for example, real time image processing. Here, an important feature of neural networks is touched upon; these circuits can operate very fast since the 'signal processing' is done simultaneously in parallel neurons, rather than in sequential time steps. The major disadvantage of most neural network implementations however is the number of interconnections between neurons. In order to reduce the number of interconnections but keep the advantages of parallel processing, Chua and Yang proposed a so-called Cellular Neural Network (CNN) in 1988 where neurons were only connected to other neurons within a certain neighborhood.
In 1992, Rodriguez-Vazquez proposed a modified model for neurons; the so-called

Full Range (PR) model and an implementation for a CNN in the current domain based on this FR-model. In 1996 P. Bruin suggested an implementation for a FR-CNN in the voltage domain. However, after realization of the implementation, it was found to display instabilities. Every neuron in a cellular neural network can be described by a certain stateequation. If this network consists of two interconnected cells, the behavior that is displayed by those cells can be plotted in a so-called state space. It is interesting to be
2

able to 'predict' to what final state the CNN will converge. For this, the state-space will be partitioned into 'basins of attraction', separated by certain boundaries. In this report a method to find the boundaries of the basins of attraction is proposed.

A Cellular Neural Network

A Cellular Neural Network (CNN) is an artificial neural network consisting of separate neurons or cells. It has several properties that may be advantageous compared to other neural networks. The dynamic range of a CNN for example is bounded. Also a CNN can easily be extended without having to re-adjust the entire network because a cell is not connected to every other cell in the network but rather to cells within a certain neighborhood. Albeit its cellular structure it still displays the complex dynamic behavior as seen with other neural networks. Due to this complex behavior it can be used in image processing (e.g. noise-removal, connected component detection (CCD), 'thinning' etc.) It can also be used to simulate certain equations or be used as an associative memory. Another advantage is that although the factor with which the output of a cell affects the behavior of other cells (template parameter) may be different for spatially different neighboring cells, the template thus formed is translational invariant ('cloning templates'). As stated above, these cells can interact with other cells within a certain neighborhood. A rectangular twodimensional CNN consisting of 16 cells where every cell interacts with only its directly neighboring cells would look like the circuit shown in Figure 1.

If C(i,j) is the cell on the ilh row and jth column then cell C(2,2) is connected to C(O,I), C(O,2), C(O,3), C(2,I), C(2,2), C(2,3), C(3,l), C(3,2) and C(3,3).

Architecture of Cellular Neural Networks

Any cell in a CNN is connected only to its neighbor cells. The adjacent cells can interact directly with each other. Cells not directly connected together may affect each other indirectly because of the propagation effects of the dynamics of CNNs. An example of a two-dimensional CNN is shown below.

Every cell is influenced by a limited number of cells in its environment. This locality of connections between the units is the main difference between CNNs and other neural networks. Large CNN chips can be implemented using VLSI techniques. The figure above shows the emphasized cell (black) connected to the nearest neighbors (gray). The cells marked in gray represent the neighborhood cells of the black cell. The neighborhood includes the black cell itself. This is called a "3*3neighborhood". Similarly, we could define a "5*5-neighborhood", a "7*7-neigborhood" and so on. The basic circuit unit of CNNs is called a cell. It contains linear and nonlinear circuit elements, which typically are linear capacitors, linear resistors, linear and nonlinear controlled sources, and independent sources. All the cells of a CNN have the same circuit structure and element values. A typical circuit of a single cell is shown in the figure below.

Each cell contains one independent voltage source E uij (Input), one independent current source I (Bias), several voltage controlled current sources Inu ij, Iny ij, and one voltage controlled voltage source Ey ij (Output). The controlled current sources Inu ij are coupled to neighbor cells via the control input voltage of each neighbor cell. Similarly, the controlled current sources Iny ij are coupled to their neighbor cells via the feedback from the output voltage of each neighbor cell. The cell C(i,j) has direct connections to its neighbors through two kinds of weights: the feedback weights a(k,l;i,j) and the control weights b(k,l;i,j), where the index pair (k,l;i,j) represents the direction of signal from C(i,j) to C(k,l). The coefficients a(k,l;i,j) are arranged in the feedback-Template or A-Template. The coefficients b(k,l;i,j) are arranged in the control-Template or B-Template. The A-Template and the B-Template are assumed to be the same for all the cells in the network. The global behavior of a CNN is characterized by a Template Set containing the A-Template, the B-Template, and the Bias I. If we assume a "3*3-neighborhood", the Template Set consists of 19 coefficients. The external input to the cell is typically assumed to be constant over a certain operation interval. Therefore, the total input current to the cell is given by the weighted sum of control inputs and weighted sum of feedback outputs. In addition, a constant bias term (I) is added to the cell. Due to the capacitance C and resistance R, the state voltage x(i,j) satisfies the following differential equation:

k denotes the neighborhood of the specific cell

Without loss of generality, the time constant T = R*C can be set to 1. The only nonlinear element in each cell is a piecewise-linear voltage controlled voltage source with characteristic y(i,j) = f(x(i,j)). A widely used nonlinearity is the piecewise-linear function as given by: y(i,j) = f(x(i,j)) = 0.5*(|x + 1| - |x - 1|)

The block diagram of a cell C(i,j) is shown in the figure below.

Modeling

and

Simulation

of

the

CNN

architecture
-Simulation plays an important role in the design of the CNN cloning templates. -Therefore, it has to be fast enough to allow the design phase of various templates be accomplished in reasonable time. -At the same time, the simulation has to be accurate enough, to reflect the behavior of the analog circuitry correctly. -In practice, the simulation of the CNN involves a trade-off between accuracy and computation time -The true processing capabilities of CNNs for high-speed parallel processing are only fully exploited by dedicated VLSI hardware realizations. -Typical CNN chips may contain up to 200 transistors per pixel. -At the same time, industrial applications require large enough grid sizes (around 100 x 100).

-Thus, CNN chip designers must confront complexity levels larger than 106 transistors, most of them operating in analogue mode -On the one hand, high-level simulation, which is focused on emulating the functional behaviour, cannot reflect realistically the underlying electronic circuitry. Their lack of detail makes them ill-suited for reliable IC simulation. -On the other hand, the SPICE-type transistor-level simulators, although very accurate, are barely capable of handling more than about 105 transistors and may take several days of CPU time for circuit netlists containing about 106 transistors. -Hence, these low-level tools are ill-suited for simulating large CNN chips. -Therefore, it would be necessary to bridge the gap between these approaches, which would give very accurate results in reasonable (but not real-) time. -However, our main concern now is fast simulation

Global behavior of Cellular Neural Networks

In image processing, n-by-m rectangular grid arrays are often used. n and m are the numbers of rows and columns, respectively. Each cell in a CNN corresponds to an element of the array. Assuming that each cell is connected to its nearest neighbors only ("3*3neighborhood") and that the local connections of a cell do not depend on the cell's position, the Template set contains 19 coefficients (A-Template: a1 .. a9, B-Template: b1 .. b9, Bias I). The behavior of the CNN is completely determined by this Template set. New CNN-Templates for arbitrary tasks may be found using a training algorithm, or by defining local rules for a given global task. The local rules describe a cell's equilibrium state depending on the inputs and outputs of the neighbor cells. The inputs and the outputs of the neighbor cells are assumed the be constant. The dynamics of the cell is not specified. If Template values for the local rules are found, simulations are very helpful to test the dynamic global behavior of the entire clone of cells. Optimal coefficient calculation leads to solutions which converge after short time. This means that the output of every cell reaches its final output y=+1 or y=-1 after short time.

Characteristics of a simple CNN Template

Template set We assume "3*3-neighborhood". A simple Template set for edge extraction is
7

Global task: Binary edge detection If the input image is a binary image (black and white), the output of the CNN will be a binary image showing the edges of the input image only. If the input image has intermediate (gray) values, the operation of the CNN with this simple Template set is not well defined.. Input: U(t) = static binary image Initial state: X(0) = arbitrary (reason: Feedback Template = 0) Output: Y(t) converges toward a binary image showing all edges of the input image.

Local rules Regarding the global tasks, we can define local rules for a single cell. The rules are deduced from the following reflections: 1. A white pixel never turns black. 2. A black pixel turns white if it is surrounded by black pixels 3. A black pixel never turns white, when at least one neighbor cell is white. In this case, this cell belongs to the edge of the object

Input u(i, j) Output y(i, j) (t = infinity) 1. white pixel white, independent of neighbors 2. black pixel white, if all nearest neighbors are black 3. black pixel black, if at least one nearest neighbor is white

Global behavior The screenshots below show the correct behavior of the Template set.

The input picture above is the input picture of the CNN. It does not change in this small time period. The four pictures below show the dynamcal behavior of the entire grid. Starting with the initial state on the left side, the pictures show the state of the CNN cluster at t = 0.2, t = 0.5 and t = 2.

After two time units (t = 2), the output of the CNN shows the edges of the input image. The Template set produces the desired output.

Application
The philosophy, interests, and methodologies of CNN researchers are varied. Due to the potential of the CNN architecture, this platform has attracted people from a variety of backgrounds and disciplines. Some are exploring practical implementations of CNN processors, others are using CNN processors to model physical phenomena, and there are even researchers exploring theoretical mathematical, computational, and philosophical ideas through CNN processors. This chapter briefly explores what are
9

CNN processors and what they are being used for. Some applications are engineering related, where some known, understood behavior of CNN processors is exploited to perform a specific task, and some are scientific, where CNN processors are used to explore new and different phenomenon. Hopefully this section will illuminate that CNN processors are versatile platforms that are being used for a variety of applications. CNN processors were designed to perform image processing; specifically, the original application of CNN processors was to perform real-time ultra-high frame-rate (>10,000 frame/s) processing unachievable by digital processors needed for applications like particle detection in jet engine fluids and spark-plug detection. Currently, CNN processors can achieve up to 50,000 frames per second, and for certain applications such as missile tracking, flash detection, and spark-plug diagnostics these microprocessors have outperformed a conventional supercomputer. CNN processors lend themselves to local, low-level, processor intensive operations and have been used in feature extraction, level and gain adjustments, color constancy detection, contrast enhancement, deconvolution, image compression, motion estimation, image encoding, image decoding, image segmentation, orientation preference maps, pattern learning/recognition, multi-target tracking, image stabilization, resolution enhancement, image deformations and mapping, image inpainting, optical flow, contouring, moving object detection, axis of symmetry detection, and image fusion. Due to their processing capabilities and flexibility, CNN processors have been used or has been prototyped for novel field applications such as flame analysis for monitoring combustion at a waste incinerator, mine-detection using infrared imagery, calorimeter cluster peak for high energy physics, anomaly detection in potential field maps for geophysics, laser dot detection, metal inspection for detecting manufacturing defects, and seismic horizon picking. They have also been used to perform biometric functions such as fingerprint recognition, vein feature extraction, face tracking, and generating visual stimuli via emergent patterns to gauge perceptual resonances. CNN processors have been used for medical and biological research in performing automated nucleated cell counting for detecting hyperplasia, segment images into anatomically and pathologically meaningful regions, measuring and quantifying cardiac function, measure the timing of neurons, detecting brain abnormalities that would lead to seizers. One potential future application of CNN microprocessors is to combine them with the DNA microarrays to allow for a near-real time DNA analysis of hundreds of thousands of different DNA sequences. Currently, the major bottleneck of this DNA microarray analysis is the amount of time needed to process data in the form of
10

images, and using a CNN microprocessor, researchers have reduced the amount of time needed to perform this calculation to 7ms. CNN processors have also been used to generate and analyze patterns and textures. One motivation was to use CNN processors to understand pattern generation in natural systems. They were used to generate Turing patterns in order to understand the situations in which they form, the different types of patterns which can emerge, and the presence of defects or asymmetries. Also, CNN processors were used to approximate pattern generation systems that create stationary fronts, spatio-temporal patterns oscillating in time, hysteresis, memory, and hetergenity. Furthermore, pattern generation was used to aid high-performance image generation and compression via real-time generation of stochastic and coarse-grained biological patterns, texture boundary detection, and pattern and texture recognition and classification. Control and Actuator Systems There is an ongoing effort to incorporate, CNN processors into sensory-computingactuating machines as part of the emerging field of Cellular Machines. The basic premise is to create an integrated system that uses CNN processors for the sensory signal processing and potentially the decision making and control. The reason is that CNN processors can provide a low power, small size, and eventually low cost computing and actuating system suited for that type of system. These Cellular Machine will eventually create a Sensor-Actuator Network (SAN), a type of Mobile Ad Hoc Networks (MANET) which can be used for military intelligence gathering, surveillance of inhospitable environments, maintenance of large areas, planetary exploration, etc. CNN processors have been proven versatile enough for some control functions. The have been used as associative memories, optimize function via genetic algorithm, measuring distances, optimal path finding in a complex, dynamic environment, and theoretically be used to learn and associate complex stimuli. They have also been used to create antonymous gaits and low-level motor for robotic nematodes, spiders, and lamprey gaits using a Central Pattern Generator (CPG). They were able to function using only feedback from the environment, allowing for a robust, flexible, biologically inspired robot motor system. CNN-based systems were able to operate in different environments and still function if some of the processing units were disabled. The variety of dynamical behavior seen in CNN processors make them intriguing for communication systems. Chaotic communications using CNN processors is being
11

researched due their potential low power consumption, robustness and spread spectrum features. The premise behind chaotic communication is to used a chaotic signal for the carrier way and to use chaotic phase synchronization to reconstruct the message. CNN processors can be used on both the transmitter and receiver end to encode and decode the message. They can also be used for data encryption and decryption, source authentication through watermarking, detecting of complex patterns in spectrogram images (sound processing), and transient spectral signals detection. CNN processors are neuromorphic processors, meaning that they emulate certain aspects of biological neural networks. The original CNN processors were based on mammalian retinas, which consist of a layer of photodetectors connected to several layers of locally coupled neurons. This makes CNN processors part of an interdisciplinary research area whose goal is to design systems that leverage knowledge and ideas from neuroscience and contribute back via real-world validation of theories. CNN processors have implemented a real-time system that replicates mammalian retinas, validating that the original CNN architecture modeled the correct aspects of biological neural networks used to perform. However, CNN processors are not only limited to verifying biological neural networks associated with vision processing; they have been used to simulate dynamic activity seen in mammalian neural networks found in the olfactory bulb and locust antennal lobe, responsible for pre-processing sensory information to detect differences in repeating patterns. CNN processors are being used to understand systems that can be modeled using simple, coupled units, such as living cells, biological networks, physiological systems, and ecosystems. The CNN architecture captures some of the dynamics often seen in nature and is simple enough to analyze and conduct experiments. They are also being for stochastic simulation techniques, which allows scientists to explore spin problems, population dynamics modes, lattice gas models, percolation, etc. Other simulation includes heat transfer, mechanical vibrating systems, protein production, Josephson Transmission Line (JTL), seismic wave propagation, and geothermal structures. The 3D (Three Dimensional) CNN have been used to prove as complex shapes are emergent phenomena establishing a link between art, dynamical systems and VLSI technology. CNN processors have been used to research a variety of mathematical concepts, such as researching non-equilibrium systems, constructing non-linear systems of arbitrary complexity using a collection of simple, well-understood dynamic systems, studying emergent chaotic dynamics, generating chaotic signals, and in general discovering new dynamic behavior. They are often used in researching systemics, a trandisiplinary, scientific field that researches natural systems. Their goal
12

is to develop a conceptual and mathematical framework necessary to analyze, model, and understand systems, including, but are not limited to, atomic, mechanical, molecular, chemical, biological, ecological, social and economic systems. Topics explore are emergences, collective behavior, local activity and its impact on global behavior, and quantifying the complexity of an approximately spatially and topologically invariant system. Although another measure of complexity many not make some people enthusiastic (Seth Lloyd, a professor from Massachusetts Institute of Technology (MIT), has identified 32 different definitions of complexity), it can be potentially be mathematically analyze systems such as economic and social systems.

References
[1] L.O. Chuo and L.Yang, "Cellular neural networks: Applications," IEEE Trans. Circuits Syst., pp. 1273-1290, this issue [2] J.J Hopfield, "Neural networks and physical system with emergent computational abilities," proc. Natl. Acad. Sci. USA. [3] D. Balya, G, Tmar, G. Cserey, and T. Roska, "A New Computational Model for CNN-Ums and its Computational Complexity", Intl Workshop on Cellular Neural Networks and Their Applications, 2004 [4] L. Chua, T. Roska, Cellular Neural Networks and Visual Computing: Foundations and Applications, 2005. [5] V. Cimagalli, M. Balsi, "Cellular Neural Networks: A Review", Neural Nets WIRN Vierti, 1993. [6] H. Harrer and J.Nossek, "Discrete-Time Cellular Neural Networks", Int'l Journal of Circuit Theory and Applications, 20:453-467, 1992. [7] S. Majorana and L. Chua, "A Unified Framework for Multilayer High Order CNN", Intl Journal of Circuit Theory and Applications, 26:567-592, 1998. [8] T. Roska, L. Chua, "The CNN Universal Machine: An Analogic Array Computer", IEEE Trans. on Circuits and Systems-II, 40(3): 163-172, 1993

13