Architecture and stability of the second–order cellular neural networks

Accepted for publication: 25/09/2020 Abstract: In this paper, we study and propose i) Second-order Cellular Neural Networks (SOCNN) architecture based on standard CNN proposed by Leon. O. Chua with the 2nd-order polynomial inputs and bounded parameter assumptions. ii) The conditions for the existence and stability of the solutions of CNN are presented by choosing appropriate Lyapunov function. iii) Simulation and computing results by the simple example are perfomed on the Matlab (2014) Simulink. Keywords: Second-Order Cellular Neural Networks, Lyapunov function, stability. 1. Introduction Architectures of Artificial Neural Networks (ANN) involve the feedforward and the reccurent (or feedback) ANN. CNN are a type of the reccurent ANN and they have successfully created the CNN Universal Machine, the analog-logic array computer [7]. In recent years, high-order CNN (HOCNN) have attracted attention due to their wide range of applications in the fields such as signal and image processing, pattern recognition, optimization, and many other subjects [3], [4], [5], [6]. Many terms “high-order” for many types of HOCNN such as: i) HOCNN in sense “order of the time-varying delays”, for example, Zuda Huang [1] study the anti-periodic solution of the high-order cellular neural network. ii) Makoto Itoh and Leon O. Chua [9] define HOCNN by the differential equation system with n-variable (an nth-order cell) function. iii) We are studying HOCNN in the different sense. It is the product of multi-action of external inputs 1 2, , ..., nu u u and internal feedback signals y1, y2 yn attached to the same cell of SOCNN as presented by equations (2.2-2.4) in this paper. We propose SOCNN architectures based on the standard CNN presented by Leon O. Chua [2] with the 2nd-order polynomial inputs and extended bounded parameter assumptions. Furthermore, the problem of existence and stability of solutions is important in nonlinear differential equation defined and proposed in the equation 2.2. Thus, it is worth while to investigate the existence and stability of solutions for SOCNN. 2. Architecture of Second-Order CNN Consider an M*N of CNN, with M*N cells sorted into M rows and N columns. We call the cell on the ith row and the jth column: cell (i, j) and indicate it by C(i, j). Now let’s determine the neighborhood of a cell in CNN. Definition 1 The r–neighborhood of a cell C(i, j) in a CNN is defined: Where r ! N* (is positive integer number). Figure 1 shows 2 neighborhoods of the same cell (located at the center and shown shaded) with r=1; r=2 respectively. With r=1 we have (2r+1)2=32, r=2, have (2r+1)2 =52 neighborhoods. It is not difficult to show that the neighborhood system identified above exhibits symmetry property in the sense that: (N Nr rC(i, j) k,l) C(k,l) (i, j), C(i, j),C(k,l)∈ → ∈ ∀ ISSN 2354-0575 Journal of Science and Technology92 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 Figure 1. The neighborhood of cell C(i, j) defined by (2.1) a) r=1, b) r=2 respectively SOCNN architecture can be defined as follows: State equation (see architecture in Figure 3): ( , ) ( , ) ( , ) ( , ) ( , ) ( , ) ( ) 1 ( ) ( , ; , ) ( ) ( , ; , ) ( , ; , , , ) ( , ; , , , ) ( ) ( ) ij ij klC k l kl mnC k l C k l C m n kl mnC k l C m n dx t C x t A i j k l y t dt B i j k l u B i j k l m n u ukl A i j k l m n y t y t I R ∑= − + + ∑ ∑ ∑+ + ∑ ∑+ + (2.2) with A(), B() is feedback coefficient, input coefficient matrix respectively; R is linear resistor usually chosen to be between 1KX and 1MX; C is linear capacitor usually chosen to be 1nF; I is cellular bias or cellular threshold of the CNN cell. Output equation: (see Figure 2): ( ) ( ) ( ) ( ) 1 1 1 1 1 1 ij ij ij ij x t y t x x t x t  ≥  = − ≤ ≤ − ≤ − (2.3) Input equation: uij = Eij = constant 1 ≤ i ≤ M; 1 ≤ j ≤ N (2.4) Constraint equations: |xij(0)| ≤ 1; |uij| ≤ 1 (2.5) We can always be normalized to satisfy these conditions. Figure 2. Output Characteristics of CNN Parameter assumptions: (symmetry properties) A(i,j; k,l) = A(k,l;i,j); A(i,j; k,l; m,n)=A(i,j; m,n; k,l)=A(k,l; i,j; m,n)= = A(k,l; m,n; i,j)=A(m,n; i,j; k,l)=A(m,n; k,l; i,j) 1 ≤ i ≤ M; 1 ≤ j ≤ N (2.6) Remarks: a) All inner cells of SOCNN that have the same element values and structure. The inner cell C(i,j) is the cell in the operand: , ; ,A i j k l,C k l ^^ hh/ .ykl(t) has r2 1 2+^ h neighborhood connections, where r is defined in (2.1). In the operand: , ; ,B i j k l,C k l ^^ hh/ .ukl we also have r2 1 2+^ h neighborhood connections. So: ( , ) ( , ) ( ) ( )kl mnc k l c m n A(i, j; k,l; m,n)y t y t∑ ∑ in these two operands have r2 2 1 2+^ h neighborhood connections. The operand ( , ) ( , ) ( ) ( )kl mnc k l c m n A(i, j; k,l; m,n)y t y t∑ ∑ and the operand ( , ) ( , ) kl mnc k l c m n B(i, j; k,l; m,n)u u∑ ∑ have r2 1 2 2+^ h6 @ neighborhood cells for each respectively. Usually, we call these two operands proposed by us are second–order operands in the sense that they attach with the production of two feedback output signals .y t y tkl mn^ ^h h and the production of two input signals .u ukl mn. Finally we have 2 r2 1 2 2+^ h6 @ neighborhood connections to the cell C(i, j). b) The dynamics of SOCNN has two parts: one part includes the input operands: ( , ) ( )klc k l tA(i, j; k,l)y∑ with the feedback , ( )k ly t and ( , ) klc m n B(i, j; k,l)u∑ with input variable ukl . Other part (added by us): ( , ) ( , ) ( ) ( )kl mnc k l c m n A(i, j; k,l; m,n)y t y t∑ ∑ with production of two the feedback variables ( ) ( )kl mny t y t and ( , ) ( , ) kl mnC k l m n B(i, j; k,l,m,n)u u∑ ∑ with production of two the input variables kl mnu u . 3. Stability of Second-Order CNN Since Our SOCNN have feedback output signals, may be not stable to the system. One of the most effective technique analizing the stability properties of dynamic nonlinear system is Lyapunov method. Hence, let us first define a Lyapunov function for the SOCNN. ISSN 2354-0575 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 Journal of Science and Technology 93 Definition 2: We define the Lyapunov function E(t) of the SOCNN by scalar function: 1 1 2 ij ijkl (i, j) (k,l) (i, j)2 2R ij kl (i, j) (k,l) 1 ij mnkl (i, j) (k,l) (m,n)3 ij mnkl (i, j) (k,l) (m,n) E(t)= - A(i, j;k,l)y (t)y (t)+ y (t) - B(i, j;k,l)y (t)u + - A(i, j;k,l;m,n)y (t)y (t)y (t) - - B(i, j;k,l;m,n)y (t)u u - Iy ∑ ∑ ∑ ∑ ∑  ∑ ∑ ∑  ∑ ∑ ∑  ij (i, j) (t);∑ (3.1) Operands in the square braskets are proposed by us. In the following theorem, we will prove E(t) is bounded. This is the 1st condition for E(t) to become Lyapunov function. Theorem 1: Function E(t) defined in (3.1) is bounded by: ( )max E t Emax≤ (3.2) max (i, j) (k,l) (i, j) (k,l) 1 E = A(i, j; k,l) B(i, j; k,l) 2 + +∑ ∑ ∑ ∑ ( ) (i, j) (k,l) (m,n) (i, j) (k,l) (m,n) 1 1 MN I A(i, j; k,l; m,n) 2R 3 B(i, j; k,l; m,n) ++ + +∑ ∑ ∑ + ∑ ∑ ∑ (3.3) Proof: Following to the definition of E(t) in (3.1), we have: ( ) 0E t ≤ (3.4) Since y tij ^ h, uij are bounded as claimed in (2.5) we have: ( ), ( ) max (i, j) (k,l) (i, j) (k,l) (i, j) (k,l) m n (i, j) (k,l) (m,n) 1 E t 2 1 I 2R E = A(i, j; k,l) B(i, j; k,l) 1 MN A(i, j; k,l; m,n) 3 B(i, j; k,l; m,n) ≤   + +    + +∑ ∑ ∑ ∑ + ∑ ∑ ∑ + ∑ ∑ ∑ 1≤ i, k, m≤ M ; 1≤ j, l, n≤ N (3.5) Depending on (3.3) and (3.5) equation that E(t) is bounded, but we can also prove that it is a monotone decreasing function. Theorem 2: The scalar function E(t) defined in (3.2) is a monotone decreasing function (or minus– defined function), that is ( ) 0 dE t dt ≤ (3.6) This is the 2nd condition for E(t) to become Lyapunov function. Proof: To differentiate E(t) in (3.2) with to time t, take the derivate of on the right side of (3.2) with respect to xij(t): ij ij (i, j) (k,l) ij (t) ij ij ij ij (i, j) (i, j)(t) ij ij ij kl (i, j) (k,l) ij (mn) dy (t) dx (t)dE(t) = A(i, j; k,l) + dt dx (t) dt dy dx (t) dy (t) dx (t)1 ij+ y (t) - I R dx dt dx dtx ij dy (t) dx (t) - B(i, j; k,l) u dx (t) dt dy A(i, j; k,l; m,n) ∑ ∑ −∑ ∑ −∑ ∑ − ∑ ij ij kl mn (i, j) (k,l) ij ij ij kl mn (i, j) (k,l) (mn) ij (t) dx (t) y y (t) dx (t) dt dy (t) dx (t) - B(i, j; k,l; m,n) u u dx (t) dt ∑ ∑ ∑ ∑ ∑ (3.7) Here, we use the symmetry properties of (2.6) to obtain (3.7). From the output function (2.3), we obtain the following relations: dx t dy t 1 0ij ij =^ ^ h h ( if |xij(t)| < 1if |xij(t)| $ 1 (3.8) and when |xij(t)| < 1, yij(t) = xij(t) Figure 3. Model of Second-Order CNN ISSN 2354-0575 Journal of Science and Technology94 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 According to our definitions of SOCNN, have: A(i, j; k, l) = A(i, j; k, l; m, n) = B(i, j; k, l)= = B(i, j; k, l; m, n) = 0 for C (k, l) g Nr(i, j); C (m, n) g Nr(i, j) It follows from (3.7) and (3.8) with the parameter assumptions: ( ) ( )( ) ( ) ( ) ( ) ( ) ( ) ( ) 1ij ij ij ij R(i, j) ij kl kl C(k,l) N (i, j) C(k,l) N (i, j)r r kl mn C(k,l) N (i, j) C(m,n) N (i, j)r r C(k,l) N (i, j) C(m,nr dy t dx tdE t 1= - x t x t dt dx t dt R B(i, j;k,l)u A(i, j;k,l)y t A(i, j;k,l;m,n)y t y t ∈ ∈ ∈ ∈ ∈ +  − +∑  +∑ ∑ + ∑ ∑ + ∑ ∑ kl mn ) N (i, j)r B(i, j;k,l;m,n)u u ∈    = 2 ( ) 0 (3.9) dx tijC dtij   ∑     − ≤ (3.9) Remark: Colollary: For any given inputs uij and any initial states xij(t) of the SOCNN, we have: limE t t = "3 ^ h constant (3.10) and: lim dt dE t t = "3 ^ h 0 (3.10a) Proof: From theorems 1 and 2, E(t) is bounded monotone decreasing of time t. Therefore, E(t) converges to a limit and its derivate converges to zero (0). Theorem 3: All states xij(t) in SOCNN are limitted for all time t >0 and the xmax can be computed by: ( ) ( ) 1 ( , ) ( , ) ( , ) ( , ) ;( , ; , ; , ) ( , ; , ; , ) (1 i M; j N ) C(k,l) N (i, j)r C k l Nr i j C m n Nr i j x = R max A(i, j;k,l) + B(i, j;k,l) 1max R I A i j k l m n B i j k l m n ≤ ≤ ≤ ≤ ∈ ∈ ∈ +  ∑   + + +∑ ∑   (3.10) Proof: From formula (2.1), we can rewrite the kinetic equation of the cell as follow: ( ) 1 ( ) ( ) ( ) ( ) ( ) ' ij ij ij ij ij ij dx t x t f t g u dt RC h t p u I = − + + + + + + (3.11) where: I C I=l ( , ) ( , ) ( , ) ( , ) ( , ), ( , ) ( , ) ( , ), ( , ) ( , ) 1 ( ) ( , ; , ) ( ) 1 ( ) ( , ; , ) 1 ( ) ( , ; , ; , ) ( ) ( ) 1 ( ) ( , ; , ; , ) ij kl C k l Nr i j ij kl C k l Nr i j ij kl mn C k l C m n Nr i j ij kl mn C k l C m n Nr i j f t A i j k l y t C g t B i j k l u C h t A i j k l m n y t y t C p u B i j k l m n u u C ∈ ∈ ∈ ∈ = ∑ = ∑ = ∑ = ∑ with u Eij xMN1= 6 @ indicates an M*N dimentional constant input vector. Formula (3.11) is a 1st–order ordinary diffential equation and its solution is given by: x t x 0ij ij=_ _i ie RC1- + e RCt t 0 x- -_ i # f tij +_ i8 g tij +_ i + p tij +_ i q t I dij x+ l_ i B (3.12) It follow that: ( ) (0) t RC ij ijx t x e − = + ( ) 0 '( ) ( ) ( ) ( ) tt RC ij ij ij ije f h g u p u I d τ τ τ τ − − + + + + +∫    ( ) 0 ( ) (0) '( ) ( ) ( ) ( ) t RC ij ij tt RC ij ij ijj x t x e e f h g u p u I di τ τ τ τ − − − ≤ + + + + + +∫    (0) t RC ijx e − ≤ + ( ) 0 ( ) 0 '( ) ( ) ( ) ( ) '(0) '(0) tt RC ij ij ijj ttt RC RC ij ij ij ij ij ij ij ij ij ij e f g u h p u I di x e F G H P I e d x RC F G H P I τ τ τ τ τ τ − − − −− +  + + + + ≤∫     ≤ + + + + + ∫    ≤ + + + + +   (3.13) where: maxF f tij t ij= _ i C 1 # , ,C k l Nr i j!_ _i i / , ; ,A i j k l_ i max y t t kl _ i (3.14) G max g tij u ij= _ i C1# ,C k l_ i/ , ; ,B i j k l_ i max uu kl (3.14b) H max h tij t ij= _ i # C 1 # , , ,C k l C m n_ _i i / , ; , ; ,A i j k l m n^ h max y t t kl ^ h max y t t mn ^ h (3.14c) ISSN 2354-0575 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 Journal of Science and Technology 95 P max p uij u ij= _ i # C 1 # , , ,C k l C m n_ _i i / , ; , ; ,B i j k l m n^ h max u u kl max u u mn (3.14d) Since xij(0) and uij satisfy the conditions in (2.5), while |yij(t)| satisfies the conditions: ( ) ,y t t1ij 6# In view of characteristics function (2.3), it follows from (3.13) and (3.14): |xij(t)| # |xij(0)| + R , ; ,A i j k l ( , )C m n ^ h9 / max|ykl(t)| + I l + , ; ,B i j k l ,C m n ^^ hh/ max ukl + + ,, C m nC k l ^^ hh // , ; , , ,A i j k l m n^ h max y t t kl ^ h max y t t mn ^ h + C mnC kl ^^ hh // , ; , ; ,B i j k l m n^ h max u u kl +max u u mnC (3.15) xmax = 1 + R|I| + ,, C m nC k l ^^ hh // , ; , , ,A i j k l m n^ h + + ,, C m nC k l ^^ hh // , ; , ; ,B i j k l m n^ h (3.16) Because xmax depend on the time t and the cell C(i,j), 6 (i, j). we have: max x xmax t ij # (3.17) For any SOCNN, the parameters R, C, I, A(i, j; k, l), B(i, j; k, l), A(i, j; k, l; m, n); B(i, j; k, l; m, n) are boundary constants. So that, the bound on the states of the cells, xmax is finited and can be calculated by equation (3.10b). 4. Simulation of the Second-Order CNN In this section, we will present an example to illustrate how the SOCNN described in section 2 work. Suppose we have the networks 4*4 (N=4, M=4) with r=1, we have neighborhood system (see Appendix). Data for standard CNN and SOCNN feedback and input matrix are choose equal each other and similar to [2], that as: A(i, j; k, l)=A(i, j; k, l; m, n)=A= 0 1 0 1 2 1 0 0 0 R T SSSSSSSS V X WWWWWWWW input matrix: B(i, j; k, l)=B(i, j; k, l; m, n)=B B= . . . . . . . . . 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 0 5 R T SSSSSSSS V X WWWWWWWW ; I= 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 R T SSSSSSSSSS V X WWWWWWWWWW Parameters for standard CNN and SOCNN 4*4 cells (N=4, M=4); C=10-9 F; R=103X ; cellular bias I; Initial state values: x(t=0) = All the data are being used to simulate on Matlab (2014) for standard CNN and SOCNN with 4*4=16 state variables X for purpose to compare. Figure 4a and 4b are the transient behaviors of the standard CNN and SOCNN respectively, that take two cells X11, X22 for the examples (in total 16 transient behaviors). From Fig.4a and 4b we can remark that: i) CNN and SOCNN reach to the equilibrium stability state after some time. ii) The transient behaviors of SOCNN monoton-increasing to the equilibrium state, no has overshoot. In standard CNN presented by Leon O. Chua [2] the transient behaviors are oscillated and maximum percent overshoot (see Figure 4a). In this case it reaching (Cmax- C3 )/ C3 =(3-2)/2=50%) with Cmax, C3 is maximum peak and final value respectively of the response curve measured from the cellular state. iii). The reserve or the stable strength of SOCNN higher standard CNN (that is: (XSOCNN=23)>(XCNN =3)). Figure 4a. The transient behaviors of standard CNN [2] ISSN 2354-0575 Journal of Science and Technology96 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 Figure 4b. The transient behaviors of SOCNN 5. Conclusion In the paper, we have tree contributions: i) We have presented the architecture SOCNN with the 2nd-order polynomial inputs and extended bounded parameter assumptions. ii) We proposed the conditions for the existence and stability of solutions of CNN by choosing appropriate Lyapunov function. iii) We have also used Matlab Simulink software to illustrate and compare some dynamic properties of simple CNN with SOCNN. Some advantages of SOCNN compared to CNN are presented in the section 4. There are many theorical and practical problems to be solved in our future research on this subject, for example, learning problems, associative memory of SOCNN. Nevertheless some rather impressive and promising application have already been archieved and was reported in paper [8]. References [1]. Zuda Huang, Lequn Peng, Min Xu. “Anti-Periodic Solutions for High-Order Cellular Neural Networks with Time-Varying Delays. Electronic Journal of Differential Equations”, 2010, Vol. 2010, No. 59. [2]. Leon O. Chua and Lin Yang “Cellular Neural Networks: Theory”, IEEE Trans. on Circuits and Systems, October 1988, Vol. 35 No 10. [3]. Angela Slavova “Cellular Neural Networks: Dynamics and Modeling”, Kluwer Academic Publishers, 2003. [4]. Hoan Nguyen Quang, “On Stability of Hopfield Neural Networks and Application Ability in Robot Control”. PhD. Dissertation, 1996. [5]. Valerio Cimagalli and Marco Balsi. “Cellular Neural Networks: A Review”. Proceedings Italian of 6-th Workshop on Parallel Architecture and Neural Networks. Vietri sul Mare, Italy, May 12-14, 1993. Wold Scientific (E.Caianiello, ed.) [6]. Pier Paolo-Civalleri and Marco Gilli. “On Stability of Cellular Neural Networks” Journal of VLSI Signal Processing 23, 1999, pp. 429-435. [7]. Tamas Roska. “Cellular Wave Computers for Brain-Like Spatial-Temporal Sensory Computing”. IEEE ” Circuits and Systems Magazine 1531-636X/5520.000c, 2005. [8]. Nguyen Tai Tuyen, Nguyen Quang Hoan, “An Application of Multi-Interaction Cellular Neural Network on the Basis of STM32 and FPGA”, International Journal for Research in Applied Science & Engineering Technology, January 2018, Volume 6 Issue I. [9]. Makoto Itoh, Leon o. Chua. “Star Cellular Neural Networks for Associative and Dynamic Memories. International Journal of Bifurcation and Chaos, 2004, Vol. 14, No. 5, World Scientific Publishing Company, pp. 1725–1772. KIẾN TRÚC VÀ ỔN ĐỊNH CỦA MẠNG NƠ RON TẾ BÀO BẬC HAI Tóm tắt: Trong bài báo này, chúng tôi nghiên cứu và đề xuất: i) kiến trúc mạng nơn tế bào bậc hai dựa trên mạng nơ ron tế bào chuẩn của Leon O.Chua với các đầu vào ngoài, đầu vào phản hồi và với các điều kiện ràng buộc giả định. ii) Các điều kiện cho tồn tại và ổn định các nghiệm của mạng SOCNN được trình bày bằng cách tìm các hàm Lyapunov thích hợp. iii) Các kết quả mô phỏng được tính toán trên phần mềm Matlab. Từ khóa: Mạng nơ ron tế bào bậc hai, hàm Lyapunov, tính ổn định. ISSN 2354-0575 Khoa học & Công nghệ - Số 27/Tháng 9 - 2020 Journal of Science and Technology 97 Appendix. Array 4*4 Second-Order CNN Simulation on Matlab