Coactive Neural Fuzzy Modeling

ANFIS I CANFIS

==F
+ w2c2)

61% 1 0 F-Y
Arroslsllon

+ w3)
2. CANFIS Architecture

1. Introduction

Adaptive iic~iiro-fuzqiiiodds clijoy both illally 2.1. Comparison with a simple back-prop NN
of the advantagos claiiii~tlfor Y S s a i d FS's linguis-
tic interpretability, which a1low.s prior k ~ i o w l d g e
to be c~nl~eddecl in tlivir coilst riiction iiiitl allows
the results of lrarliiiig to l>c possi1)ly iuidcrstoocl.
easy iniljleiiieiitatioii of suc4 i l ~ie~iro-fiizzp ltiodel
by modifying an avai1al)lc 1i;trt~l)onetl1,ack-prop
program. MJe contrast t lie iieiiro-fii
well-discussed black-l)os XNs. TVIIOS~~ vi-ciglit, coeffi-
cients are just, iieunieric cmiiirac.tioli st.rwgt,lis I)llt,
not, good laiiguage: tlie liiddm l a p r of tlie sim-
ple back-~jropNN is taiitiiiiioiiiit to tlie coiisecineiit
layer of CANFIS. P n t tiiig iiior(' liiddw iiotlcs in the
EN is equivalent to adding iiiorv rii1t.s in CXNFIS.
The NWs w e i g h h t w w i i tlir oiitput layer and t.lie
hidden layer correspond t o niciiibc~rsliipd u e s on
fuzzy association l a y r iii CASFIS. This compari-
son emphasizes the iiisidc transpiireiicj- of C-ANFIS.
From an archit,cct.nrid poiiit of T-iw. CXYFIS's
powerful capability stciiis froiii ~~atterii-tlel~eiideiit,
weights between the constqueiit layer and the fuzzy
association layer. Nanielj-. iiienihership values cor-
respond to those tlyiianiicdly changmhlc w e i g h
that depend on input, pat,teriis. In cont,rast.. t-he
simple back-prop EN tries to find one specific set
of weights coiiiiiioii to all t.raiiiiiig pat terns: in other
words, the weights are iisc~lin i~ global fitshion. One
-
typical criticisiii of t,lie siiiiple hack-prop ?T%sis in
globally updating weight coefficients. On t,he ot.her
hand, CANFIS and a Radial Basis Function Yet-
work (RBFN) are locally tuned [13. 191. With bell-
shaped fuzzy membership fiiiictions (XIFs). CAN-
FIS has a niechanisiii whereby it. caii pfoduce a
center-weighted response' to sinal1 recept.ire fields,
localizing t,he primary iiiput escit,ation. In this
sense. CANFIS can he functionally equivalent, t o an
RBFN [6]. Just, as aii RBFN enjoys it.s quick con-
vergence capability, ANFIS also can evolve t.o recog-
nize some feature in a training tlat,aset. w r y quickly
compared with siiiipk back-propagation KNs.

2.2. Non-linear Rule

In pursuit of a truli &q)tive nc.twork. there
should be no constraints on iiciiron functions. Re-
placing a neuron function with a noiiliiirar neuron
function at the coiiseqnciit Lyier may b e able to
improve the perfoiniant c.

2.2.1. Sigmoidal Rule

Suppose we have a sigiiioidal function it5 a. iieuroii
function 2 in Figure 1 (iibow). Then W P have it
nonlinear coliserpent. C1 :
1
Cl=
1 + esp[-(]'l s + '11 I- + )'I ,].
In this case, we h a w a 4gnioitlal rulc,.

2.2.2. Neural Rule

Furthermore. when etic.11 ( onsrqii(~iitis rciilized by
an NW, wc have nc~uralrider as illustrated in Fig-
ure 2 (above). Note that mlieii the fonr cwisecliient
(q) CANFIS wilh 2 linear rules (0)CANFIS wlth 2 linear rules
FJSS)i i i o d ~ l 110
, iiiodifia1)lc l)iiriLiil('t('r\ iir(' ~ I i r t r t ~ l
by tlic juxtaposed .ANFISs. Saiiic~ly.(tiL(.li zZNFIS !,
at epoch=O
Desaed O"lpu1
.,
,, .#
: !
lias aii iiitlepeiidriit set of fi riilvs. It inay be .,., : !
, ,
hard to rcdize c e r t t h ( ~ r r 011s I)c~t\vcY%out-
.. ,.
0 ,

puts. Ai1 additioiiJ c o i i c ( ~ r i ilies iii tlic i i i i i i i l > c ~of , ,

adjustahl(~paraiiietws. wliicli (hasti( ally iiicrci(istscs I

as outputs iiicwasc. .................. . ( MFZ

................
Aiiot1ic.r way of goiivr:ttiiig iiiiiltipk oiitpiits is ..........
to iiiaiiitaiii the saiii(1 u k d v i i t s of fii/Lp rules: c,. . , ,. . ~ .' , .. J_ h.

Figure 2 (a1)ovc~)visuali/;tts thi\ cwicept. In short. (f) CANFIS with 3 linear rules ( R ) CANFiS with 3 linear rules
. rulcs are coiistriirtcd with slinrcd iiiciii1)t.r- I ,

values t o exprtw po41)1(~corr(~htioiis1)etwer.n

011t put s .
Filrtli(~r1iiorc.W ~ I C I I ( oii\(~(ii1~~iit1 ) c ~ ti 5 iLre also

correlated; that is, wli(w two iiriual coiiseyiimts,

L'Nearal Rulel" and bbNenralRtrlej" ill'? fused into
oiw NN (Local Expert I ). m t l *'nTeural Rule;?"
and "Neural Rule," arc' fiiscd iiito aiiotlier NN (Lo-
cal Expert LVN2).w~ liilvc a ( oiistiuctioii siiiiilar
t o a typical iiiodular iic,twork as illiistratcd in Fig- (S) NN with UHU = 3
ure 2 (below); tlie ontpiits of two loc.al cspcbrt NNs 1

are iiicdiated by aii iiitclgratiiig uiiit (typically a

gating network). 1\Iaiij- iiiodiilnr iirtwork models
i
,
were discussed iii [15, 16, 14, IO. 3. 91; tlicir design
procedure detailed in [ l G ] is 5troiigly based 011 a n I

independent traiiiiiig sclieiiie w1ierel)y aiitecedeiit 1

parts aiid coiiseclueiit parts iLr(' traiiictl individually,
aiid then put togetlirr. <4iiotlicr training approach
is t o traiii both aiitcwxlciit and coiisccpciit parts
concurrently [ l , 4, 8. 111.
We sliall let t lic following coiicrctc csaiiiple fur-
tliw clarify CANFIS's leariiiiig c y a l d i t i e s .

3. Simulation example (N-shape prob-

lem)
each has itlciitity fiuic-tioii5 iii (D).
This siiiiiilatioii csaiiiple is quite siiiipk: fit tiiig In tlik siiiiiilatioii, the follomiiig hcll-sliap(~1
ail iv-shaped letter that lias t v o (oriiers: <L poiiited h4Fs w'cr~iisc~l:
top left-hand roriior ant1 a roniidetl bottoiii right-
liaiitl corner. This lcttrr is sliowii as <L dxslied liiie
iii Figures 3 (q), (r).
To see how adaptivo ( apa1)ilitg tlepeiids oii ar-
chitecture itself. four siiiiple back-prop NNs aiid
five CANFIS iiiocltds (-4)-(E) w r t ~tcastcd usiiig
the sruiie coiiwiitioiial gratlieiit dwriit (GD) back-
propagation algoritliin iiictliotl witli n fiscd iiioiiieii-
tuiii (0.8) and a siiiall fixcd lrariiing rate. The
five iiiodels are (A) CASFIS with linear rules (i.e.,
ANFIS), (B) CANFIS with sigiiloidal rules, a d
(C,D.E) with iiet1riLl rnlrs. CXNFIS (E) has sig-
iiioidal functions t o gciicriLt(' filial outputs at f11zzy
associatioii layer wlicrea5 C' AXFISs (C) :tilt1 ( D )
have ideiitity fii1ictioii.J a t fii . associatioii 1ayc.r.
Tlie reSdt5 are sliowii iii Tal)lc 1 aiid 2 . The
diffcreiice between ( C ) i ~ i i d( D ) Irsidm iii iitiiroii
fiiiictions in t lie oatpiit l a y r of iiriirnl coiiseqiieiits.
More precisely, each iiviiral cwiiscyncwt lias sig-
iiioitlal fuiictions iii its ontprtt l a ~ r iii r (C) while
(b) Initial.Outp!its
,,( i ' ' '
I Klli ,
'i
# of hidden u n i t s 3 5 7 3x3
checking error 9.91 4.99 '2.84 2.05
training error 8.61 1.75 '2.73 1.84
Dammeter # 1(J 16 2'2 2'2

Klih .

(6)Final Outputs 8
iI r U",. ,

:\p/
I
/ '

MI? ,

Fig. 4: Trajectories of t,he cc>iitrr ponii.ioiih of t.hree hIFs (left)

a n d a result of the N-sh;r.pe probleiii obta.iiiec1 by ANFIS with
three rules (right) based 0 1 1 !.lie G D inethotl alone. Though
actual output,s (solid line) WCYY almost. pt~rfrctlyfit.t.ed into
the desired N-shape (dashed line), the question is how t,o
ext,ract rules.

CANFIS with two/tliree rules is able t~ocapture

the peculiarity of tlie X-shape as shown in Figures 3
(Q) and (R), fitt,ing very well both the pointed
and round corners of t.he E-shape. +A sgst.ematic
t,rainiiig procedure leads t,lie hell-shaped XIFs t,o the
results presented iii Figures 3 (Q) aiid (R,). T h e y
never cease to amaze us iii t.liat. t,lieir neat. shapes
can metamorphose into uiiexpctted ones: manually-
tuned MFs may not niatcli tJiciii.
Due to tShediscontiiiiiity of tlie left-hand corner
of tlie N-shape, NNs with a siiiall iiuiiiber of hid-
den neurons (three or four neurons), possessing sig-
moidal neuron functions. (lo not evolve to piecewise
fit, the pointed coriier sliowii in Figure 3 (S) in
spite of having a coii1p;lra~)leriiiiiiber of adjustable
parameters (compare Tables 1 aiid 2 ) . They never
reach the fitt,ing level of Figuw 3 ( R ) within a preset.
it,eration limit (200,000).
This exaniple has rthforced the strei1gt.h of
CANFIS with the bell I\IFs in tlie convergence ca-
pacit,y. It also prcwiits it.s tra1ispareiic.y of t.he
learning result,s: drt,ails arc. c~sl)lainetliii t.he next
section.

4. Analysis of results
4.1. Limitations of Interpretability
W ~ W I <three rulcs aic iiitrot1iicNI. it i5 oliseivect
4.2. Evolution of antecedents (MFs)
that two MFs (MF2, H F 3 ) tiansit h t l i niid forth
dt the beginning of their cvdiition if struggling to Aiiot1ic.r iiitcw\tiiig 1)oiiit in tlir rrsult i\ tliitt UF\*
find coiiifortahle nit lies ( ~ e Figure
c -1 ( k f t ) ) . After octxpl-tncitis tlcpciid on traiiiing inctliotlh. Tliv
origilial ANFIS enjoys i t s 1iyl)rkl lwriiiiig proce-
durc haset1 oil h t l i a licwristic, a&ptivc s’tep size
strategy and a coni1)inatioii of tlir. gridieiit de-
ceiit (CiD) nictliod iilitl the. lCit\t-h(llii\rChvstiniation
(LSE) tvlicreby. iii tlic for\47iLrtl pass, coiiseyueiit
paraliicters ai(’ updated 1)i~s.sccloii LSE iising lieu-
r o ~ ioutputs, a ~ i t in
l the I)R(.li\viLrd1 ) i i s ~iLlitecedelit
.
paraiiieters are updated l)a-se(l oii tlir GD method
using error sigmls.
111this simulation. wli(w origiiial hell N F s mere
iiitroducrd, CANFIS wit11 thc GD iiiethod alone
unespectedlp rorivcrgctl fastw tliaii CXSFIS with
t hc>liybrid learning procedurc~. IIoreovcr. CANFIS
based oii t lie Iiyl>ridlcariiiiig proccdrirc> did not fit
the N-shape very woll as Figures 6 (\-),(TI‘) show,
while CANFIS with tlic GD iiietliotl aloiie recog-
nized the features of the ?rT-sliq>ewell as showii in
Figure 4 (right). In mails mws. tlic hybrid learning
algorithm works better t l i m tlic GD inctliod alone,
but it may be worth cmisitlcring possi1)le reasons
for this observation. The 1iyl)ritllearniiig procedure
lmsically predominates whtw intuiti~cl~-positioiied
MFs do iiot iieecl to cvolv(. w r y iiinch. Initially,
LSE inay specialize coiiscqucnts to a great extent,
which niay preveiit A l F s from erolr-ing. LSE can
find certain d u e s of (miisecpwts that have min-
inial errors with tlie currelit h l F setup, Imt after
updatiiig coefficients, it iiiay c>iitl u p losing its way
to a global iiiininiuni.
Without a prior kiioml(~lgc,initial setups of
MFs’ parameters (ant ccwlcnts) aiitl consequents
inay iiot be perfect; to trust the initial guess may
turn out to be a possilAt1 ol)stac.le iii obtaining bet-
ter results. Note that the rc.snltant MFs in Fig-
ures 3 (Q),(R) inay not miitch ally initial gne
Let us now asunw tliiit IW arquirc two coiise-
queiits froin data sets. 111 this cas(’. is i t a good
idea t o stick to thein’! Figirrc. 5 4ion.b two re-
siilts obtained iiriiig two fixed conscqucn ts. which
purposely coiiicidc witli two side lines of tlie N- 4.4. Evolution of Consequents (Rules)
shape; Figure 5 ( A ) , rcsiiltiiig froni (a) suggests
that clinging to thow two coiistqneiits niay not he Tlierc. iiiiist I x soiiic’ optiiiial coiiibiiiations of
urefiil in ohtaining a good fit. On tlita other hand, slia1)eb of IIFs and foriiis of conscquents. Figure T
Figure 5 (B) suggests that AXFIS is al)k to adapt shon.5 S0111(’ of tllrlll.
to the N-shape to soiiie c.stc.iit ( ~ c wli(~ii n the ini- Iiiterrstiiigly3 oittputs of tlw atlaptcd (~)iis(i-
tial MFs are poorly set up as iii (1)) wlierc~tlicre is quciits tlrpic.tetl in Figure 7 (A)-(D) cwl up Iwiiig
no intersection betwwli tlie two iiiotlificd bell LIFs. diff(wnt iLli(1 f;lr froni tlir dcsired N-sliapfl. This is
As in (B). the left-hand 1)ase of JIF1 rcacliecl the because IIY.liar-cl triiiiicd hoth aiiteccdciit arid colis(’-
pointed coriier of the N-slia1x’. aiid therefore Jf Fl qimit parts siiiiiiltaiieously. Thus,t w d i outpiit cloc.5
evolved to a shape differc~ntfrom tlw M F l as in iiot haw to fit tlic tlcsircd N-shape: tlit. coiiil,iiic~tl
Figure 3(Q). outpnts fit the N-shiLpr~.
These results confirni that C‘AKIFS can grasp
the peculiarity of a givcii (lata set becmse of its 5 . Conclusion
adaptive capability.

4.3. Asymmetric MFs

Figures 6 (X),(Y) show rc.su1t.s obtaiiicd based
on the hybrid learniiig algorithiii using the following
Fig. 7 : Different outputs o r optiinized ~ o ~ i s ~ q i i e iafter its
training phase; The niotlels are C'ANE'IS \\,it11 linear rules
( A ) , sigmoidal rules ( B ) . aiicl iieuriil rules ( i ' )a i ~ t (l D ) .

experiments. Some of the disrussetl CAKFIS niocl-

els are test,ed and disciissed by appliration to a more
complicated problcin 111, 121.
We have also described several probleiiis faced in
designing neuro-fuzzy tcins for iise in practical
environments. Thong1 le adaptive syst.elns may
suffer from the difficultic~s.they most likely ill out-
perform insnually designed tenis: tlie adaptive
learning procedure snrrly helps.
.4ut,omat.ic rule ext,ractioii is a useful aspect of
adaptive neuro-fuzzy iiiotlels. Yet, we slioulcl pay
at.t>entiontro tlie 1iiiiitat.ion lwliind niany successful
report,s. Acquired rules iiiay sonietiiiies lie hard t ~ o
understand. In t,liis context. it may be fiitltileto
pursue int,erpret,ahilit,yof vwigllt rocfficielits froin a
fuzzy logic standpoint.
To obtain higlier precision wit.liii1 a fixed
aniount of coniputat,ion. and t o limit the number
of MFs so we can also rct.aiii interprct.abilit,y, we
may const,ruct nonlinear rules siicli as n e n r d rides.
Or we may prefer to choosc a niorc sophist.icat,ed
MF such as an asyinnietric h4F. =Ilso. inanipulatiiig
neuron functions iiiay I i a w it lwndicid Pffect on
perforniance enhaiiceinent . Tlic qiicstion is what
coinbinations of thcni are bcst. Exploring this idea
furt,lier inlist, he a good small step ton-arc1 finding
an ideal iiiodel in the true seiise of ail "adaptive
ne t,work."

References
[I] "Modular Networks." llayliiii, S i l n o r i Ncliral N e t i v o r l i s :
a comprehensive fountlat ioii rlinl)tc~r 12.. klarmillan