Sensitivity analysis of multilayer neural networks
| dc.contributor.advisor | Cloete, I. | en_ZA |
| dc.contributor.author | Engelbrecht, Andries Petrus | en_ZA |
| dc.contributor.other | Stellenbosch University. Faculty of Science. Dept. of Mathematical Sciences. | |
| dc.date.accessioned | 2012-08-27T11:34:29Z | |
| dc.date.available | 2012-08-27T11:34:29Z | |
| dc.date.issued | 1999-12 | |
| dc.description | Dissertation (Ph.D.) -- University of Stellenbosch, 1999. | |
| dc.description.abstract | ENGLISH ABSTRACT: The application of artificial neural networks to solve classification and function approximation problems is no longer an art. Using a neural network does not simply imply the presentation of a data set to the network and relying on the so-called "black-box" to produce - hopefully accurate - results. Rigorous mathematical analysis now provides a much better understanding of what is going. on inside the "black-box". The knowledge gained from these mathematical studies allows the development of specialized tools to increase performance, robustness and efficiency. This thesis proposes that sensitivity analysis of the neural network output function be used to learn more about the inner working of multilayer feedforward neural networks. New sensitivity analysis techniques are developed to probe the knowledge embedded in the weights of networks, and to use this knowledge within specialized sensitivity analysis algorithms to improve generalization performance, to reduce learning and model complexity, and to improve convergence performance. A general mathematical model is developed which uses first order derivatives of the neural network output function with respect to the network parameters to quantify the effect small perturbations to these network parameters have on the output of the network. This sensitivity analysis model is then used to develop techniques to locate and visualize decision boundaries, and to determine which boundaries are implemented by which hidden units. The decision boundary detection algorithm is then used to develop an active learning algorithm for classification problems which trains only on patterns close to decision boundaries. Patterns that convey little information about the position of boundaries are therefore not used for training. An incremental learning algorithm for function approximation problems is also developed to incrementally grow the training set from a candidate set by adding to the training set those patterns that convey the most information about the function to be approximated. The sensitivity of the network output to small perturbations of the input pattern is used as measure of pattern informativeness. Sensitivity analysis is also used to develop a network pruning algo-rithm to remove irrelevant network parameters. The significance of a parameter is quantified as the influence small perturbations on that parameter have on the network output. Variance analysis is employed as pruning heuristic to decide if a parameter should be removed or not. Elaborate experimental evidence is provided to illustrate how each one of the developed sensitivity analysis techniques addresses the objectives of improved performance, robustness and efficiency. These results show that the different models successfully utilize the neural network learner's current knowledge to obtain optimal architectures and to make optimal use of the available training data. | |
| dc.description.abstract | AFRIKAANSE OPSOMMING: Die toepassing van kunsmatige neurale netwerke om klassifikasie- en funksiebenaderingsprobleme op te los, is nie meer 'n kuns nie. Die gebruik van 'n neurale netwerk impliseer nie meer bloot die toepassing van 'n data stel op die netwerk, en die verwagting dat die "swart boks" - hoopvol akkurate - result ate lewer nie. Omvattende wiskundige analises verskaf nou 'n baie beter begrip van wat binne die "swart boks" aangaan. Die kennis wat van hierdie wiskundige analises gewin is, laat die ontwikkeling van gespesialiseerde hulpmiddels toe om prestasie, robuustheid en effektiwiteit te verbeter. Hierdie tesis stel voor dat sensitiwiteitsanalise van die neurale netwerk afvoer funksie aangewend word om meer oor die inner werking van multi-vlak vorentoe-voer neurale netwerke te leer. Nuwe sensitiwiteitsanalise tegnieke word ontwikkel om die kennis vervat in die gewigte van netwerke te ondersoek, en om hierdie kennis aan te wend binne gespesialiseerde sensitiwiteitsanalise algoritmes om sodoende veralgemeningseienskappe te verbeter, om die kompleksiteit van leer en model kompleksiteit te verminder, en om konvergensie eienskappe te verbeter. 'n Algemene wiskundige model is ontwikkel wat gebruik maak van die eerste orde afgeleides van die neurale netwerk afvoer funksie met betrekkiIig tot netwerk parameters om die effek van klein versteurings aan hierdie netwerk parameters op die afvoer van die rietwerk te kwantifiseer. Hierdie sensitiwiteitsanalise model word dan gebruik om tegnieke te ontwikkel om besluitnemingsgrense op te spoor en te visualiseer, en om te bepaal watter besluitnemingsgrense word deur watter versteekte eenhede geimplementeer. Die algoritme om besluitnemingsgrense op te spoor word dan aangewend om 'n aktiewe-leer algoritme vir klassifikasie probleme te ontwikkel, wat leer deur gebruik te maak van slegs daardie patrone wat naby besluitnemingsgrense Ie. Gevolglik word patrone wat min inligting bevat in verband met die ligging van besluitnemingsgrense nie vir leer aangewend nie. 'n Inkrementele leer algoritme is ook ontwikkel vir funksiebenaderingsprobleme waarin die leerversameling inkrementeel vanuit 'n kandidaat leerversameling gegroei word deur daardie patrone by te voeg wat die meeste inligting vervat oor die funksie wat benader word. Die sensitiwiteit van die netwerk afvoer tot versteurings in die toevoer patroon word gebruik as 'n maatstaf van die informatiwiteit van daardie patroon. Sensitiwiteitsanalise is ook gebruik om 'n algoritme te ontwikkel wat irrelevante parameters van die netwerk snoei. Die belangrikheid van 'n parameter word gekwantifiseer as die invloed wat klein versteurings in daardie parameter het op die afvoer van die netwerk. Variansie analise word gebruik as heuristiek om te besluit of 'n parameter gesnoei kan word al dan nie. Omvattende eksperimentele bewyse word verskaf om te illustreer hoe elkeen van die sensitiwiteitsanalise tegnieke wat in hierdie tesis ontwikkel is, die doelwitte van verbeterde prestasie, robuustheid en effektiwiteit adresseer. Hierdie resultate toon aan dat die onderskeie modelle suksesvol gebruik maak van die neurale netwerk se huidige kennis om optimale argitekture op te stel, en om optimaal van die beskikbare leerdata gebruik te maak. | |
| dc.format.extent | 255 pages : ill. | |
| dc.identifier.uri | http://hdl.handle.net/10019.1/51420 | |
| dc.language.iso | en_ZA | |
| dc.publisher | Stellenbosch : Stellenbosch University | |
| dc.rights.holder | Stellenbosch University | |
| dc.subject | Neural networks (Computer science) | en_ZA |
| dc.subject | Sensitivity theory (Mathematics) | en_ZA |
| dc.subject | Dissertations -- Computer science | en_ZA |
| dc.title | Sensitivity analysis of multilayer neural networks | en_ZA |
| dc.type | Thesis | en_ZA |
Files
Original bundle
1 - 1 of 1
Loading...
- Name:
- engelbrecht_sensitivity_1999.pdf
- Size:
- 51.28 MB
- Format:
- Adobe Portable Document Format
- Description: