Synthesis Lectures on Intelligent Technologies Ardahir Mohammadazadeh · Mohammad Hosein Sabzalian · Oscar Castillo · Rathinasamy Sakthivel · Fayez F. M. El-Sousy · Saleh Mobayen Neural Networks and Learning Algorithms in MATLAB Synthesis Lectures on Intelligent Technologies Series Editor Janusz Kacprzyk, Systems Research Institute, Polish Academy of Science, Warsaw, Poland Synthesis Lectures on Intelligent Technologies provides highly interdisciplinary research with the potential to change the fundamental principles of our society. It covers appli- cations such as Intelligent Transportation, Humanoids, Self-Driving Cars, IoT, Ambient Intelligence, Smart Cities, Human-computer Interaction, Computational Intelligence, Industry 4.0, Medical Robotics, or Data Science. Synthesis Lectures on Intelligent Tech- nologies brings together up-to-date resources from trusted authors working around the world in all aspects of Intelligent Systems. Ardahir Mohammadazadeh · Mohammad Hosein Sabzalian · Oscar Castillo · Rathinasamy Sakthivel · Fayez F. M. El-Sousy · Saleh Mobayen Neural Networks and Learning Algorithms in MATLAB Ardahir Mohammadazadeh Mohammad Hosein Sabzalian Multidisciplinary Center for Infrastructure LabREI-Smart Grid Laboratory, Department Engineering of Systems and Energy, FEEC-School Shenyang University of Technology of Electrical and Computer Engineering Shenyang, China University of Campinas Campinas, Brazil Oscar Castillo Division of Graduate Studies and Research Rathinasamy Sakthivel Tijuana Institute of Technology Department of Applied Mathematics Tijuana, Mexico Bharathiar University Coimbatore, Tamil Nadu, India Fayez F. M. El-Sousy Department of Electrical Engineering, College Saleh Mobayen of Engineering Multidisciplinary Center for Infrastructure Prince Sattam Bin Abdulaziz University Engineering Al Kharj, Saudi Arabia Shenyang University of Technology Shenyang, China ISSN 2731-6912 ISSN 2731-6920 (electronic) Synthesis Lectures on Intelligent Technologies ISBN 978-3-031-14570-4 ISBN 978-3-031-14571-1 (eBook) https://doi.org/10.1007/978-3-031-14571-1 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland Preface This book explains the ins and outs of neural networks in a simple approach with clear examples and simulations in MATLAB. The scripts herein are coded for general purposes to be easily extended to a variety of problems. They are vectorized and optimized to run faster and be applicable to high-dimensional engineering problems. Shenyang, China Ardahir Mohammadazadeh Campinas, Brazil Mohammad Hosein Sabzalian Tijuana, Mexico Oscar Castillo Coimbatore, India Rathinasamy Sakthivel Al Kharj, Saudi Arabia Fayez F. M. El-Sousy Shenyang, China Saleh Mobayen v Contents 1 Introduction ......................................................... 1 1.1 Overview ....................................................... 1 1.2 Some Applications of Neural Networks ............................. 2 1.3 Different Types of Neural Network Training ......................... 3 1.4 Learning Principles in Neural Networks ............................. 4 References ........................................................... 4 2 Multilayer Perceptron (MLP) Neural Networks ......................... 5 2.1 Training Based on Error Backpropagation ........................... 8 2.2 Implementation in MATLAB ...................................... 9 2.3 Application of Neural Network in Classification ...................... 12 2.4 Over-Parameterization ............................................ 15 2.5 Over-Training .................................................... 18 2.6 Training Based on Full Propagation ................................ 18 References ........................................................... 21 3 Neural Networks Training Based on Recursive Least Squares (RLS) ...... 23 3.1 RLS Training Technique .......................................... 23 3.2 Implementation in MATLAB ...................................... 24 3.3 Comparison with Gradient Descent ................................. 29 4 Neural Networks Training Based on Second-Order Optimization Technique ........................................................... 31 4.1 Introduction ..................................................... 31 4.2 Newton’s Method ................................................ 32 4.3 Levenberg–Marquardt Algorithm ................................... 32 4.4 Conjugate Gradient (CG) Method .................................. 33 4.5 Implementation in MATLAB ...................................... 33 References ........................................................... 38 vii viii Contents 5 Neural Networks Training Based on Genetic Algorithm .................. 39 5.1 Introduction ..................................................... 39 5.1.1 What is the Genetic Algorithm (GA)? ....................... 39 5.1.2 Operators of a Genetic Algorithm ........................... 41 5.1.3 Applications of Genetic Algorithm .......................... 42 5.2 Genetic Algorithm in MATLAB .................................... 43 5.3 Optimization of Neural Network Parameters Based on Genetic Algorithm ....................................................... 54 Reference ............................................................ 59 6 Neural Network Training Based Particle Swarm Optimization (PSO) ..... 61 6.1 Introduction ..................................................... 61 6.2 Algorithm Formulation ............................................ 61 6.3 Implementation in MATLAB ...................................... 64 References ........................................................... 68 7 Neural Network Training Based on UKF ............................... 69 7.1 UKF Algorithm .................................................. 69 7.2 Implementation in MATLAB ...................................... 72 References ........................................................... 78 8 Designing Neural-Fuzzy PID Controller Through Multiobjective Optimization ......................................................... 79 8.1 Introduction ..................................................... 79 8.2 Classic Methods ................................................. 79 8.2.1 Ziegler–Nichols Method ................................... 79 8.2.2 Cohen-Coon Method ...................................... 80 8.2.3 Smart Methods ........................................... 80 8.2.4 Single-Objective Optimization .............................. 81 8.2.5 Multiobjective Optimization ................................ 82 8.2.6 Primary Definitions ....................................... 83 8.2.7 Decision Variables ........................................ 84 8.2.8 Constraints ............................................... 84 8.2.9 Objective Functions ....................................... 84 8.2.10 Dominance .............................................. 84 8.2.11 Non-Dominated Set ....................................... 85 8.2.12 Pareto Principle .......................................... 85 8.2.13 Optimal Pareto Solution ................................... 86 8.2.14 Optimal Pareto Set ........................................ 86 8.3 Objectives of Multiobjective Optimization ........................... 87 8.3.1 Common Algorithms in Solving Multiobjective Optimization ............................................. 87 8.4 Designing Multiobjective PID Controller ............................ 88 Contents ix 8.5 Designing a MOPID Controller for a Sample Power System ........... 89 8.5.1 First State ............................................... 90 8.5.2 Second State ............................................. 93 8.6 Using Fuzzy-Neural Network for Gain Schedule ..................... 94 8.7 Fuzzy-Neural Network Training for PID Controller Regulation ......... 96 8.7.1 Simulation for Fuzzy-Neural Controller of Gain Schedule ...... 98 8.8 Conclusion ...................................................... 100 8.9 Implementation in MATLAB ...................................... 101 8.9.1 Dynamic Model of Power System .......................... 101 8.9.2 First Example ............................................ 105 8.9.3 Supplementary Ideas on Modeling the Power System for the Frequency Load Problem ............................ 106 Uncited Reference .................................................... 117 Introduction 1 1.1 Overview Artificial neural networks are now extensively studied in order to achieve human-like efficiency. These networks consist of some linear and nonlinear computational elements that operate in tandem. Neural networks are cutting-edge computational systems and methods for machine learning, knowledge representation, and, finally, the application of acquired knowledge to predict outputs from complex systems. The main concept behind these networks is (to some extent) inspired by how the biological neural system processes data and information in order to learn and create knowledge. The main component of this concept is the development of novel structures for information processing systems. This system consists of many extremely interconnected processing elements known as the neurons that cooperate to solve problems and transfer information via synapses (electromagnetic communications). If a cell is damaged in these networks, other cells can compensate for its absence and contribute to its reconstruction. These networks are capable of learning. For instance, by applying burn to touch nerve cells, the cells learn not to approach hot objects, and the system learns to correct its mistake thanks to the algorithm. These systems learn comparatively; in other words, a new input is provided, and the weights of synapses change in a way that the system can generate accurate responses. There is no agreement among researchers on how to define a neural network; however, most agree that it consists of a network of simple processing elements (neurons) capable of displaying an overall complex behavior determined by the relationship between pro- cessing elements and element parameters. The main and inspiring source for this tech- nique is to test the central nervous system and neurons (axons, multiple branches of nerve cells, and junctions of two nerves), which are among the most important components of nervous system information processing. Simple nodes (processing elements) or units are interlinked to form a network of nodes in a neural network model. This is why they are referred to as “neural networks.” Although a neural network should not be adaptable in and of itself, it can be used practically thanks to certain algorithms designed to change the communication weight in a network (to create the desired signal). © The Author(s), under exclusive license to Springer Nature Switzerland AG 2022 1 A. Mohammadazadeh et al., Neural Networks and Learning Algorithms in MATLAB, Synthesis Lectures on Intelligent Technologies, https://doi.org/10.1007/978-3-031-14571-1_1