Jose C. Principe

We have come to know that our ability to survive and grow as a nation to a very large degree depends upon our scientific progress. Moreover, it is not enough simply to keep 1 abreast of the rest of the world in scientific matters. We must maintain our leadership. President Harry Truman spoke those words in 1950, in the aftermath of World War II and in the midst of the Cold War. Indeed, the scientific and engineering leadership of the United States and its allies in the twentieth century played key roles in the successful outcomes of both World War II and the Cold War, sparing the world the twin horrors of fascism and totalitarian communism, and fueling the economic prosperity that followed. Today, as the United States and its allies once again find themselves at war, President Truman’s words ring as true as they did a half-century ago. The goal set out in the Truman Administration of maintaining leadership in science has remained the policy of the U. S. Government to this day: Dr. John Marburger, the Director of the Office of Science and Technology (OSTP) in the Executive Office of the President, made remarks to that effect during his 2 confirmation hearings in October 2001. The United States needs metrics for measuring its success in meeting this goal of maintaining leadership in science and technology. That is one of the reasons that the National Science Foundation (NSF) and many other agencies of the U. S.

This book provides several efficient Kalman filters (linear or nonlinear) under information theoretic criteria. They achieve excellent performance in complicated non-Gaussian noises with low computation complexity and have great practical application potential. The book combines all these perspectives and results in a single resource for students and practitioners in relevant application fields. Each chapter starts with a brief review of fundamentals, presents the material focused on the most important properties and evaluates comparatively the models discussing free parameters and their effect on the results. Proofs are provided at the end of each chapter. The book is geared to senior undergraduates with a basic understanding of linear algebra, signal processing and statistics, as well as graduate students or practitioners with experience in Kalman filtering.

This book provides several efficient Kalman filters (linear or nonlinear) under information theoretic criteria. They achieve excellent performance in complicated non-Gaussian noises with low computation complexity and have great practical application potential. The book combines all these perspectives and results in a single resource for students and practitioners in relevant application fields. Each chapter starts with a brief review of fundamentals, presents the material focused on the most important properties and evaluates comparatively the models discussing free parameters and their effect on the results. Proofs are provided at the end of each chapter. The book is geared to senior undergraduates with a basic understanding of linear algebra, signal processing and statistics, as well as graduate students or practitioners with experience in Kalman filtering.

This bookisan outgrowthoften yearsof researchatthe Universityof Florida Computational NeuroEngineering Laboratory (CNEL) in the general area of statistical signal processing and machine learning. One of the goals of writing the book is exactly to bridge the two ?elds that share so many common problems and techniques but are not yet e?ectively collaborating. Unlikeotherbooks thatcoverthe state ofthe artinagiven?eld,this book cuts across engineering (signal processing) and statistics (machine learning) withacommontheme:learningseenfromthepointofviewofinformationt- orywithanemphasisonRenyi'sde?nitionofinformation.Thebasicapproach is to utilize the information theory descriptors of entropy and divergence as nonparametric cost functions for the design of adaptive systems in unsup- vised or supervised training modes. Hence the title: Information-Theoretic Learning (ITL). In the course of these studies, we discovered that the main idea enabling a synergistic view as well as algorithmic implementations, does not involve the conventional central moments of the data (mean and covariance). Rather, the core concept is the ?-norm of the PDF, in part- ular its expected value (? = 2), which we call the information potential. This operator and related nonparametric estimators link information theory, optimization of adaptive systems, and reproducing kernel Hilbert spaces in a simple and unconventional way.

We have come to know that our ability to survive and grow as a nation to a very large degree depends upon our scientific progress. Moreover, it is not enough simply to keep 1 abreast of the rest of the world in scientific matters. We must maintain our leadership. President Harry Truman spoke those words in 1950, in the aftermath of World War II and in the midst of the Cold War. Indeed, the scientific and engineering leadership of the United States and its allies in the twentieth century played key roles in the successful outcomes of both World War II and the Cold War, sparing the world the twin horrors of fascism and totalitarian communism, and fueling the economic prosperity that followed. Today, as the United States and its allies once again find themselves at war, President Truman’s words ring as true as they did a half-century ago. The goal set out in the Truman Administration of maintaining leadership in science has remained the policy of the U. S. Government to this day: Dr. John Marburger, the Director of the Office of Science and Technology (OSTP) in the Executive Office of the President, made remarks to that effect during his 2 confirmation hearings in October 2001. The United States needs metrics for measuring its success in meeting this goal of maintaining leadership in science and technology. That is one of the reasons that the National Science Foundation (NSF) and many other agencies of the U. S.

This bookisan outgrowthoften yearsof researchatthe Universityof Florida Computational NeuroEngineering Laboratory (CNEL) in the general area of statistical signal processing and machine learning. One of the goals of writing the book is exactly to bridge the two ?elds that share so many common problems and techniques but are not yet e?ectively collaborating. Unlikeotherbooks thatcoverthe state ofthe artinagiven?eld,this book cuts across engineering (signal processing) and statistics (machine learning) withacommontheme:learningseenfromthepointofviewofinformationt- orywithanemphasisonRenyi'sde?nitionofinformation.Thebasicapproach is to utilize the information theory descriptors of entropy and divergence as nonparametric cost functions for the design of adaptive systems in unsup- vised or supervised training modes. Hence the title: Information-Theoretic Learning (ITL). In the course of these studies, we discovered that the main idea enabling a synergistic view as well as algorithmic implementations, does not involve the conventional central moments of the data (mean and covariance). Rather, the core concept is the ?-norm of the PDF, in part- ular its expected value (? = 2), which we call the information potential. This operator and related nonparametric estimators link information theory, optimization of adaptive systems, and reproducing kernel Hilbert spaces in a simple and unconventional way.

Online learning from a signal processing perspective There is increased interest in kernel learning algorithms in neural networks and a growing need for nonlinear adaptive algorithms in advanced signal processing, communications, and controls. Kernel Adaptive Filtering is the first book to present a comprehensive, unifying introduction to online learning algorithms in reproducing kernel Hilbert spaces. Based on research being conducted in the Computational Neuro-Engineering Laboratory at the University of Florida and in the Cognitive Systems Laboratory at McMaster University, Ontario, Canada, this unique resource elevates the adaptive filtering theory to a new level, presenting a new design methodology of nonlinear adaptive filters. Covers the kernel least mean squares algorithm, kernel affine projection algorithms, the kernel recursive least squares algorithm, the theory of Gaussian process regression, and the extended kernel recursive least squares algorithm Presents a powerful model-selection method called maximum marginal likelihood Addresses the principal bottleneck of kernel adaptive filters—their growing structure Features twelve computer-oriented experiments to reinforce the concepts, with MATLAB codes downloadable from the authors' Web site Concludes each chapter with a summary of the state of the art and potential future directions for original research Kernel Adaptive Filtering is ideal for engineers, computer scientists, and graduate students interested in nonlinear adaptive systems for online applications (applications where the data stream arrives one sample at a time and incremental optimal solutions are desirable). It is also a useful guide for those who look for nonlinear adaptive filtering methodologies to solve practical problems.

Neural interfaces are one of the most exciting emerging technologies to impact bioengineering and neuroscience because they enable an alternate communication channel linking directly the nervous system with man-made devices. This book reveals the essential engineering principles and signal processing tools for deriving control commands from bioelectric signals in large ensembles of neurons. The topics featured include analysis techniques for determining neural representation, modeling in motor systems, computing with neural spikes, and hardware implementation of neural interfaces. Beginning with an exploration of the historical developments that have led to the decoding of information from neural interfaces, this book compares the theory and performance of new neural engineering approaches for BMIs. Contents: Introduction to Neural Interfaces / Foundations of Neuronal Representations / Input-Outpur BMI Models / Regularization Techniques for BMI Models / Neural Decoding Using GenerativeBMI Models / Adaptive Algorithms for Point Processes / BMI Systems

The first truly up-to-date look at the theory and capabilities of nonlinear dynamical systems that take the form of feedforward neural network structures Considered one of the most important types of structures in the study of neural networks and neural-like networks, feedforward networks incorporating dynamical elements have important properties and are of use in many applications. Specializing in experiential knowledge, a neural network stores and expands its knowledge base via strikingly human routes–through a learning process and information storage involving interconnection strengths known as synaptic weights. In Nonlinear Dynamical Systems: Feedforward Neural Network Perspectives, six leading authorities describe recent contributions to the development of an analytical basis for the understanding and use of nonlinear dynamical systems of the feedforward type, especially in the areas of control, signal processing, and time series analysis. Moving from an introductory discussion of the different aspects of feedforward neural networks, the book then addresses: *Classification problems and the related problem of approximating dynamic nonlinear input-output maps *The development of robust controllers and filters *The capability of neural networks to approximate functions and dynamic systems with respect to risk-sensitive error *Segmenting a time series It then sheds light on the application of feedforward neural networks to speech processing, summarizing speech-related techniques, and reviewing feedforward neural networks from the viewpoint of fundamental design issues. An up-to-date and authoritative look at the ever-widening technical boundaries and influence of neural networks in dynamical systems, this volume is an indispensable resource for researchers in neural networks and a reference staple for libraries.

This text provides for the development of intuition about the behavior of adaptive systems by experimenting with them using real data. It is an interactive learning tool thereby introducing a new teaching methodology, offering over 200 fully functional simulations.