On-Line Learning in Neural Networks

On-line learning is one of the most powerful and commonly used techniques for training large layered networks and has been used successfully in many real-world applications. Traditional analytical methods have been recently complemented by ones from statistical physics and Bayesian statistics. This powerful combination of analytical methods provides more insight and deeper understanding of existing algorithms and leads to novel and principled proposals for their improvement. This book presents a coherent picture of the state-of-the-art in the theoretical analysis of on-line learning. An introduction relates the subject to other developments in neural networks and explains the overall picture. Surveys by leading experts in the field combine new and established material and enable non-experts to learn more about the techniques and methods used. This book, the first in the area, provides a comprehensive view of the subject and will be welcomed by mathematicians, scientists and engineers, whether in industry or academia.

Foreword C. Bishop; 1. Introduction D. Saad; 2. On-line learning and stochastic approximations Léon Bottou; 3. Exact and perturbative solutions for the ensemble dynamics Todd Leen; 4. A statistical study of on-line learning Noboru Murata; 5. On-line learning in switching and drifting environments Klaus-Robert Mueller, Andreas Ziehe, Noboru Murata and Shun-ichi Amari; 6. Parameter adaptation in stochastic optimization Luis B. Almeida, Thibault Langlois, José D. Amaral and Alexander Plakhov; 7. Optimal on-line learning for multilayer neural networks David Saad and Magnus Rattray; 8. Universal asymptotics in committee machines with tree architecture Mauro Copelli and Nestor Caticha; 9. Incorporating curvature information in on-line learning Magnus Rattray and David Saad; 10. Annealed on-line learning in multilayer networks Siegfried Bös and Shun-ichi Amari; 11. On-line learning of prototypes and principal components Michael Biehl, Ansgar Freking, Matthias Hölzer, Georg Reents and Enno Schlösser; 12. On-line learning with time-correlated patterns Tom Heskes and Wim Wiegerinck; 13. On-line learning from finite training sets David Barber and Peter Sollich; 14. Dynamics of supervised learning with restricted training sets Anthony C. C. Coolen and David Saad; 15. On-line learning of a decision boundary with and without queries Yoshiyuki Kabashima and Shigeru Shinomoto; 16. A Bayesian approach to on-line learning Manfred Opper; 17. Optimal perception learning: an on-line Bayesian approach Sara A. Solla and Ole Winther.

On-line learning is one of the most commonly used techniques for training large layered networks. Traditional methods have been recently complemented by ones from statistical physics and Bayesian statistics to provide more insight and deeper understanding of existing algorithms. This book presents a coherent picture of the state-of-the-art.