Neural Information Processing - Letters and Reviews

Vol. 12, Nos. 1-3, January-March 2008

 

pp. 11-30

 

Theory and Experiments of Exchange Ratio for Exchange Monte Carlo Method

 

Kenji Nagata

Department of Computational Intelligence and Systems Science, Tokyo Institute of Technology

MailBox R2-5, 4259, Nagatsuta-chou, Midori-ku, Yokohama, 226-8503, Japan

E-mail: kenji.nagata@cs.pi.titech.ac.jp

 

Sumio Watanabe

P&I. Lab., Tokyo Institute of Technology

MailBox R2-5, 4259, Nagatsuta-chou, Midori-ku, Yokohama, 226-8503, Japan

E-mail: swatanab@cs.pi.titech.ac.jp

 

Abstract

In hierarchical learning machines such as neural networks, Bayesian learning provides better generalization performance than maximum likelihood estimation. However, its accurate approximation using the Markov chain Monte Carlo (MCMC) method requires a huge computational cost. The exchange Monte Carlo (EMC) method was proposed as an improvement on the MCMC method. Although it has been shown to be effective not only in Bayesian learning but also in many fields, the mathematical foundation of the EMC method has not yet been established. In our previous work, we derived the asymptotic behavior of the average exchange ratio, which is used as a criterion for designing the EMC method. In this paper, we verify the accuracy of our theoretical result by the simulation of Bayesian learning in linear neural networks, and propose the method to check the convergence of EMC method based on our theoretical result.

 

Keywords − Markov Chain Monte Carlo Method, Exchange Monte Carlo Method, Exchange Ration.