Statistical Physics Analysis of Training of Restricted Boltzmann Machines

A restricted Boltzmann machine is a generative probabilistic graphic network. A probability of finding the network in a certain configuration is given by the Boltzmann distribution. It has wide applications from the image generation to the neural network representation of quantum many-body states. We analyze the training process of the restricted Boltzmann machine in the context of statistical physics. For the Bar-and-Stripe pattern as a small size restricted Boltzmann machine, thermodynamic quantities such as free energy, internal energy, work, and entropy are calculated as a function of training epochs. We also investigate the Jarzynski equality that connects the work done during training and the difference in free energies before and after training. It is found that, even after long training, the probabilities of possible outcomes are not even. Some possible sources that cause imperfect training are discussed.