Graham Taylor

Two distributed-state models for generating high-dimensional time series

Graham W. Taylor, Geoffrey E. Hinton, and Sam T. Roweis

In this paper we develop a class of nonlinear generative models for high-dimensional time series. We first propose a model based on the Restricted Boltzmann Machine (RBM) that uses an undirected model with binary latent variables and real-valued ``visible'' variables. The latent and visible variables at each time step receive directed connections from the visible variables at the last few time-steps. This ``conditional'' RBM (CRBM) makes on-line inference efficient and allows us to use a simple approximate learning procedure. We demonstrate the power of our approach by synthesizing various motion sequences and by performing on-line filling in of data lost during motion capture.

We extend the CRBM in a way that preserves its most important computational properties and introduces multiplicative three-way interactions that allow the effective interaction weight between two variables to be modulated by the dynamic state of a third variable. We introduce a factoring of the implied three-way weight tensor to permit a more compact parameterization. The resulting model can capture diverse styles of motion with a single set of parameters, and the three-way interactions greatly improve its ability to blend motion styles or to transition smoothly among them.

In submission (JMLR).


While we have made every effort to ensure that this paper is a self-contained work, it is difficult to emphasize the strong generative ability of the models presented herein without demonstrating visually some examples of synthesis. In Sections 3.5 and 4.6 we make reference to several videos. They are available here, organized by section.

The videos were encoded using H.264 and so to view them with this integrated player you will need a relatively new version of Adobe Flash Player (called version 9 Update 3 or v9.0.115.0 which was released on December 3, 2007).

Section 3 Conditional Restricted Boltzmann Machines
Section 4 Factored Conditional Restricted Boltzmann Machines

Source code

Section 3 Conditional Restricted Boltzmann Machines - originally released with (Taylor, Hinton and Roweis 2007)
Section 4 Factored Conditional Restricted Boltzmann Machines