In this chapter, we present a learning algorithm specifically designed to learn dynam- ical features of time series that are directly predictive of the associated labels. Rather than depending on label-free unsupervised learning to discover relevant features of the time series, we build a system that expressly learns the dynamics that are most rele- vant for classifying time series labels. Our goal is to obtain compact representations of nonstationary and multivariate time series (representation learning)(Bengio, Courville & Vincent 2013). To accomplish this we use a connection between dynamic bayesian networks (e.g., the switching VAR model) and artificial neural networks (ANNs) to perform inference and learning in state-space models in a manner analogous to back- propagation in neural networks (Rumelhart, Hinton & Williams 1988). This connection stems from the observation that the directed acyclic graph structure of a state-space model can be unrolled both as a function of time and inference steps to yield a deter- ministic neural network with efficient parameter tying across time (see Fig. 1.2). Thus, the parameters governing the dynamics and observation model of a state-space model can be learned in a manner analogous to that of a neural network. Indeed, the resulting system can be viewed as a compactly-parameterized recurrent neural network (RNN) (Sutskever 2013). Although the standard use of RNNs has been for time series pre- diction (network output is the predicted input time series in the future) or sequential labeling (when output is a label sequence associated with the input data sequence), with additional processing layers one may obtain a time series classifier from this class of models (Graves, Ferna ńdez, Gomez & Schmidhuber 2006). Nevertheless, RNNs have proven hard to train, since the optimization surface tend to include multiple local min- ima. Moreover, standard RNN are ’black box’ algorithms(as apposed to ’model-based’) and therefore do allow for incorporation of physiological models of the underlying sys- tems. The framework proposed here addresses both these shortcomings. First, knowl- edge of the underlying physiology can be directly incorporated into the state-space mod- els that constitute the basic building blocks of a dynamic Bayesian network. Secondly, equipped with a generative model, we can rely on unsupervised pre-training (via expec- tation maximization) to systematically initialize the parameters of the equivalent RNN; in a manner analogous to pre-training of very large neural networks (deep learning) (Erhan, Bengio, Courville, Manzagol, Vincent & Bengio 2010).
@inbook{nemati2015identifying,
year = {2015},
author = {Nemati, Shamim and Adams, Ryan P.},
title = {Identifying Outcome-Discriminative Dynamics in Multivariate Physiological Cohort Time Series},
booktitle = {Advanced State Space Methods for Neural and Clinical Data},
publisher = {Cambridge University Press},
address = {Cambridge, UK},
keywords = {deep learning, graphical models, biomedical engineering, time series}
}