unsupervised

1. summary Tutorial on unsupervised learning using pmtk3

Basics

We can create an unconditional density model of type 'foo' as follows

The arguments '...' depends on the form of the model, and are explained below.

Currently, 'foo' can be one of the following (in alphabetical order)

 * crf2 (Conditional random field with discrete hidde nodes, pairwise potentials)
 * hmm (hidden Markov model, any kind of observation)
 * lds (linear dynamical system)
 * markov (discrete time, discrete state Markov chain)
 * mix (finite mixture models)
 * mrf2 (Markov random field with discrete nodes, pairwise potentials)
 * ppca (probabilistic principal components analysis)

Below we describe the methods that each model supports. Note that not all models support all methods. Furthermore, this list is currently a specification of what the future interface will look like, rather than a description of the current interface (which can be inferred from the auto-generated documentation). To determine if a given model supports a given method, just type 'help fooMethod' and if Matlab says that 'fooMethod' is undefined, then it is not supported.

The basic operations on an unconditional density model are as follows

Note that fooFit(X,...) is shorthand for fooTrain(fooCreate(...),X,...).

For some models, fooEnergy is the same as -fooLogprob, but for those with intractable normalization constants Z (e.g., MRFs), fooEnergy is more efficient to compute.

X is an N*D matrix, as usual. It may have NaN's in it, which means the corresponding entry is missing (hidden). Whether NaN's are supported depends on the particular model and the particular fitting/ inference method you choose; this is controlled by the optional arguments '...' passed to create and/or fit (see below for details).

All of the above methods are vectorized (conceptually, at least), in the sense that they work on a set of datacases in batch mode.

For joint probability models (with more than one variable), it is possible to condition on one set of variables and infer the values of others. We therefore define the following additional methods (datacase is like a single row of X, and may contain NaN's for 'unclamped' variables):

For graphical models (with no explicit latent variables), we define the following additional methods (we assume hidden nodes are discrete):

For models with latent variables, we define the following additional methods:

Below we describe some particular models in more detail.

Mixture models

We will discuss mixtures of Gaussians as an example, but we can make mixtures of other kinds of distributions in a similar way.

We can fit the model, consisting of K mixture components, as follows:

We can then infer the posterior responsibilities of each data point (i.e., a soft clustering) using

See the following demos for examples of how to use this code in practice:

 * [http://pmtk3.googlecode.com/svn/trunk/docs/demoOutput/Mixture_models/mixGaussDemoFaithful.html mixGaussDemoFaithful] Fit a mixture of Gaussians  using EM to the Old Faithful dataset.

 * [http://pmtk3.googlecode.com/svn/trunk/docs/demoOutput/Mixture_models/mixGaussVbDemoFaithful.html  mixGaussVbDemoFaithful] Fit a mixture of Gaussians  using variational Bayes to the Old Faithful dataset. This illustrates automatic 'killing off' of unwanted components.

.