Skip to content

Latest commit

 

History

History

gated_linear_networks

Gated Linear Networks

Gated Linear Networks (GLNs) are a family of backpropation-free neural networks. Each neuron in a GLN predicts the target density (or probability mass) based on the outputs of the previous layer and is trained under a logarthmic loss.

GLN variants

Neurons have probabilistic "activation functions". Implementations are provided for the following distributions:

  • Gaussian, for regression.

  • Bernoulli, for binary classification and multi-class classification using a one-vs-all scheme.

Examples

Usage examples are provided in examples.

Implementation details

Constraint satisfaction

Because each neuron implements a probability density/mass function we need to ensure that they are well defined. For example, the scale parameter for a Gaussian density needs to be positive. We implement these constraints using linear projections and clipping.

Aggregation

Because each neuron predicts the target, we can use any neuron output as the "network output", and are not bound to the last layer. Typically last layer neuron(s) are the best predictors, but they might take longer to converge in theory. In this implementation, we use a single neuron at the last layer, which then forms the network output.

There are alternative ways of aggregating, e.g. see Switching Aggregation in Appendix D of Gaussian Gated Linear Networks (link: https://arxiv.org/pdf/2006.05964.pdf).

References

Coming soon.