Merge branch 'release-1.1'

.travis.yml

@@ -7,15 +7,12 @@ python:
   - "3.6"
 env:
   matrix:
-    - TF_VERSION=1.0
-    - TF_VERSION=1.1
-    - TF_VERSION=1.2
-    - TF_VERSION=1.3
-    - TF_VERSION=1.4
+    - TF_VERSION=1.12.2
+    - TF_VERSION=1.13.2
+    - TF_VERSION=1.14
 # command to install dependencies
 install:
-  # Python 2.7 has very old pip by default that doesn't have tensorflow.
-  - pip install numpy tensorflow==$TF_VERSION
+  - pip install tensorflow==$TF_VERSION
   - pip install coveralls
 # command to run tests
 script:

CHANGELOG.md

@@ -6,6 +6,24 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 
 ## [Unreleased]
 
+## [1.1.0] - 2019-10-22
+### Added
+- Proper documentation with tutorials.
+- Benchmarks.
+- Asymptotic complexities of some function in the docstrings.
+- Eager mode support.
+- Ops have name scopes now.
+- Riemannian autodiff (Yey!)
+- Better support for using dtypes other than float32.
+- Neural network module.
+
+### Changed
+- Change to support TF 1.12-1.14 instead of TF 1.0-1.4
+- Bug fixes.
+- Bug fix in benchmarks.
+- Rename quadratic_form to bilinear_form.
+- Better test coverage.
+
 ## [1.0.0] - 2018-01-08
 ### Added
 - API reference documentation.
@@ -73,5 +91,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/).
 - frobenius_norm
 
 [Unreleased]: https://github.com/Bihaqo/t3f/compare/master...develop
+[1.1.0]: https://github.com/Bihaqo/t3f/compare/1.0.0...1.1.0
+[1.0.0]: https://github.com/Bihaqo/t3f/compare/0.3.0...1.0.0
+[0.3.0]: https://github.com/Bihaqo/t3f/compare/0.2.0...0.3.0
 [0.2.0]: https://github.com/Bihaqo/t3f/compare/0.1.0...0.2.0
 [0.1.0]: https://github.com/Bihaqo/t3f/compare/f24409508...0.1.0

LICENSE

@@ -1,7 +1,7 @@
-Copyright 2017 Alexander Novikov, Pavel Izmailov, Ivan Oseledets, Mikhail Trofimov, Valentin Khrulkov
+Copyright 2017 Alexander Novikov, Pavel Izmailov, Ivan Oseledets, Michael Figurnov, Valentin Khrulkov
 
 Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
 
 The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
 
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

README.md

@@ -1,93 +1,34 @@
 [![Build Status](https://travis-ci.org/Bihaqo/t3f.svg?branch=develop)](https://travis-ci.org/Bihaqo/t3f)
 [![Coverage Status](https://coveralls.io/repos/github/Bihaqo/t3f/badge.svg?branch=develop)](https://coveralls.io/github/Bihaqo/t3f?branch=develop)
 
-TensorFlow implementation of the Tensor Train (TT) -Toolbox.
+TensorFlow implementation of a library for working with Tensor Train (TT) decomposition which is also known as Matrix Product State (MPS).
 
-# API
-API is available via [readthedocs](https://t3f.readthedocs.io/en/latest/).
+# Documentation
+The documentation is available via [readthedocs](https://t3f.readthedocs.io/en/latest/index.html).
 
-# Installation
-T3f assumes you have a working TensorFlow installation, supported versions are from 1.0 to 1.4 (see [here](https://www.tensorflow.org/versions/r1.4/install/) for TF 1.4 installation instructions).
-We don't include it into pip requirements since the installation of TensorFlow varies depending on your setup.
-Then simply run
-```bash
-pip install t3f
-```
-
-# Basic usage
-Import the libraries
-```python
-import tensorflow as tf
-import t3f
-```
-
-Generate a random tensor and compute its norm.
-```python
-# Create a random tensor of shape (3, 2, 2).
-a = t3f.random_tensor((3, 2, 2), tt_rank=3)
-norm = t3f.frobenius_norm(a)
-# Convert TT-tensor into a dense tensor for printing.
-a_full = t3f.full(a)
-# Run a tensorflow session to run the operations.
-with tf.Session() as sess:
-  # Run the operations. Note that if you run these
-  # two operations separetly (sess.run(a_full), sess.run(norm))
-  # the result will be different, since sess.run will
-  # generate a new random tensor a on each run because `a' is
-  # an operation 'generate me a random tensor'.
-  a_val, norm_val = sess.run([a_full, norm])
-  print('The norm is %f' % norm_val)
-  print(a_val)
-```
-
-### Arithmetic
-```python
-a = t3f.random_tensor((3, 2, 2), tt_rank=3)
-b_dense = tf.random_normal((3, 2, 2))
-# Use TT-SVD on b_dense.
-b_tt = t3f.to_tt_tensor(b_dense, max_tt_rank=4)
-sum_round = t3f.round(t3f.add(a, b_tt), max_tt_rank=2)
-```
-
-### Tensor operations
-```python
-# Inner product (sum of products of all elements).
-a = t3f.random_tensor((3, 2, 2), tt_rank=3)
-b = t3f.random_tensor((3, 2, 2), tt_rank=4)
-inner_prod = t3f.flat_inner(a, b)
-```
-
-### Matrix operations
-```python
-A = t3f.random_matrix(((3, 2, 2), (2, 3, 3)), tt_rank=3)
-b = t3f.random_matrix(((2, 3, 3), None), tt_rank=3)
-# Matrix-by-vector
-matvec = t3f.matmul(A, b)
-
-# Matrix-by-dense matrix
-b_dense = tf.random_normal((18, 1))
-matvec2 = t3f.matmul(A, b_dense)
-```
-# More examples
-For more examples (e.g. how to build neural networks or how to do Riemannian optimization) see ```examples``` folder.
-
-# Benchmarking
-Here are the results im ms of benchmarking T3F on CPU and GPU and comparing against the [TTPY library](https://github.com/oseledets/ttpy)
-<img src="examples/profile/results.png" height="200">
-
-For more details see ```examples/profile``` folder.
+# Comparison with other libraries
+There are about a dozen other libraries implementing Tensor Train decomposition. 
+The main difference between `t3f` and other libraries is that `t3f` has extensive support for Riemannian optimization and that it uses TensorFlow as backend and thus supports GPUs, automatic differentiation, and batch processing. For a more detailed comparison with other libraries, see the [corresponding page](https://t3f.readthedocs.io/en/latest/comparison.html) in the docs.
 
 # Tests
 ```bash
 nosetests  --logging-level=WARNING
 ```
 
-# Building API documentation
-The documentation is build by sphinx and hosted on readthedocs.org. To rebuild the documentation, install sphinx and compile the docs by
+# Building documentation
+The documentation is build by sphinx and hosted on readthedocs.org. To locally rebuild the documentation, install sphinx and compile the docs by
 ```bash
 cd docs
 make html
 ```
 
-# Other implementations
-There are also implementations of the TT-toolbox in [plain Python](https://github.com/oseledets/ttpy) and [Matlab](https://github.com/oseledets/TT-Toolbox).
+# Citing
+If you use T3F in your research work, we kindly ask you to cite [the paper](https://arxiv.org/abs/1801.01928) describing this library
+```
+@article{novikov2018tensor,
+  title={Tensor Train decomposition on TensorFlow (T3F)},
+  author={Novikov, Alexander and Izmailov, Pavel and Khrulkov, Valentin and Figurnov, Michael and Oseledets, Ivan},
+  journal={arXiv preprint arXiv:1801.01928},
+  year={2018}
+}
+```

docs/api.rst

@@ -0,0 +1,51 @@
+.. t3f documentation master file, created by
+   sphinx-quickstart on Sun Mar 12 10:06:09 2017.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+API
+===
+
+Here you can find description of functions and methods avaliable in t3f.
+
+Module contents
+---------------
+
+.. automodule:: t3f
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+t3f\.nn module
+-----------------
+
+.. automodule:: t3f.nn
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+
+t3f\.utils module
+-----------------
+
+.. automodule:: t3f.utils
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+t3f\.kronecker module
+---------------------
+
+.. automodule:: t3f.kronecker
+    :members:
+    :undoc-members:
+    :show-inheritance:
+
+t3f\.approximate module
+-----------------------
+
+.. automodule:: t3f.approximate
+    :members:
+    :undoc-members:
+    :show-inheritance:

docs/benchmark.rst

@@ -0,0 +1,33 @@
+Benchmark
+================
+
+The performance of different libraries implementing Tensor Train decomposition is a bit tricky to compare fairly and is actually not that different because everyone relies on the same BLAS/MKL subruitines.
+
+So the main purpose of this section is not to prove that T3F is faster than every other library (it is not), but rather to assess GPU gains on different ops and identify bottlenecks by comparing to *some* other library. As a reference implementation, we decided to use ttpy_ library.
+
+.. _ttpy: https://github.com/oseledets/ttpy
+
+See the following table for time in ms of different opeartions run in ttpy_ (second column) and in t3f (other columns).
+
+===============  ==================  ============  ============  ==============  ==============
+Operation        ttpy, one on CPU    one on CPU    one on GPU    batch on CPU    batch on GPU
+===============  ==================  ============  ============  ==============  ==============
+matvec                       11.142         1.19          0.744           1.885           0.14
+matmul                       86.191         9.849         0.95           17.483           1.461
+norm                          3.79          2.136         1.019           0.253           0.044
+round                        73.027        86.04        165.969           8.234         161.102
+gram                          0.145         0.606         0.973           0.021           0.001
+project_rank100             116.868         3.001        13.239           1.645           0.226
+===============  ==================  ============  ============  ==============  ==============
+
+The timing in the "batch" columns represent running the operation for a 100 of objects at the same time and then reporting the time per object. E.g. the last number in the first row (0.14) means that multiplying a single TT-matrix by 100 different TT-vectors takes 14 ms on GPU when using T3F, which translates to 0.14 ms per vector.
+
+Note that rounding operation is slow on GPU. This is a `known TensorFlow bug`_, that the SVD implementation is slower on GPU than on CPU.
+
+.. _`known TensorFlow bug`: https://github.com/tensorflow/tensorflow/issues/13603
+
+The benchmark was run on NVIDIA DGX-1 server with Tesla V100 GPU and Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz with 80 logical cores
+
+To run this benchmark on your own hardware, see `docs/benchmark`_ folder.
+
+.. _`docs/benchmark`: https://github.com/Bihaqo/t3f/tree/develop/docs/benchmark

docs/benchmark/README.md

@@ -0,0 +1,19 @@
+## Benchmarking T3F
+To benchmark the library, use the following commands
+```bash
+# Running on CPU.
+CUDA_VISIBLE_DEVICES= python benchmark.py --file_path logs_cpu.pkl
+# Running on GPU.
+CUDA_VISIBLE_DEVICES=0 python benchmark.py --file_path logs_gpu.pkl
+```
+To visualize the results in a table, see ```results.ipynb``` Jupyter notebook.
+Here are the numbers you can get on NVIDIA DGX-1 server with Tesla V100 GPU and Intel(R) Xeon(R) CPU E5-2698 v4 @ 2.20GHz with 80 logical cores
+ <img src="results.png" height="200">
+
+See [here](https://t3f.readthedocs.io/en/latest/benchmark.html) for a bit more details on how to interpret these numbers.
+
+## Comparing against TTPY
+To benchmark T3F against another library for Tensor Train decomposition [TTPY](https://github.com/oseledets/ttpy), install TTPY and run the following command in the bash shell
+```bash
+python benchmark_ttpy.py --file_path logs_ttpy.py
+```