S3-backed notebook manager for IPython

Setup

1. Install:

   from pypi:

   pip install s3nb

   from source with pip:

   pip install git+https://github.com/monetate/s3nb

   or from source the old fashioned way:

   git clone [email protected]:monetate/s3nb.git
   cd s3nb
   python ./setup.py install

2. Configure

   # set this - notebooks will be stored relative to this uri
   S3_NOTEBOOK_URI=s3://path/to/notebooks/
   
   # and this
   IPYTHON_MAJOR_VERSION=4
   
   # optionally set this - checkpoints will be stored locally, relative to this path (for IPython 3)
   CHECKPOINT_ROOT_DIR=~/.checkpoints
   
   # optionally set this
   PROFILE=s3nbserver
   
   # shouldn't need to edit beyond this point
   
   ## IPython 2.x
   IPYNB_MANAGER=S3NotebookManager
   IPYNB_MANAGER_CFG=notebook_manager_class
   
   ## IPython 3.x or 4.x
   if [ $IPYTHON_MAJOR_VERSION == 3 ] || [ $IPYTHON_MAJOR_VERSION == 4 ]; then
       IPYNB_MANAGER=S3ContentsManager
       IPYNB_MANAGER_CFG=contents_manager_class
   fi
   
   IPYTHONDIR=${IPYTHONDIR:-$HOME/.ipython}
   PROFILE_DIR=${IPYTHONDIR}/profile_${PROFILE}
   
   if [ ! -d $PROFILE_DIR ]; then
       ipython profile create $PROFILE
       IPYNB_CONFIG=${PROFILE_DIR}/ipython_notebook_config.py
       mv $IPYNB_CONFIG $IPYNB_CONFIG.orig
       cat > $IPYNB_CONFIG <<EOF
   c = get_config()
   c.NotebookApp.${IPYNB_MANAGER_CFG} = 's3nb.${IPYNB_MANAGER}'
   c.${IPYNB_MANAGER}.s3_base_uri = '$S3_NOTEBOOK_URI'
   EOF
   fi
   
   
   if  [ $IPYTHON_MAJOR_VERSION == 3 ] || [$IPYTHON_MAJOR_VERSION == 4 ] ; then
       echo "c.S3ContentsManager.checkpoints_kwargs = {'root_dir': '${CHECKPOINT_ROOT_DIR}'}"  >> ${IPYNB_CONFIG}
   fi

3. If you haven't already, configure AWS variables for boto. Follow these instructions.

4. Run

   jupyter notebook --config=~/.ipython/s3nbserver/ipython_notebook_config.py

Development

1. Provision a virtual machine with vagrant up
2. Create an IPython profile with make configure -e S3_BASE_URI=YOUR_BUCKET
3. Share you AWS credentials with the virtual machine with make creds -e AWS_USER=YOUR_USER
4. Run the notebook server with make run