Add docs around installing packages via pip

docs/docs/faq.md

@@ -5,57 +5,19 @@
 Nebari regenerates this file on every run. This means it will be removed by the operating system during its cleanup process,
 but running the `nebari deploy` command again as a Nebari administrator will update/create a `NEBARI_KUBECONFIG` file for you.
 
-## How are Nebari conda user environments created? Who creates them?
+## How are conda user environments created? Who creates them?
 
-The short answer: there are currently _two_ ways of creating environments, as we are in the process of migrating Nebari to conda-store,
-and so which way depends on your use-case.
+`Conda-store` manages all environments on the Nebari. It allows users to create private environments in their own namespace, or shared environments under a group namespace.
 
-The longer answer:
+Additionally, there is a legacy approach which is still available in Nebari. Administrators can create global environments by specifying the environment in `nebari-config.yml`. Environments specified in this way will be made available for all users and services under the `nebari-git` namespace on `conda-store`. We do not recommend this approach as it requires a redeploy in order to update the environment, though it may be useful in some circumstances.
 
-- For global environments, you can specify the environment in `nebari-config.yml`, and it will be made available for all users and services.
-- By comparison, creating the environments through conda-store will provide more granular control over certain settings and permissions.
-
-As Nebari and conda-store mature, the intent is to migrate exclusively to conda-store for environment creation and management.
-
-## What if I need to install package `X`, and it's not available in the environment?
-
-You can add the package to the `nebari_config.yml`. If you don't have access to the deployment repo,
-you'll need to contact your Nebari administrator to include the required package.
-
-## What's included in the conda environment if I want to use Dask?
-
-<!-- TODO: will need to update the conda-feedstock -->
+## What should be included in the environment if I want to use Dask?
 
 There are drop-in replacements for `distributed`, `dask`, and `dask-gateway` with the correct pinned versions available via the [Nebari Dask metapackage](https://github.com/conda-forge/nebari-dask-feedstock). Example: `nebari-dask==||nebari_VERSION||`.
 
-## What packages are needed in your environment to create a dashboard?
-
-When deploying an app with JHub App Launcher, you need to have the following in your environment:
-- `jhub-apps` package
-- packages corresponding  to the dashboard framework (for example, `panel`, `gradio`, etc.)
-- any other libraries required for the analysis in the dashboard creation script/notebook
-
 ## How can I install a package locally? Will this package be available to Dask workers?
 
-:::caution
-
-We _strongly recommend_ installing packages by adding them through the conda-store UI. If you're developing a package and need to install the package through `pip`, `conda`, or similar, the following approach may be used.
-
-:::
-
-If you are using a `setuptools` package, you can install it into your local user environment by:
-
-```shell
-pip install --no-build-isolation --user -e .
-```
-
-If you're using a `flit` package, you can install it through the following command:
-
-```shell
-flit install -s
-```
-
-If the package requires build tools like `gcc` and `cmake`, remember that you can create a conda environment through the conda-store UI that includes the build tools, then just activate the environment and install the package locally.
+If you want to install a package locally, we suggest following the guide for [developing local packages][developing-packages].
 
 It's important to note that packages installed this way aren't available to the Dask workers. See our [Dask tutorial][dask-tutorial] for more information.
 
@@ -69,7 +31,7 @@ Nebari automatically creates and manages `.bashrc` and `.profile`, so if the int
 source ~/.bashrc
 ```
 
-You can use `.bashrc` on Nebari, but it's important to note that by default Nebari sources `.bash_profile`. You should double-check to source the `.bashrc` inside the `.bash_profile`. Also, note that if you set environment variables in this way, these variables aren't available inside the notebooks.
+You can use `.bashrc` on Nebari, but it's important to note that by default Nebari sources `.bash_profile`. You should double-check to source the `.bashrc` inside the `.bash_profile`. Also, note that if you set environment variables in this way, these variables aren't available inside the notebooks. Additionally, the VS Code terminal does _not_ source `.bash_profile` by default.
 
 ## What if I can't see the active conda environment in the terminal?
 
@@ -81,9 +43,11 @@ conda config --set changeps1 true
 
 The `conda` config is located in the `/home/{user}/.condarc` file. You can change the conda config with a text editor (for example: `nano`, which is included in Nebari by default), and the changes will be applied on saving the file.
 
-## How do I clean up or delete the conda-store pod, if I need to?
+## How do I clean up old environment builds in conda-store?
 
-You may find that the pods hosting your environment get full over time, prompting you to clear them out. To delete old builds of your environment on conda-store, click the "delete" button in the conda-store UI.
+You may find that the pods hosting your environment get full over time, prompting you to clear them out. You can delete environments completely (including all builds of the environment) in the conda-store UI. Go to the environment, click `Edit` and then click `Delete`.
+
+If you'd like to retain the latest version of an environment and only remove specific builds, you'll need to navigate to the conda-store admin page located at `<nebari-domain/conda-store/admin>`. Click on the environment you'd like to clean up. At the bottom of the page, there is a list of each environment build, each with it's own "delete" button.
 
 ## How do I use preemptible and spot instances on Nebari?
 
@@ -163,9 +127,11 @@ Nebari automatically shuts down servers when users are idle, as described in Neb
 :::note
 Until this issue is addressed, we recommend manually shutting down your VS Code server when it is not in use.
 :::
+
 <!-- Internal links -->
 
 [dask-tutorial]: tutorials/using_dask.md
 [idle-culler-settings]: https://www.nebari.dev/docs/how-tos/idle-culling
 [selecting a profile]: tutorials/login-keycloak#4-select-a-profile
 [using gpus]: how-tos/use-gpus
+[developing-packages]: how-tos/develop-local-packages

docs/docs/how-tos/develop-local-packages.md

@@ -0,0 +1,74 @@
+---
+id: develop-local-packages
+title: Develop Local Packages on Nebari
+description: How to install a dev package from a local project
+---
+
+# How to Develop Local Packages on Nebari
+
+As a software development platform, Nebari can be used for active package development. Since `conda-store` manages all environments, a dynamic installation of a local package requires additional care (after all, `pip install -e .` is explicitly disallowed for various reasons).
+
+While the setup for developing local packages may require unfamiliar extra steps for new users, the total time for setup is minimal.
+
+## Why can't I just `pip install -e` my package?
+
+Nebari prohibits the standard local installation of packages, e.g. `pip install -e .`. On your personal computer, when you `pip install` a package into a conda environment, it will be placed inside of the conda environment folder structure. However, on Nebari, the environments are managed by `conda-store` in a directory structure which is read-only to users.
+
+For this reason, all pip installs would instead default to the user's `.local` directory. Unfortunately, _all_ environments will automatically pick up _everything_ has been installed in this directory. It is _extremely_ easy to create a situation in which all environments are broken due to a conflict. In fact, its possible to create a situation that causes a user's JupyterLab server to be unable to start. For this reason, local pip installs are prohibited in Nebari.
+
+:::note
+For more information, check out [the docs on installing pip packages](/docs/how-tos/install-pip-packages).
+:::
+
+## Installing local packages
+
+Despite the limitations on local installations, Nebari does provide a mechanism to install a local package currently under development through the use of virtual environments.
+
+### Building the environment
+
+For this example, we'll walk through creating an environment called `myenv` for a user named `myusername`.
+
+Step 1: Use conda-store to build a basic environment that only includes Python called `myenv` in the namespace `myusername` (i.e. your own namespace).
+
+Step 2: Open a terminal in JupyterLab or VS Code
+
+Step 3: list your envs to find the exact spelling: `conda env list`
+
+Step 4: Activate the environment: `conda activate myusername-myenv`
+
+Step 5: Create a virtual environment (venv) in a folder of your choice: `python -m venv .venv_myenv`
+
+Step 6: Activate the venv: `source .venv_myenv/bin/activate`
+
+Step 7: Install your package via pip: `pip install -e .` or `pip install -r requirements.txt` (as needed)
+
+### Usage from terminal
+
+To use this environment from a terminal in JupyterLab or VS Code, you'll first activate the conda environment, then activate the virtual environment.
+
+Step 1: Activate the env: `conda activate myusername-myenv`
+
+Step 2: Activate the venv: `python -m venv .venv_myenv`
+
+### Usage from Jupyter:
+
+To use this environment in Juptyer, you'll need to add the path to the environment to the system path since its location isn't automatically added.
+
+Step 1: Open a notebook with the `myusername-myenv` kernel
+
+Step 2: In the first notebook cell run (note that the path and python version might be different in your usecase):
+
+```python
+import sys
+sys.path.append('/home/myusername/venv_myenv/lib/python3.10/site-packages/')
+```
+
+### Usage from VS Code Python extension
+
+To use this environment to run code via the VS Code Python extension, you'll only need to point the VS Code UI to the virtual environment. This will be automatically recognized by VS Code. You will not need to add to the `sys.path` for this approach.
+
+## Conclusion
+
+Developing local packages on Nebari requires a few extra steps compared to developing on your local machine, but the additional time requirement is minimal.
+
+To learn more about installing pip packages in general, check out the documentation on [installing pip packages from various sources](/docs/how-tos/install-pip-packages).

docs/docs/how-tos/install-pip-packages.md

@@ -0,0 +1,95 @@
+---
+id: install-pip-packages
+title: Install pip packages
+description: How to install pip packages on Nebari
+---
+
+# Installing pip packages
+
+Nebari is supported by `conda-store` for environment management. Although all environments in Nebari are - at their core - conda environments, it is still possible to install packages via `pip`.
+
+In this document we'll cover some common usecases. Please refer to the [pip VCS Support documentation](https://pip.pypa.io/en/stable/topics/vcs-support/) for additional usecases.
+
+Also note that conda-store does not have the ability to pick up your SSH keys so all references must be Open Source or utilize PATs (see examples below)
+
+:::warning
+Mixing pip and conda can result in conflicts at runtime. It is best to avoid this.
+
+If you really _must_, then continue reading.
+:::
+
+## Why can't I just use `pip install` from command line?
+
+Nebari prohibits the standard local installation of packages, e.g. `pip install ...`. On your personal computer, when you `pip install` a package into a conda environment, it will be placed inside of the conda environment folder structure. However, on Nebari, the environments are managed by `conda-store` in a directory structure which is read-only to users.
+
+For this reason, all pip installs would instead default to the user's `.local` directory. Unfortunately, _all_ environments will automatically pick up _everything_ has been installed in this directory. It is _extremely_ easy to create a situation in which all environments are broken due to a conflict. In fact, its possible to create a situation that causes a user's JupyterLab server to be unable to start. For this reason, local pip installs are prohibited in Nebari.
+
+:::note
+For more information on developing local packages, check out [the docs on developing packages on Nebari](/docs/how-tos/develop-local-packages).
+:::
+
+## Installing packages published on pypi.org
+
+To install packages which are published on pypi.org, you can create an environment in conda-store as you typically would [link] but add additional section for pip. Let's use `ragna` as an example:
+
+```yaml
+channels:
+- conda-forge
+dependencies:
+- python=3.12
+- pip
+- pip:
+  - ragna
+```
+
+## Installing packages from an Open Source git repo
+
+To install packages from an Open Source repository, you can use the http syntax. Its important to note that a tag or hash is required. Without it, git will attempt to install the `master` branch and will likely fail since most projects have moved away from this convention in favor of `main`.
+
+```yaml
+channels:
+  - conda-forge
+dependencies:
+  - python=3.11
+  - pip:
+      - ragna @ git+https://github.com/Quansight/[email protected]
+```
+
+## Installing packages from a private GitHub repository
+
+To install a packages from a private repository on GitHub, you'll first need to [create a Personal Access Token (PAT)](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens#creating-a-fine-grained-personal-access-token). Please be aware that this token will be visible to anyone with access to the environment so it is best to delete the token immediately after environment creation. We suggest limiting the scope of the PAT to read-only access to the contents for a single repository.
+
+For this example, let's suppose your GitHub user is named `username` and the PAT generated by GitHub is `github_pat_asdf`. We will assume the repo is located at https://github.com/username/reponame and we want to grab the `main` branch.
+
+```yaml
+channels:
+  - conda-forge
+dependencies:
+  - python=3.11
+  - pip
+  - pip:
+      - greatpackage @git+https://[email protected]/username/reponame.git@main
+```
+
+## Installing packages from a private GitLab repository
+
+To install a package from a private repository on GitLab, you'll first need to [create a PAT](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) in order to access the repo. Please be aware that this token will be visible to anyone with access to the environment so it is best to delete the token immediately after environment creation.
+
+In this example, let's suppose your GitLab user is named `username` and the PAT generated by GitLab is `glpat-asdf`. We will assume the repo is located at https://gitlab.domain.net/teamname/projectname/reponame, the package gets installed as `greatpackage`, and we want to grab the `0.7.0` release. Your GitLab repo may have more or less team/project embedding, but this is just one example.
+
+```yaml
+channels:
+  - conda-forge
+dependencies:
+  - python=3.11
+  - pip
+  - pip:
+      - greatpackage @
+        git+https://username:[email protected]/teamname/projectname/[email protected]
+```
+
+## Conclusion
+
+Now that you've seen several examples of installing pip packages in conda-store, you can go try it out on your own packages!
+
+To learn more about using a "dev" install of local pip package for active development, check out the documentation on [installing a development package](/docs/how-tos/develop-local-packages).

docs/sidebars.js

@@ -80,6 +80,8 @@ module.exports = {
         "how-tos/telemetry",
         "how-tos/monitoring",
         "how-tos/use-gpus",
+        "how-tos/develop-local-packages",
+        "how-tos/install-pip-packages",
         "how-tos/fine-grained-permissions",
       ],
     },