From 62b1c68f08188a506ffe93aaf48b781a753b21bb Mon Sep 17 00:00:00 2001 From: = Date: Wed, 28 Feb 2024 16:40:11 -0700 Subject: [PATCH 1/8] initial commit: updated docs for kestrel, cleanup text --- .../Development/Jupyter/index.md | 137 +++++++++++------- 1 file changed, 87 insertions(+), 50 deletions(-) diff --git a/docs/Documentation/Development/Jupyter/index.md b/docs/Documentation/Development/Jupyter/index.md index 324c28355..b265dca31 100644 --- a/docs/Documentation/Development/Jupyter/index.md +++ b/docs/Documentation/Development/Jupyter/index.md @@ -12,7 +12,9 @@ - Customizable - Now supports other languages besides Python (R, Julia..) - https://github.com/jupyter/jupyter/wiki/Jupyter-kernels - - these slides were created using Markdown in Jupyter! + + +## Example Notebook Code ```python import chart_studio.plotly as py @@ -46,14 +48,13 @@ axes[3].set_title("fill_between"); ![png](../../../../assets/images/output_4_0.png) -## Terminology - a Confusion of Words - +## Jupyter Terminology ### **Jupyterhub** * Multi-user "backend" server * Controls launching the single-user Jupyter server - * NREL's "Europa" runs Jupyterhub + * NREL's "Europa" (Eagle-only) runs Jupyterhub (In general, don't worry about JupyterHub--unless you're a sysadmin) @@ -78,7 +79,11 @@ axes[3].set_title("fill_between"); ## Using Europa -We run a Jupyterhub server that is available. +The NREL HPC team runs a Jupyterhub server, called Europa, that is available for internal (NREL) Eagle users only. + +Europa is connected to Eagle's Lustre storage system for access to /projects data. + +A replacement for Europa on Kestrel is in the planning stage. ### Europa's Advantages: * Fast and easy access @@ -88,8 +93,7 @@ We run a Jupyterhub server that is available. ### Europa's Disadvantages: * Limited resource: 8 cores/128GB RAM per user beefore automatic throttling * Compete with other users for CPU/RAM on a single machine - * No custom environments (for now) - + * Not available for Kestrel (yet) ### Simple Instructions: @@ -113,55 +117,99 @@ We run a Jupyterhub server that is available. * Compete with other users for nodes * Costs AU - +## Launching Your Own Jupyter Server on an HPC System + +Both Kestrel and Eagle support running your own Jupyter Notebook server. + +External (non-NREL) **Kestrel** users may follow the directions below for Kestrel, but please use `kestrel.nrel.gov` instead of `kestrel.hpc.nrel.gov`. + +External (non-NREL) **Eagle** users will no longer be able to use Jupyter in this fashion as of February 2024. If you require Jupyter, please consider transitioning to Kestrel as soon as possible. + +## Using a Compute Node to run Jupyter Notebooks + +Connect to a login node and request an interactive job using the `salloc` command. + +The examples below will start a 2-hour job. Edit the `` to the name of your allocation, and adjust the time accordingly. Since these are interactive jobs, they will get some priority, especially if they're shorter, so only book as much time as you will be actively working on the notebook. +Before you get started, we recommend installing your own Jupyter inside of a conda environment. The default conda/anaconda3 modules contain basic Jupyter Notebook servers, but you will likely want your own Python libraries, notebook extensions, and other features. Basic directions are included later in this document. -#### ...Is more than one node possible? +### Kestrel: -Yes... please see our advanced Jupyter documentation. +`[user@laptop:~]$ ssh kestrel.hpc.nrel.gov` -## Using a Compute Node - Hard Way +`[user@kl1:~]$ salloc -A -t 02:00:00` -### Start on a login node: +### Eagle: -`ssh eagle.hpc.nrel.gov` +`[user@laptop:~]$ ssh eagle.hpc.nrel.gov` -`[user@el1:]$ srun -A -t 02:00:00 --pty /bin/bash` +`[user@el1:~]$ salloc -A -t 02:00:00` +## Starting Jupyter Inside the Job +Once the job starts and you are allocated a compute node, load the appropriate modules, activate your Jupyter environment, and launch the Jupyter server. +#### Kestrel: -### When the job starts on the compute node: +`[user@x1000c0s0b0n1:~]$ module load anaconda3` -`[user@r2i7n35]$ module load conda` +`[user@x1000c0s0b0n1:~]$ source activate myjupenv` + +`[user@x1000c0s0b0n1:~]$ jupyter-notebook --no-browser --ip=$(hostname -s)` + +Take note of the node name that your job is assigned. (x1000c0s0b0n1 in the above example.) + +Also note the url that Jupyter displays when starting up, e.g. `http://127.0.0.1:8888/?token=`. + +The `` is a long string of letters and numbers. This is a unique authorization token for your Jupyter session. you will need it, along with the full URL, for a later step. + +#### Eagle: + +`[user@r2i7n35:~]$ module load conda` `source activate myjupenv` `jupyter-notebook --no-browser --ip=$(hostname -s)` -note the node name (r2i7n35 in this example) +Take note of the node name that your job is assigned. (r2i7n35 in this example.) -and the url, e.g. `http://127.0.0.1:8888/?token=` +Also note the url that Jupyter displays when starting up, e.g. `http://127.0.0.1:8888/?token=`. -### In a terminal on your computer: +The `` is a long string of letters and numbers. This is a unique authorization token for your Jupyter session. you will need it, along with the full URL, for a later step. -`[user@laptop]$ ssh -N -L 8888::8888 username@eagle.hpc.nrel.gov` - -copy full url from jupyter startup into your web browser. e.g.: +### On Your Own Computer + +Next, open an SSH tunnel through a login node to the compute node. Log in when prompted using your regular HPC credentials, and put this terminal to the side or minimize it, but leave it open until you are done working with Jupyter for this session. + +#### Kestrel: + +`[user@laptop:~]$ ssh -N -L 8888::8888 username@eagle.hpc.nrel.gov` + + +#### Eagle: + +`[user@laptop:~]$ ssh -N -L 8888::8888 username@eagle.hpc.nrel.gov` + + +### Open a Web Browser + +Copy the full url and token from Jupyter startup into your web browser. For example: `http://127.0.0.1:8888/?token=` -## Using a Compute Node - Easy Way +## Using a Compute Node on Eagle - Easy Way + +Scripted assistance with launching a Jupyter session on Eagle is available. -Automation makes life better! +These scripts are designed for Eagle and have not yet been adapted for Kestrel, but may be downloaded and adapted manually. ### pyeagle - NREL Users The [pyeagle](https://github.nrel.gov/MBAP/pyeagle) package is available for internal users to handle launching and monitoring a jupyter server on a compute node. This package is maintained by an NREL HPC user group, and provides utilities for working on Eagle and Kestrel. -### Auto-launching with an sbatch script - All Users +### Auto-launching on Eagle With an sbatch Script Full directions included in the [Jupyter repo](https://github.com/NREL/HPC/tree/master/general/Jupyterhub/jupyter). @@ -179,16 +227,10 @@ That's it! ## Using a Login Node - -Yes, you can run jupyter directly on a login node. - - - -Should you run jupyter directly on a login node? - - +Please avoid running Jupyter on a login node on either Kestrel or Eagle. ### Reasons to Not Run Jupyter Directly on a Login Node + * Heavy lifting should be done via Europa or compute nodes * Using a highly shared resource (login nodes) * Competition for cycles @@ -196,7 +238,13 @@ Should you run jupyter directly on a login node? ## Custom Conda Environments and Jupyter Kernels -### Creating a conda environment: +On Kestrel, the module 'anaconda3' is available for the conda environment manager. As an alternative, the module 'mamba' is available as well. Mamba is conda-compatible and has a faster solver. + +On Eagle, the module 'conda' contains the conda environment manager. The Eagle conda module also contains mamba installed as a conda package. + +### Creating a Conda Environment + +Create an environment and install the base jupyter packages, then activate the environment and install other libraries, e.g. scipy, numpy, and so on. `conda create -n myjupyter -c conda-forge jupyter ipykernel` @@ -204,25 +252,22 @@ Should you run jupyter directly on a login node? `conda install -c conda-forge scipy numpy matplotlib` - - -### Add custom ipykernel +### Add Custom iPykernel `python -m ipykernel install --user --name=myjupyter` -Restart your jupyter server +If you already have a Jupyter server running, restart it to load the new kernel. -New kernel will appear in drop-down as an option +The new kernel will appear in the drop-down as an option to open a new notebook. -### Remove custom ipykernel +### Remove Custom iPykernel `jupyter kernelspec list` `jupyter kernelspec remove myoldjupyter` -## Magic commands - +## Magic Commands Magic commands are "meta commands" that add extra functionality. @@ -235,17 +280,11 @@ Magic commands begin with % or %%. * %%time - placed at top of cell, prints execution time * %who - list all defined variables in notebook - - - ``` %lsmagic ``` - - - Available line magics: %alias %alias_magic %autoawait %autocall %automagic %autosave %bookmark %cat %cd %clear %colors %conda %config %connect_info %cp %debug %dhist %dirs %doctest_mode %ed %edit %env %gui %hist %history %killbgscripts %ldir %less %lf %lk %ll %load %load_ext %loadpy %logoff %logon %logstart %logstate %logstop %ls %lsmagic %lx %macro %magic %man %matplotlib %mkdir %more %mv %notebook %page %pastebin %pdb %pdef %pdoc %pfile %pinfo %pinfo2 %pip %popd %pprint %precision %prun %psearch %psource %pushd %pwd %pycat %pylab %qtconsole %quickref %recall %rehashx %reload_ext %rep %rerun %reset %reset_selective %rm %rmdir %run %save %sc %set_env %store %sx %system %tb %time %timeit %unalias %unload_ext %who %who_ls %whos %xdel %xmode @@ -254,8 +293,6 @@ Magic commands begin with % or %%. Automagic is ON, % prefix IS NOT needed for line magics. - - ## Shell Commands You can also run shell commands inside a cell. For example: From 94e481d8ecad2d75555a3c98fad00e3f18afbdf1 Mon Sep 17 00:00:00 2001 From: = Date: Fri, 1 Mar 2024 14:22:50 -0700 Subject: [PATCH 2/8] improvements --- .../Development/Jupyter/index.md | 141 ++++++++++-------- 1 file changed, 77 insertions(+), 64 deletions(-) diff --git a/docs/Documentation/Development/Jupyter/index.md b/docs/Documentation/Development/Jupyter/index.md index b265dca31..9d3560844 100644 --- a/docs/Documentation/Development/Jupyter/index.md +++ b/docs/Documentation/Development/Jupyter/index.md @@ -5,17 +5,21 @@ **A web app for interactive Python in a browser** -- "Live coding" -- Instant visualization -- Sharable -- Reproducible -- Customizable -- Now supports other languages besides Python (R, Julia..) - - https://github.com/jupyter/jupyter/wiki/Jupyter-kernels +Jupyter offers a number of benefits for researchers in many fields, including: + +- Live coding: Make changes and see the effects in real-time. +- Instant visualization: Charts and graphics render quickly in a browser window. +- Sharable: Notebooks can be copied and sent to others, or multiple users can edit a single shared notebook. +- Reproducible: Create a shareable environment with pinned Python and scientific library versions. +- Customizable: Many configuration options, extensions, and libraries are available. +- Not just for Python: Supports many other languages (including R, Julia, and many others.) + - See [https://github.com/jupyter/jupyter/wiki/Jupyter-kernels](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels) for examples. ## Example Notebook Code +With the appropriate libraries installed into the Jupyet environment, the following code can be placed in one cell in a notebook, or split across multiple cells, and executed to produce quick graphs: + ```python import chart_studio.plotly as py import plotly.figure_factory as ff @@ -52,74 +56,77 @@ axes[3].set_title("fill_between"); ### **Jupyterhub** - * Multi-user "backend" server - * Controls launching the single-user Jupyter server - * NREL's "Europa" (Eagle-only) runs Jupyterhub - -(In general, don't worry about JupyterHub--unless you're a sysadmin) + +This is the multi-user "backend" server. The "Hub" allows users to login, then launches the single-user Jupyter server for them. Hubs are usually installed and managed by system administrators, not Jupyter users. + +NREL's "Europa" (Eagle-only) runs Jupyterhub. More on Europa later in this document. ### **Jupyter/Jupyter Server/Notebook server** - * The single-user server/web interface - * Create/save/load .ipynb notebook files - * What users generally interact with + +The single-user server/web interface. Use to create, save, or load .ipynb notebook files. This is what users generally interact with. ### **Jupyter Notebook** - * An individual .pynb file - * Contains your Python code and visualizations - * Sharable/downloadable + +A Notebook is an individual .pynb file. It contains your Python code and visualizations, and is sharable/downloadable. ### **Jupyter lab** - * A "nicer" web interface for Jupyter - "notebooks 2.0" - * Preferred by some - * Lacking some features of "classic" notebooks + +A "nicer" redesigned web interface for your Jupyter Server - "Notebooks 2.0". Preferred by some, and promoted as the next evolution of Notebooks. +Lab has many new and different extensions, but many are also not compatible between Notebook and Lab. Lab is still under development, so is lacking some features of "classic" notebooks. ### **Kernel** - * The Python environment used by a notebook - * More on kernels later -## Using Europa +Kernels define the Python environments used by your notebooks. Derived from ipykernel, a predecessor project to Jupyter: you may see Jupyter kernels referred to as "ipykernels". Custom kernels require the "ipykernel" package installed in your Jupyter conda environment. + +More on kernels later. -The NREL HPC team runs a Jupyterhub server, called Europa, that is available for internal (NREL) Eagle users only. +## Eagle's "Europa" Jupyterhub Server + +The NREL HPC team runs a Jupyterhub server called Europa that is available for internal (NREL) Eagle users only. Europa is connected to Eagle's Lustre storage system for access to /projects data. A replacement for Europa on Kestrel is in the planning stage. ### Europa's Advantages: - * Fast and easy access - * Use regular Eagle credentials - * Great for light to moderate processing/debugging/testing + +* Fast and easy access to notebooks with no setup. +* Use regular Eagle credentials to log in. +* Great for simple tasks, including light to moderate data processing, code debugging/testing, and light to moderate visualization using standard/basic scientific and visualization libraries. ### Europa's Disadvantages: - * Limited resource: 8 cores/128GB RAM per user beefore automatic throttling - * Compete with other users for CPU/RAM on a single machine - * Not available for Kestrel (yet) -### Simple Instructions: +* Limited resources: Only 48 CPU cores and 190GB RAM total. +* Managed usage: Up to 8 cores/128GB RAM per user before automatic throttling will greatly slow down processing. +* Must compete with other users for CPU and RAM on a single machine. +* Limited list of scientific libraries and visualization tools are available, and may not be latest versions. +* Custom environments are difficult to configure. +* No access for external (non-NREL) users. +* Not available for Kestrel (yet). + +### Simple Instructions to access Europa: - - Visit Europa at (https://europa.hpc.nrel.gov/) - - - Log in using your HPC credentials - - - Opens a standard "notebooks" interface - - - Change url end /tree to /lab for Lab interface +* Visit Europa at (https://europa.hpc.nrel.gov/) in a web browser and log in using your HPC credentials. + +Europa opens a standard "notebooks" interface by default. Change the url ending from "/tree" to "/lab" in your web browser to use the Jupyter Lab interface, if preferred. -## Using a Compute Node +## Using a Compute Node to Run Your Own Jupyter Notebooks ### Advantages: - * Custom environments - * 36 cores and up to ~750GB RAM - * No competing with other users for cores + +* Custom conda environments to load preferred libraries. +* Full node usage: Exclusive access to the resources of the node your job is reserved on, including up to 36 CPU cores and up to ~750GB RAM on Eagle bigmem nodes, and up to 104 CPU cores and up to ~2TB RAM on Kestrel bigmem nodes. See the system specifications page for the cluster you are working on. +* No competing with other users for CPU cores and RAM, and no Arbiter2 process throttling. ### Disadvantages: - * Compete with other users for nodes - * Costs AU + +* Must compete with other users for a node via the job queue. +* Costs your allocation AU. ## Launching Your Own Jupyter Server on an HPC System -Both Kestrel and Eagle support running your own Jupyter Notebook server. +Both Kestrel and Eagle support running your own Jupyter Notebook server. This is highly recommended over Europa for advanced Jupyter use and heavy computational processing. External (non-NREL) **Kestrel** users may follow the directions below for Kestrel, but please use `kestrel.nrel.gov` instead of `kestrel.hpc.nrel.gov`. @@ -205,7 +212,7 @@ Scripted assistance with launching a Jupyter session on Eagle is available. These scripts are designed for Eagle and have not yet been adapted for Kestrel, but may be downloaded and adapted manually. -### pyeagle - NREL Users +### Internal NREL Users only: pyeagle The [pyeagle](https://github.nrel.gov/MBAP/pyeagle) package is available for internal users to handle launching and monitoring a jupyter server on a compute node. This package is maintained by an NREL HPC user group, and provides utilities for working on Eagle and Kestrel. @@ -225,26 +232,24 @@ Run [auto_launch_jupyter.sh](https://github.com/NREL/HPC/blob/master/general/Jup That's it! -## Using a Login Node +## Reasons to Not Run Jupyter Directly on a Login Node -Please avoid running Jupyter on a login node on either Kestrel or Eagle. - -### Reasons to Not Run Jupyter Directly on a Login Node - - * Heavy lifting should be done via Europa or compute nodes - * Using a highly shared resource (login nodes) - * Competition for cycles - * arbiter2 will throttle moderate to heavy usage +* Data processing and visualization should be done via Europa or compute nodes. +* Uses a highly shared resource (login nodes): there will be competition for CPU, RAM, and network I/O for storage. Arbiter2 software will automatically throttle moderate to heavy usage on login nodes, greatly slowing down processing. ## Custom Conda Environments and Jupyter Kernels -On Kestrel, the module 'anaconda3' is available for the conda environment manager. As an alternative, the module 'mamba' is available as well. Mamba is conda-compatible and has a faster solver. +On Kestrel, the module 'anaconda3' is available to run the conda command and manage your environments. + +As an alternative, the module 'mamba' is available instead. Mamba is a conda-compatible environment manager with very similar usage. Most conda command in this documentation may be used with mamba instead and they may generally be considered interchangeable. -On Eagle, the module 'conda' contains the conda environment manager. The Eagle conda module also contains mamba installed as a conda package. +On Eagle, the module 'conda' contains the conda command. The Eagle conda module also contains mamba installed as a conda package. ### Creating a Conda Environment -Create an environment and install the base jupyter packages, then activate the environment and install other libraries, e.g. scipy, numpy, and so on. +To add your own packages to conda on Kestrel or Eagle: + +Create an environment and install the base jupyter packages. Then activate the environment and install other libraries that you want to use, e.g. scipy, numpy, and so on. `conda create -n myjupyter -c conda-forge jupyter ipykernel` @@ -254,26 +259,34 @@ Create an environment and install the base jupyter packages, then activate the e ### Add Custom iPykernel +A kernel is what allows Jupyter to use your customized conda environment inside Jupyter, in a notebook. Use ipykernel to build your kernel. Inside your custom conda environment, run: + `python -m ipykernel install --user --name=myjupyter` If you already have a Jupyter server running, restart it to load the new kernel. The new kernel will appear in the drop-down as an option to open a new notebook. -### Remove Custom iPykernel +You can have multiple kernels, allowing you to load different conda environments for your different projects into Notebooks. + +### Jupyter Kernel Management + +Use the kernelspec list command inside your Jupyter conda environment to see what ipykernels you have installed: `jupyter kernelspec list` +To remove an old kernel, use the kernelspec remove command: + `jupyter kernelspec remove myoldjupyter` ## Magic Commands -Magic commands are "meta commands" that add extra functionality. +Magic commands are "meta commands" that add extra functionality to Jupyter. Magic commands begin with % or %%. -### A Few Useful Examples +### Example Magic Commands * %lsmagic - list all magic commands * %run _file.py_ - run an external python script @@ -305,14 +318,14 @@ You can also run shell commands inside a cell. For example: !ls ``` - /home/tthatche/jup + /home/username/jup auto_launch_jupyter.sh Jupyter Presentation.ipynb slurm-6445885.out geojsondemo.ipynb old sshot1.png Interesting Graphs.ipynb sbatch_jupyter.sh sshot2.png jup-logo.png slurm -## Interesting/Useful Notebooks +## Interesting/Useful Notebooks, Extensions, and Learning Resources [Awesome Jupyter](https://github.com/markusschanta/awesome-jupyter) From 7ffaff5ce5c0a4bc225b36633ee6463f28a3737f Mon Sep 17 00:00:00 2001 From: Tim T Date: Wed, 6 Mar 2024 08:48:02 -0700 Subject: [PATCH 3/8] Update docs/Documentation/Development/Jupyter/index.md Co-authored-by: Haley Yandt <46908710+yandthj@users.noreply.github.com> --- docs/Documentation/Development/Jupyter/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Development/Jupyter/index.md b/docs/Documentation/Development/Jupyter/index.md index 9d3560844..157a71ace 100644 --- a/docs/Documentation/Development/Jupyter/index.md +++ b/docs/Documentation/Development/Jupyter/index.md @@ -18,7 +18,7 @@ Jupyter offers a number of benefits for researchers in many fields, including: ## Example Notebook Code -With the appropriate libraries installed into the Jupyet environment, the following code can be placed in one cell in a notebook, or split across multiple cells, and executed to produce quick graphs: +With the appropriate libraries installed into the Jupyter environment, the following code can be placed in one cell in a notebook, or split across multiple cells, and executed to produce quick graphs: ```python import chart_studio.plotly as py From 70d3730528504d0e00985cb8c4c469d0b766e2e2 Mon Sep 17 00:00:00 2001 From: Tim T Date: Wed, 6 Mar 2024 08:48:14 -0700 Subject: [PATCH 4/8] Update docs/Documentation/Development/Jupyter/index.md Co-authored-by: Haley Yandt <46908710+yandthj@users.noreply.github.com> --- docs/Documentation/Development/Jupyter/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Development/Jupyter/index.md b/docs/Documentation/Development/Jupyter/index.md index 157a71ace..36f28241f 100644 --- a/docs/Documentation/Development/Jupyter/index.md +++ b/docs/Documentation/Development/Jupyter/index.md @@ -191,7 +191,7 @@ Next, open an SSH tunnel through a login node to the compute node. Log in when p #### Kestrel: -`[user@laptop:~]$ ssh -N -L 8888::8888 username@eagle.hpc.nrel.gov` +`[user@laptop:~]$ ssh -N -L 8888::8888 username@kestrel.hpc.nrel.gov` #### Eagle: From 304cb594b82ed969bb262ee5011ad7f2da8ea8fc Mon Sep 17 00:00:00 2001 From: Tim T Date: Wed, 6 Mar 2024 08:48:25 -0700 Subject: [PATCH 5/8] Update docs/Documentation/Development/Jupyter/index.md Co-authored-by: Haley Yandt <46908710+yandthj@users.noreply.github.com> --- docs/Documentation/Development/Jupyter/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Documentation/Development/Jupyter/index.md b/docs/Documentation/Development/Jupyter/index.md index 36f28241f..663fae2a2 100644 --- a/docs/Documentation/Development/Jupyter/index.md +++ b/docs/Documentation/Development/Jupyter/index.md @@ -241,7 +241,7 @@ That's it! On Kestrel, the module 'anaconda3' is available to run the conda command and manage your environments. -As an alternative, the module 'mamba' is available instead. Mamba is a conda-compatible environment manager with very similar usage. Most conda command in this documentation may be used with mamba instead and they may generally be considered interchangeable. +As an alternative, the module 'mamba' is available instead. Mamba is a conda-compatible environment manager with very similar usage. Most conda commands in this documentation may be used with mamba instead and they may generally be considered interchangeable. On Eagle, the module 'conda' contains the conda command. The Eagle conda module also contains mamba installed as a conda package. From 19823f379b449e37d930b51cbd3eb03e46a6d9d4 Mon Sep 17 00:00:00 2001 From: = Date: Wed, 6 Mar 2024 09:25:59 -0700 Subject: [PATCH 6/8] minor fix for pyeagle/scripts section --- docs/Documentation/Development/Jupyter/index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/Documentation/Development/Jupyter/index.md b/docs/Documentation/Development/Jupyter/index.md index 663fae2a2..9326c1a5a 100644 --- a/docs/Documentation/Development/Jupyter/index.md +++ b/docs/Documentation/Development/Jupyter/index.md @@ -208,9 +208,8 @@ Copy the full url and token from Jupyter startup into your web browser. For exam ## Using a Compute Node on Eagle - Easy Way -Scripted assistance with launching a Jupyter session on Eagle is available. +Scripted assistance with launching a Jupyter session on Eagle or Kestrel is available. -These scripts are designed for Eagle and have not yet been adapted for Kestrel, but may be downloaded and adapted manually. ### Internal NREL Users only: pyeagle @@ -218,6 +217,8 @@ The [pyeagle](https://github.nrel.gov/MBAP/pyeagle) package is available for int ### Auto-launching on Eagle With an sbatch Script +These scripts are designed for Eagle and may not yet be adapted for Kestrel, but may be downloaded and adapted manually. + Full directions included in the [Jupyter repo](https://github.com/NREL/HPC/tree/master/general/Jupyterhub/jupyter). Download [sbatch_jupyter.sh](https://github.com/NREL/HPC/blob/master/general/Jupyterhub/jupyter/sbatch_jupyter.sh) and [auto_launch_jupyter.sh](https://github.com/NREL/HPC/blob/master/general/Jupyterhub/jupyter/auto_launch_jupyter.sh) From 4d777b3f3efe26c2965f04dc3379e2210ae07bed Mon Sep 17 00:00:00 2001 From: = Date: Wed, 6 Mar 2024 09:30:38 -0700 Subject: [PATCH 7/8] minor fix for pyeagle/scripts section 2 --- docs/Documentation/Development/Jupyter/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/Documentation/Development/Jupyter/index.md b/docs/Documentation/Development/Jupyter/index.md index 9326c1a5a..7cc83aba7 100644 --- a/docs/Documentation/Development/Jupyter/index.md +++ b/docs/Documentation/Development/Jupyter/index.md @@ -206,12 +206,12 @@ Copy the full url and token from Jupyter startup into your web browser. For exam `http://127.0.0.1:8888/?token=` -## Using a Compute Node on Eagle - Easy Way +## Using a Compute Node - The Easy Way Scripted assistance with launching a Jupyter session on Eagle or Kestrel is available. -### Internal NREL Users only: pyeagle +### Internal NREL Users Only: pyeagle The [pyeagle](https://github.nrel.gov/MBAP/pyeagle) package is available for internal users to handle launching and monitoring a jupyter server on a compute node. This package is maintained by an NREL HPC user group, and provides utilities for working on Eagle and Kestrel. From 574e9d544e70103c561de1f58eff08e53b9c6cf7 Mon Sep 17 00:00:00 2001 From: = Date: Wed, 6 Mar 2024 15:28:20 -0700 Subject: [PATCH 8/8] remove outdated jhub page/mkdocs link --- .../Development/Jupyter/jupyterhub.md | 45 ------------------- mkdocs.yml | 1 - 2 files changed, 46 deletions(-) delete mode 100644 docs/Documentation/Development/Jupyter/jupyterhub.md diff --git a/docs/Documentation/Development/Jupyter/jupyterhub.md b/docs/Documentation/Development/Jupyter/jupyterhub.md deleted file mode 100644 index d2ffae218..000000000 --- a/docs/Documentation/Development/Jupyter/jupyterhub.md +++ /dev/null @@ -1,45 +0,0 @@ ---- -layout: default -title: JupyterHub -parent: Jupyter ---- - -# JupyterHub -Prior to using Jupyterhub, you will have had to have logged into Eagle via the command line at least once. - -Given that, to start using Jupyterhub on Eagle, go to [Europa](https://europa.hpc.nrel.gov) in your local machine's browser, and log in with your Eagle username and password. -You should land in your home directory, and see everything there via the standard Jupyter file listing. - -From the "New" pulldown on the right hand side, you can start a notebook, open a terminal, or create a file or folder. -The default installation is Python version 3, and a variety of Conda modules are installed already. You can start a -Python3 notebook right away, and access the Python modules that are already present. To see what's installed, from a notebook -you can use the following command: - -``` -!conda list -``` - -Alternatively, you can start a Terminal, and use the usual conda commands from the shell. - -## Creating a custom environment to access from the notebook - -Start a Terminal session, and follow the instructions on the [HPC website](https://www.nrel.gov/hpc/eagle-software-python.html) -to create an environment. Now, to make this environment visible from your future notebooks, run the following command: - -``` -source activate -python -m ipykernel install --user --name --display-name "How-you-want-your-custom-kernel-to-appear-in-the-notebook-pulldown ()" -``` - -where `` is the argument to `-n` you used in your `conda create` command. - -After running this command, when you open a new notebook, you should see as an option your new environment, and once loaded -be able to access all Python modules therein. - -## Using Jupyterhub from Eagle - -To use inside Eagle, the Jupyterhub server exists on the internal network @ https://europa-int/. - -## Customizing - -JupyterHub provides the ability to use custom kernels including ones for other popular programming languages such as Julia and R. [NREL's custom kernels documentation](https://www.nrel.gov/hpc/jupyterhub.html) provides more information on how to setup JupyterHub with other languages. \ No newline at end of file diff --git a/mkdocs.yml b/mkdocs.yml index 7e0c60dcc..b962cacc4 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -116,7 +116,6 @@ nav: - Documentation/Development/Languages/Python/dask.md - Jupyter: - Documentation/Development/Jupyter/index.md - - JupyterHub: Documentation/Development/Jupyter/jupyterhub.md - Programming Models: # - Documentation/Development/Programming_Models/index.md - MPI: Documentation/Development/Programming_Models/mpi.md