From 001ea87ca69f43f16802c78de04969c0c197566a Mon Sep 17 00:00:00 2001 From: "Dr. K. D. Murray" <1560490+kdm9@users.noreply.github.com> Date: Sun, 3 Jul 2022 17:11:21 +1000 Subject: [PATCH 1/3] Add notes on error-tolerant execution to CLI docs --- docs/using-cli.md | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-) diff --git a/docs/using-cli.md b/docs/using-cli.md index 2f21985e4..78501e65a 100644 --- a/docs/using-cli.md +++ b/docs/using-cli.md @@ -26,13 +26,6 @@ jupytext --set-formats ipynb,py notebook.ipynb # Turn notebook.ipynb into a pai jupytext --sync notebook.ipynb # Update whichever of notebook.ipynb/notebook.py is outdated ``` -For convenience, when creating a notebook from text you can execute it: -```bash -jupytext --set-kernel - notebook.md # create a YAML header with kernel metadata matching the current python executable -jupytext --set-formats md:myst notebook.md # create a YAML header with an explicit jupytext format -jupytext --to notebook --execute notebook.md # convert notebook.md to an .ipynb file and run it -``` - If you wanted to convert a collection of Markdown files to paired notebooks, and execute them in the current Python environment, you could run: ```bash jupytext --set-formats ipynb,md --execute *.md @@ -58,6 +51,32 @@ Note also that on Windows you need to use double quotes instead of single quotes Execute `jupytext --help` to access the full documentation. +### Execute notebook cells + +For convenience, when creating a notebook from text you can execute it: + +```bash +jupytext --set-kernel - notebook.md # create a YAML header with kernel metadata matching the current python executable +jupytext --set-formats md:myst notebook.md # create a YAML header with an explicit jupytext format +jupytext --to notebook --execute notebook.md # convert notebook.md to an .ipynb file and run it +``` + +Please note: if any notebook cell errors, execution will terminate and `jupytext` will not save the notebook. This can cause headaches as the details of any error would be encoded in the notebook that isn't saved. But there's a way: `jupyter nbconvert` has a mode which will still save a notebook if a cell errors, producing something akin to what would happen if you ran all cells manually in Jupyter's notebook UI. + +```bash +# First, convert script (py/sh/R/jl etc) -> notebook. May need additional args to define input format etc as above. +jupytext --to ipynb script.sh +# Then, execute notebook in place and allowing cells to produce errors +jupyter nbconvert --to ipynb --inplace --execute --allow-errors script.ipynb +# One can also combine these to a single command using jupytext --pipe +jupytext --to ipynb --pipe-fmt ipynb \ + --pipe 'jupyter nbconvert --to ipynb --execute --allow-errors --stdin --stdout' \ + script.sh +``` + +In each of the above, `jupyter nbconvert` could be replaced with any alternative tool to execute a jupyter notebook non-interactively, including [papermill](https://github.com/nteract/papermill) which would allow notebook parameterisation. + + ## Notebook and cell metadata filters If you want to preserve (or filter out) certain notebook or cell metadata, change the value of either `notebook_metadata_filter` or `cell_metadata_filter` with the `--update-metadata` option. For instance, if you wish to convert an `.ipynb` document to a `.md` file and preserve all the notebook metadata in that document, run From 1161311ed012d1bcbe908d6c06ef3568a5bf4101 Mon Sep 17 00:00:00 2001 From: "Dr. K. D. Murray" <1560490+kdm9@users.noreply.github.com> Date: Sun, 3 Jul 2022 17:55:30 +1000 Subject: [PATCH 2/3] move all execute-related things to the execute heading --- docs/using-cli.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/using-cli.md b/docs/using-cli.md index 78501e65a..191881641 100644 --- a/docs/using-cli.md +++ b/docs/using-cli.md @@ -26,11 +26,6 @@ jupytext --set-formats ipynb,py notebook.ipynb # Turn notebook.ipynb into a pai jupytext --sync notebook.ipynb # Update whichever of notebook.ipynb/notebook.py is outdated ``` -If you wanted to convert a collection of Markdown files to paired notebooks, and execute them in the current Python environment, you could run: -```bash -jupytext --set-formats ipynb,md --execute *.md -``` - You may also find useful to `--pipe` the text representation of a notebook into tools like `black`: ```bash jupytext --sync --pipe black notebook.ipynb # read most recent version of notebook, reformat with black, save @@ -61,6 +56,11 @@ jupytext --set-formats md:myst notebook.md # create a YAML header with an e jupytext --to notebook --execute notebook.md # convert notebook.md to an .ipynb file and run it ``` +If you wanted to convert a collection of Markdown files to paired notebooks, and execute them in the current Python environment, you could run: +```bash +jupytext --set-formats ipynb,md --execute *.md +``` + Please note: if any notebook cell errors, execution will terminate and `jupytext` will not save the notebook. This can cause headaches as the details of any error would be encoded in the notebook that isn't saved. But there's a way: `jupyter nbconvert` has a mode which will still save a notebook if a cell errors, producing something akin to what would happen if you ran all cells manually in Jupyter's notebook UI. ```bash From d64678dfa70f3f0ff93cb9a1162d75816116ed3c Mon Sep 17 00:00:00 2001 From: "Dr. K. D. Murray" <1560490+kdm9@users.noreply.github.com> Date: Mon, 4 Jul 2022 01:12:09 +1000 Subject: [PATCH 3/3] Incorporate @mwouts' suggestions --- docs/using-cli.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/using-cli.md b/docs/using-cli.md index 191881641..f02c9fabf 100644 --- a/docs/using-cli.md +++ b/docs/using-cli.md @@ -61,20 +61,22 @@ If you wanted to convert a collection of Markdown files to paired notebooks, and jupytext --set-formats ipynb,md --execute *.md ``` -Please note: if any notebook cell errors, execution will terminate and `jupytext` will not save the notebook. This can cause headaches as the details of any error would be encoded in the notebook that isn't saved. But there's a way: `jupyter nbconvert` has a mode which will still save a notebook if a cell errors, producing something akin to what would happen if you ran all cells manually in Jupyter's notebook UI. +#### Advanced usage: error tolerance + +If any notebook cell errors, execution will terminate and `jupytext` will not save the notebook. This can cause headaches as the details of any error would be encoded in the notebook, which would not have been saved. But there's an error-tolerant way to execute a notebook: `jupyter nbconvert` has a mode which will still save a notebook if a cell errors, producing something akin to what would happen if you ran all cells manually in Jupyter's notebook UI. ```bash # First, convert script (py/sh/R/jl etc) -> notebook. May need additional args to define input format etc as above. -jupytext --to ipynb script.sh +jupytext --to ipynb script.py # Then, execute notebook in place and allowing cells to produce errors jupyter nbconvert --to ipynb --inplace --execute --allow-errors script.ipynb # One can also combine these to a single command using jupytext --pipe jupytext --to ipynb --pipe-fmt ipynb \ --pipe 'jupyter nbconvert --to ipynb --execute --allow-errors --stdin --stdout' \ - script.sh + script.py ``` -In each of the above, `jupyter nbconvert` could be replaced with any alternative tool to execute a jupyter notebook non-interactively, including [papermill](https://github.com/nteract/papermill) which would allow notebook parameterisation. +In each of the above, `jupyter nbconvert` could be replaced with any alternative tool to execute a jupyter notebook non-interactively, including [papermill](https://github.com/nteract/papermill) which would allow notebook parameterisation (see [@mwouts' post on the topic here](https://github.com/CFMTech/jupytext_papermill_post/blob/master/README.md)). ## Notebook and cell metadata filters