Add notes on error-tolerant execution to CLI docs #972
Conversation
docs/using-cli.md
Outdated
| jupytext --set-formats ipynb,md --execute *.md | ||
| ``` | ||
|
|
||
| Please note: if any notebook cell errors, execution will terminate and `jupytext` will not save the notebook. This can cause headaches as the details of any error would be encoded in the notebook that isn't saved. But there's a way: `jupyter nbconvert` has a mode which will still save a notebook if a cell errors, producing something akin to what would happen if you ran all cells manually in Jupyter's notebook UI. |
There was a problem hiding this comment.
I agree that having the notebook fail is an unpleasant experience, yet failing on error is also nbconvert's standard, so I'd rather introduce the next paragraph as something like advanced notebook execution. What do you think?
docs/using-cli.md
Outdated
|
|
||
| ```bash | ||
| # First, convert script (py/sh/R/jl etc) -> notebook. May need additional args to define input format etc as above. | ||
| jupytext --to ipynb script.sh |
There was a problem hiding this comment.
I like this but I am afraid most users convert .py scripts to notebooks... maybe the first example can take a .py script, and then recall that .sh scripts can also be converted to Bash notebooks (do you execute them with bash_kernel by the way?)
There was a problem hiding this comment.
I'll s/.sh/.py then :)
Also, yes, with bash_kernel (though it seems the latest release & HEAD are broken, see takluyver/bash_kernel#120)
| jupyter nbconvert --to ipynb --inplace --execute --allow-errors script.ipynb | ||
| # One can also combine these to a single command using jupytext --pipe | ||
| jupytext --to ipynb --pipe-fmt ipynb \ | ||
| --pipe 'jupyter nbconvert --to ipynb --execute --allow-errors --stdin --stdout' \ |
docs/using-cli.md
Outdated
| script.sh | ||
| ``` | ||
|
|
||
| In each of the above, `jupyter nbconvert` could be replaced with any alternative tool to execute a jupyter notebook non-interactively, including [papermill](https://github.com/nteract/papermill) which would allow notebook parameterisation. |
There was a problem hiding this comment.
In my experience, papermill is a great tool, and by default it catches execution errors. May I ask if you have reasons not to use it in your personal workflow?
Also would you mind to add a link to a post I authored a few years ago on Jupytext + Papermill ? It is available on medium (
https://medium.com/capital-fund-management/automated-reports-with-jupyter-notebooks-using-jupytext-and-papermill-619e60c37330 ) and on github ( https://github.com/CFMTech/jupytext_papermill_post ), maybe adding the two links is important as medium looks nicer but github is more freely available...
There was a problem hiding this comment.
RE papermill: this is why
$ jupytext --to ipynb --set-kernel bash test.sh --pipe-fmt ipynb --pipe 'papermill -f param.yml'
[jupytext] Reading test.sh in format sh
[jupytext] Setting kernel bash
[jupytext] Updating notebook metadata with '{"kernelspec": {"name": "bash", "language": "bash", "display_name": "Bash"}}'
[jupytext] Executing papermill -f param.yml
Traceback (most recent call last):
File "/home/kevin/.local/bin/papermill", line 8, in <module>
sys.exit(papermill())
File "/usr/lib/python3/dist-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/kevin/.local/lib/python3.9/site-packages/papermill/cli.py", line 250, in papermill
execute_notebook(
File "/home/kevin/.local/lib/python3.9/site-packages/papermill/execute.py", line 94, in execute_notebook
nb = parameterize_notebook(
File "/home/kevin/.local/lib/python3.9/site-packages/papermill/parameterize.py", line 83, in parameterize_notebook
param_content = translate_parameters(kernel_name, language, parameters, comment)
File "/home/kevin/.local/lib/python3.9/site-packages/papermill/translators.py", line 560, in translate_parameters
return papermill_translators.find_translator(kernel_name, language).codify(parameters, comment)
File "/home/kevin/.local/lib/python3.9/site-packages/papermill/translators.py", line 31, in find_translator
raise PapermillException(
papermill.exceptions.PapermillException: No parameter translator functions specified for kernel 'bash' or language 'bash'
[jupytext] Error: The command 'papermill -f param.yml' exited with code 1 (use --warn-only to turn this error into a warning)
Though I really ought just make a PR there for Bash as it would be pretty simple.
There was a problem hiding this comment.
As expected it turned out to be pretty easy. See nteract/papermill#674
|
@mwouts I think I have addressed your comments above. |
|
Excellent! Thank you @kdm9 , that's a great addition to the documentation. And thanks for nteract/papermill#674, it would be great if we can get that solved as well! |
See discussion in #968