Build environment vs Environment.yml and finding out which packages are causing issues

[bugsbunny88]

Could you please explain why packages need to be in the build environment and which in the environment.yml ?
My understanding right now is that
build-environment.yml = extensions you want enabled in the jupyter lite build stage
Environment.yml = packages you want to be preloaded on your jupyterlite page

on my custom demo page Link here

name: build-env
channels:
  - conda-forge
dependencies:
  - python
  - pip
  - jupyter_server
  - jupyterlab-night
  - ipywidgets>=8.1
  - plotly>=5,<6
  - jupyterlab-cell-flash
  - jupyterlab-fasta>=3,<4
  - jupyterlab-geojson>=3,<4
  - jupyterlite-xeus>=0.1.2,<0.2
  - jupyterlite-core>=0.2,<0.3
  - pip:
    - libarchive-c 

When I was running the ci-cd pipeline it threw up also a warning for libarchive-c ; from an optimization perspective is it recommended to do this via pip like above ?

name: xeus-python-kernel
channels:
  - https://repo.mamba.pm/emscripten-forge
  - https://repo.mamba.pm/conda-forge
dependencies:
  - xeus-python
  - altair
  - bqplot
  - geopy
  - ipycanvas
  - ipydatagrid
  - ipympl
  - matplotlib
  - missingno
  - numpy
  - openpyxl
  - pandas
  - folium
  - plotly
  - pycountry
  - scikit-learn
  - scipy
  - seaborn
  - sqlalchemy
  - vega_datasets

Is it recommended to use repo.mamba for the second channel or just simple conda-forge ?

channels:
  - https://repo.mamba.pm/emscripten-forge
  - https://repo.mamba.pm/conda-forge

vs

channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge

[martinRenou]

build-environment.yml = extensions you want enabled in the jupyter lite build stage

Yeah 👍🏽

Environment.yml = packages you want to be preloaded on your jupyterlite page

I would say: packages loaded in the xeus-python kernel.

When I was running the ci-cd pipeline it threw up also a warning for libarchive-c ; from an optimization perspective is it recommended to do this via pip like above ?

Not aware of this issue. Through pip or conda/mamba is fine I guess? I would not be so worried about performance at this stage, but if you're thinking about it, adding a pip install command just for that one package will make your install slower, I would install it throught conda/mamba like the rest.

Is it recommended to use repo.mamba for the second channel or just simple conda-forge ?

repo.mamba for emscripten-forge package is mandatory for now, official conda-forge does not have emscripten packages.

You can use (should use?) conda-forge instead of our mirror https://repo.mamba.pm/conda-forge though. As of today I believe there is an issue with updating our mirror so you may not find the very latest versions of your packages there.

[bugsbunny88]

Update 1: The following packages have issues:

• altair doesn't work because of some error with jsonschema
• if seaborn is installed alongside scipy or scikit-learn there ends up an issue with statsmodels which causes issues with both scipy or scikit-learn ; therefore seaborn can't be used as a plotting library either

build-environment.yml = extensions you want enabled in the jupyter lite build stage

Yeah 👍🏽

Environment.yml = packages you want to be preloaded on your jupyterlite page

I would say: packages loaded in the xeus-python kernel.

When I was running the ci-cd pipeline it threw up also a warning for libarchive-c ; from an optimization perspective is it recommended to do this via pip like above ?

Not aware of this issue. Through pip or conda/mamba is fine I guess? I would not be so worried about performance at this stage, but if you're thinking about it, adding a pip install command just for that one package will make your install slower, I would install it throught conda/mamba like the rest.

Is it recommended to use repo.mamba for the second channel or just simple conda-forge ?

repo.mamba for emscripten-forge package is mandatory for now, official conda-forge does not have emscripten packages.

You can use (should use?) conda-forge instead of our mirror https://repo.mamba.pm/conda-forge though. As of today I believe there is an issue with updating our mirror so you may not find the very latest versions of your packages there.

Many thanks Martin , that was extremely helpful insight. Although I put in a generic ask on the Jupyterlite page I was wondering perhaps you could suggestions for improving my deployment. I am wanting to figure out which packages might be possibly causing warnings & errors on my browser console. For instance maybe its openpyxl or something and I would just remove it on my end. You can have a look at my repo here just off the demo

ie.

Unsatisfied version 4.0.11 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlab/notebook (required ^4.1.0) remoteEntry.9f387e5e108e458f62c3.js:1:7192
Unsatisfied version 4.0.11 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlab/logconsole (required ^4.1.0) remoteEntry.9f387e5e108e458f62c3.js:1:7192
No satisfying version (^4.1.0) of shared module @jupyterlab/outputarea found in shared scope default.
Available versions: 4.0.11 from _JUPYTERLAB.CORE_OUTPUT remoteEntry.9f387e5e108e458f62c3.js:1:7192
Unsatisfied version 4.0.11 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlab/translation (required ^4.1.0) remoteEntry.9f387e5e108e458f62c3.js:1:7192
Unsatisfied version 4.0.11 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlab/rendermime (required ^4.1.0) remoteEntry.9f387e5e108e458f62c3.js:1:7192
Unsatisfied version 7.0.11 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlab/services (required ^7.1.0) remoteEntry.9f387e5e108e458f62c3.js:1:7192
Unsatisfied version 4.0.11 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlab/settingregistry (required ^4.1.0) remoteEntry.9f387e5e108e458f62c3.js:1:7192
Unsatisfied version 4.0.11 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlab/mainmenu (required ^4.1.0) remoteEntry.9f387e5e108e458f62c3.js:1:7192
Unsatisfied version 6.0.11 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlab/coreutils (required ^6.1.2) remoteEntry.ff7ee85ad1087146a275.js:1:5299
Unsatisfied version 2.0.1 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @lumino/messaging (required ^1.10.0) remoteEntry.a36d13f475360b3d8988.js:1:6038
Unsatisfied version 2.3.1 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @lumino/widgets (required ^1.37.2) remoteEntry.a36d13f475360b3d8988.js:1:6038
Unsatisfied version 4.1.11 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @jupyterlab/apputils (required ^3.5.1) remoteEntry.6a37df5d4590b29196a3.js:1:4825
Unsatisfied version 2.3.1 from _JUPYTERLAB.CORE_OUTPUT of shared singleton module @lumino/widgets (required ^1.37.2) remoteEntry.c764a537ae4fed4fb4ca.js:1:5956
Synchronous XMLHttpRequest on the main thread is deprecated because of its detrimental effects to the end user’s experience. For more help https://xhr.spec.whatwg.org/#sync-warning 413.df2804f5ae3431aa4c17.js:1:2354