python 3.11 with parcels

[JamiePringle]

In benchmarks, python 3.11 is significantly faster than earlier versions (10 -40%). Below, I show how I installed it with conda on ubuntu 18.04 (the default method of just adding python=3.11 runs into trouble with the specified python versions). First install all dependencies except parcels (I add python, because I use it as my environment):

conda create -n test3p11 -c conda-forge python=3.11 mpi4py mpich scikit-learn jupyter cartopy ffmpeg ipython dask xarray cftime netCDF4 tqdm cgen zarr pymbolic

switch to your new environment with conda activate test3p11.

Then download the 2.4.0 parcels release from GitHub, unzip it, and go to the parcels directory within the downloaded structures, and install in the normal way: python setup.py install.

I have verified that this installation successfully produces a great mass of numbers. I have not confirmed that those numbers make any sense. This information is just here to save others time.

In 5 or so days, I will (1) check that the numbers agree with those produced by older versions of python and (2) give some performance numbers.

Jamie

[JamiePringle]

Have confirmed that python 3.11 gives the exact same results as python 3.10 in both serial and MPI runs. However, while I have not done any proper benchmarking, it does not seem to be much faster. More soon.

[erikvansebille]

Wow, that's very exciting to hear! I'd be keen to see some benchmarking indeed! Any idea why python 3.11 would be faster?

[JamiePringle]

So python 3.11 is supposed to be faster for a number of reasons: https://docs.python.org/3/whatsnew/3.11.html

However, for ocean parcels, it is nearly identical in speed to 3.10 for large runs with many particles. I suspect because most of the time is not spent doing python things, but in the particle kernels, Numpy stuff, and other out-of-python activities. Regardless, I find negligible differences between python versions in my particle tracking runs.

My analysis code shows the expected speedup when it is not dominated by IO and big numpy functions.

Jamie

[erikvansebille]

OK, thanks for the update, @JamiePringle. I've now also read into the python 3.11 code and indeed, while cpython is faster in v3.11, this is not the case for IO-heavy operations and/or when the code does most of its computation in a C extension library like numpy. Both are the case in Parcels.

Nevertheless, good to confirm that Parcels at least works in 3.11 too!