Skip to content

Commit

Permalink
Add support for Python 3.12, ping proj back to 9.3.x, require geopand…
Browse files Browse the repository at this point in the history
…as>=1.0 (#1453)

Contributes to rapidsai/build-planning#40

This PR adds support for Python 3.12.

## Notes for Reviewers

This is part of ongoing work to add Python 3.12 support across RAPIDS.
It temporarily introduces a build/test matrix including Python 3.12, from rapidsai/shared-workflows#213.

A follow-up PR will revert back to pointing at the `branch-24.10` branch of `shared-workflows` once all
RAPIDS repos have added Python 3.12 support.

Other changes required to add that support:

* pinning `proj` back to 9.3.x, reverting #1449 (#1453 (comment))
* requiring `geopandas >= 1.0` (#1453 (comment))

### This will fail until all dependencies have been updates to Python 3.12

CI here is expected to fail until all of this project's upstream dependencies support Python 3.12.

This can be merged whenever all CI jobs are passing.

Authors:
  - James Lamb (https://github.com/jameslamb)
  - Bradley Dice (https://github.com/bdice)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #1453
  • Loading branch information
jameslamb authored Sep 9, 2024
1 parent 0bbd71d commit 2a30f5f
Show file tree
Hide file tree
Showing 14 changed files with 57 additions and 120 deletions.
20 changes: 10 additions & 10 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ concurrency:
jobs:
cpp-build:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -37,7 +37,7 @@ jobs:
python-build:
needs: [cpp-build]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -46,7 +46,7 @@ jobs:
upload-conda:
needs: [cpp-build, python-build]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-upload-packages.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -56,7 +56,7 @@ jobs:
if: github.ref_type == 'branch'
needs: python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12
with:
arch: "amd64"
branch: ${{ inputs.branch }}
Expand All @@ -68,7 +68,7 @@ jobs:
sha: ${{ inputs.sha }}
wheel-build-libcuspatial:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
# build for every combination of arch and CUDA version, but only for the latest Python
matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber)))
Expand All @@ -80,7 +80,7 @@ jobs:
wheel-publish-libcuspatial:
needs: wheel-build-libcuspatial
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -91,7 +91,7 @@ jobs:
wheel-build-cuspatial:
needs: wheel-publish-libcuspatial
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -101,7 +101,7 @@ jobs:
wheel-publish-cuspatial:
needs: wheel-build-cuspatial
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -111,7 +111,7 @@ jobs:
package-type: python
wheel-build-cuproj:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand All @@ -121,7 +121,7 @@ jobs:
wheel-publish-cuproj:
needs: wheel-build-cuproj
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-publish.yaml@python-3.12
with:
build_type: ${{ inputs.build_type || 'branch' }}
branch: ${{ inputs.branch }}
Expand Down
30 changes: 15 additions & 15 deletions .github/workflows/pr.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,47 +27,47 @@ jobs:
- wheel-tests-cuproj
- devcontainer
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/pr-builder.yaml@python-3.12
checks:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/checks.yaml@python-3.12
with:
enable_check_generated_files: false
conda-cpp-build:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-build.yaml@python-3.12
with:
build_type: pull-request
conda-cpp-checks:
needs: conda-cpp-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@python-3.12
with:
build_type: pull-request
enable_check_symbols: true
conda-cpp-tests:
needs: conda-cpp-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@python-3.12
with:
build_type: pull-request
conda-python-build:
needs: conda-cpp-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-build.yaml@python-3.12
with:
build_type: pull-request
conda-python-tests:
needs: conda-python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@python-3.12
with:
build_type: pull-request
conda-notebook-tests:
needs: conda-python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12
with:
build_type: pull-request
node_type: "gpu-v100-latest-1"
Expand All @@ -77,7 +77,7 @@ jobs:
docs-build:
needs: conda-python-build
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/custom-job.yaml@python-3.12
with:
build_type: pull-request
node_type: "gpu-v100-latest-1"
Expand All @@ -87,7 +87,7 @@ jobs:
wheel-build-libcuspatial:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
# build for every combination of arch and CUDA version, but only for the latest Python
matrix_filter: group_by([.ARCH, (.CUDA_VER|split(".")|map(tonumber)|.[0])]) | map(max_by(.PY_VER|split(".")|map(tonumber)))
Expand All @@ -96,34 +96,34 @@ jobs:
wheel-build-cuspatial:
needs: [checks, wheel-build-libcuspatial]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
build_type: pull-request
script: ci/build_wheel_cuspatial.sh
wheel-tests-cuspatial:
needs: wheel-build-cuspatial
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12
with:
build_type: pull-request
script: ci/test_wheel_cuspatial.sh
wheel-build-cuproj:
needs: checks
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-build.yaml@python-3.12
with:
build_type: pull-request
script: ci/build_wheel_cuproj.sh
wheel-tests-cuproj:
needs: [wheel-build-cuspatial, wheel-build-cuproj]
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12
with:
build_type: pull-request
script: ci/test_wheel_cuproj.sh
devcontainer:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/build-in-devcontainer.yaml@python-3.12
with:
arch: '["amd64"]'
cuda: '["12.5"]'
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ on:
jobs:
conda-cpp-checks:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-post-build-checks.yaml@python-3.12
with:
build_type: nightly
branch: ${{ inputs.branch }}
Expand All @@ -25,23 +25,23 @@ jobs:
enable_check_symbols: true
conda-cpp-tests:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-cpp-tests.yaml@python-3.12
with:
build_type: nightly
branch: ${{ inputs.branch }}
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
conda-python-tests:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/conda-python-tests.yaml@python-3.12
with:
build_type: nightly
branch: ${{ inputs.branch }}
date: ${{ inputs.date }}
sha: ${{ inputs.sha }}
wheel-tests-cuspatial:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12
with:
build_type: nightly
branch: ${{ inputs.branch }}
Expand All @@ -50,7 +50,7 @@ jobs:
script: ci/test_wheel_cuspatial.sh
wheel-tests-cuproj:
secrets: inherit
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@branch-24.10
uses: rapidsai/shared-workflows/.github/workflows/wheels-test.yaml@python-3.12
with:
build_type: nightly
branch: ${{ inputs.branch }}
Expand Down
42 changes: 3 additions & 39 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,13 +119,13 @@ docker run --gpus all --pull always --rm -it \
### Install with Conda

To install via conda:
> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10 and 3.11.
> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10, 3.11, and 3.12.
cuSpatial can be installed with conda (miniconda, or the full Anaconda distribution) from the rapidsai channel:

```shell
conda install -c rapidsai -c conda-forge -c nvidia \
cuspatial=24.10 python=3.11 cudatoolkit=11.8
cuspatial=24.10 python=3.12 cudatoolkit=11.8
```
We also provide nightly Conda packages built from the HEAD of our latest development branch.

Expand All @@ -134,7 +134,7 @@ See the [RAPIDS installation documentation](https://docs.rapids.ai/install) for
### Install with pip

To install via pip:
> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10 and 3.11.
> **Note** cuSpatial is supported only on Linux or [through WSL](https://rapids.ai/wsl2.html), and with Python versions 3.10, 3.11, and 3.12.
The cuSpatial pip packages can be installed from NVIDIA's PyPI index. pip installations require using the matching wheel to the system's installed CUDA toolkit.
- For CUDA 11 toolkits, install the `-cu11` wheels
Expand All @@ -145,42 +145,6 @@ pip install cuspatial-cu12 --extra-index-url=https://pypi.nvidia.com
pip install cuspatial-cu11 --extra-index-url=https://pypi.nvidia.com
```

#### Troubleshooting Fiona/GDAL versions

cuSpatial depends on [`geopandas`](https://github.com/geopandas/geopandas), which uses [`fiona >= 1.8.19`](https://pypi.org/project/Fiona/), to read common GIS formats with GDAL.

Fiona requires GDAL is already present on your system, but its minimum required version may be newer than the version of GDAL in your OS's package manager.

Fiona checks the GDAL version at install time and fails with an error like this if a compatible version of GDAL isn't installed:
```
ERROR: GDAL >= 3.2 is required for fiona. Please upgrade GDAL.
```

There are two ways to fix this:

1. Install a version of GDAL that meets fiona's minimum required version
* Ubuntu users can install a newer GDAL with the [UbuntuGIS PPA](https://wiki.ubuntu.com/UbuntuGIS):
```shell
sudo -y add-apt-repository ppa:ubuntugis/ppa
sudo apt install libgdal-dev
```
* Rocky Linux users can install a newer GDAL as follows
```shell
yum update -y
yum config-manager --set-enabled powertools
yum update -y
yum install -y gdal-devel
```
2. Pin fiona's version to a range that's compatible with your version of `libgdal-dev`
* For Ubuntu20.04 ([GDAL v3.0.4](https://packages.ubuntu.com/focal/libgdal-dev)):
```shell
pip install --no-binary fiona --extra-index-url=https://pypi.nvidia.com cuspatial-cu12 'fiona>=1.8.19,<1.9'
```
* For Ubuntu22.04 ([GDAL v3.4.1](https://packages.ubuntu.com/jammy/libgdal-dev)):
```shell
pip install --no-binary fiona --extra-index-url=https://pypi.nvidia.com cuspatial-cu12 'fiona>=1.9'
```

### Build/Install from source

To build and install cuSpatial from source please see the [build documentation](https://docs.rapids.ai/api/cuspatial/stable/developer_guide/build.html).
Expand Down
16 changes: 1 addition & 15 deletions ci/test_wheel_cuproj.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,30 +6,16 @@ set -eou pipefail
mkdir -p ./dist
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

# install build dependencies for fiona
if type -f yum > /dev/null 2>&1; then
yum update -y
# some of gdal-devel's dependencies, like 'libdap', come from the powertools repo
yum config-manager --set-enabled powertools
yum update -y
yum install -y gdal-devel
else
apt update
DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev
fi

# Download the cuproj and cuspatial built in the previous step
RAPIDS_PY_WHEEL_NAME="cuproj_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist
RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist
RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp ./dist

# echo to expand wildcard before adding `[extra]` requires for pip
python -m pip install \
--no-binary 'fiona' \
"$(echo ./dist/cuspatial*.whl)" \
"$(echo ./dist/cuproj*.whl)[test]" \
"$(echo ./dist/libcuspatial*.whl)" \
'fiona>=1.8.19,<1.9'
"$(echo ./dist/libcuspatial*.whl)"

rapids-logger "pytest cuproj"
pushd python/cuproj/cuproj
Expand Down
16 changes: 1 addition & 15 deletions ci/test_wheel_cuspatial.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,28 +6,14 @@ set -eou pipefail
mkdir -p ./dist
RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

# install build dependencies for fiona
if type -f yum > /dev/null 2>&1; then
yum update -y
# some of gdal-devel's dependencies, like 'libdap', come from the powertools repo
yum config-manager --set-enabled powertools
yum update -y
yum install -y gdal-devel
else
apt update
DEBIAN_FRONTEND=noninteractive apt install -y --no-install-recommends libgdal-dev
fi

# Download the cuspatial and libcuspatial built in the previous step
RAPIDS_PY_WHEEL_NAME="cuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 python ./dist
RAPIDS_PY_WHEEL_NAME="libcuspatial_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp ./dist

# echo to expand wildcard before adding `[extra]` requires for pip
python -m pip install \
--no-binary 'fiona' \
"$(echo ./dist/cuspatial*.whl)[test]" \
"$(echo ./dist/libcuspatial*.whl)" \
'fiona>=1.8.19,<1.9'
"$(echo ./dist/libcuspatial*.whl)"

rapids-logger "pytest cuspatial"
pushd python/cuspatial/cuspatial
Expand Down
Loading

0 comments on commit 2a30f5f

Please sign in to comment.