Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add pyproject.toml #28385

Merged
merged 8 commits into from
Oct 12, 2023
Merged

Add pyproject.toml #28385

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
93d819a
Add pyproject.toml
AnandInguva Oct 2, 2023
4d29e90
Use subprocess to run grpcio since it is not imported in pyproject.toml
AnandInguva Oct 3, 2023
3d1d454
Remove build-requirements.txt and use build to build the sdist
AnandInguva Oct 9, 2023
69c3042
Use wheels to run tox precommit tests
AnandInguva Oct 6, 2023
fb1f5a7
Remove cython tests. default tests will run with Cython extensions
AnandInguva Oct 6, 2023
d59c260
Force type cast inputs to list
AnandInguva Sep 29, 2023
4c80183
Update stager to use build. If it fails, use legacy setup to build sdist
AnandInguva Oct 3, 2023
dad39ec
Remove cython env and build-requirements for tox.ini
AnandInguva Sep 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/beam_PreCommit_Python.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ jobs:
arguments: |
-Pposargs="--ignore=apache_beam/dataframe/ --ignore=apache_beam/examples/ --ignore=apache_beam/runners/ --ignore=apache_beam/transforms/" \
-PpythonVersion=${{ matrix.python_version }} \
-PuseWheelDistribution
- name: Archive code coverage results
uses: actions/upload-artifact@v3
with:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/beam_PreCommit_Python_Coverage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ jobs:
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:python:test-suites:tox:py38:preCommitPyCoverage
arguments: |
-PuseWheelDistribution
- name: Archive code coverage results
uses: actions/upload-artifact@v3
with:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/beam_PreCommit_Python_Dataframes.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ jobs:
arguments: |
-Pposargs=apache_beam/dataframe/ \
-PpythonVersion=${{ matrix.python_version }} \
-PuseWheelDistribution
tvalentyn marked this conversation as resolved.
Show resolved Hide resolved
- name: Archive code coverage results
uses: actions/upload-artifact@v3
with:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/beam_PreCommit_Python_Examples.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ jobs:
arguments: |
-Pposargs=apache_beam/examples/ \
-PpythonVersion=${{ matrix.python_version }} \
-PuseWheelDistribution
- name: Archive code coverage results
uses: actions/upload-artifact@v3
with:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/beam_PreCommit_Python_Runners.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ jobs:
arguments: |
-Pposargs=apache_beam/runners/ \
-PpythonVersion=${{ matrix.python_version }} \
-PuseWheelDistribution
- name: Archive code coverage results
uses: actions/upload-artifact@v3
with:
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/beam_PreCommit_Python_Transforms.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ jobs:
arguments: |
-Pposargs=apache_beam/transforms/ \
-PpythonVersion=${{ matrix.python_version }} \
-PuseWheelDistribution
- name: Archive code coverage results
uses: actions/upload-artifact@v3
with:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/build_release_candidate.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,7 +314,7 @@ jobs:
pip install -U pip
pip install tox
# TODO(https://github.com/apache/beam/issues/20209): Don't hardcode py version in this file.
pip install -r build-requirements.txt && tox -e py38-docs
tox -e py38-docs
rm -rf target/docs/_build/.doctrees
- name: Build Typescript Docs
working-directory: beam/sdks/typescript
Expand Down
17 changes: 6 additions & 11 deletions .github/workflows/build_wheels.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,11 +92,6 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: 3.8
- name: Get build dependencies
working-directory: ./sdks/python
run: python -m pip install -r build-requirements.txt
- name: Install wheels
run: python -m pip install wheel
- name: Get tag
id: get_tag
run: |
Expand All @@ -117,15 +112,15 @@ jobs:
echo "RELEASE_VERSION=$RELEASE_VERSION" >> $GITHUB_OUTPUT
- name: Build source
working-directory: ./sdks/python
run: python setup.py sdist --formats=zip
run: pip install -U build && python -m build --sdist
- name: Add checksums
working-directory: ./sdks/python/dist
run: |
file=$(ls | grep .zip | head -n 1)
file=$(ls | grep .tar.gz | head -n 1)
tvalentyn marked this conversation as resolved.
Show resolved Hide resolved
sha512sum $file > ${file}.sha512
- name: Unzip source
working-directory: ./sdks/python
run: unzip dist/$(ls dist | grep .zip | head -n 1)
run: tar -xzvf dist/$(ls dist | grep .tar.gz | head -n 1)
- name: Rename source directory
working-directory: ./sdks/python
run: mv $(ls | grep apache-beam) apache-beam-source
Expand Down Expand Up @@ -155,17 +150,17 @@ jobs:
- name: Build RC source
if: steps.is_rc.outputs.is_rc == 1
working-directory: ./sdks/python
run: python setup.py sdist --formats=zip
run: pip install -U build && pythom -m build --sdist
- name: Add RC checksums
if: steps.is_rc.outputs.is_rc == 1
working-directory: ./sdks/python/dist
run: |
file=$(ls | grep .zip | head -n 1)
file=$(ls | grep .tar.gz | head -n 1)
sha512sum $file > ${file}.sha512
- name: Unzip RC source
if: steps.is_rc.outputs.is_rc == 1
working-directory: ./sdks/python
run: unzip dist/$(ls dist | grep .zip | head -n 1)
run: tar -xzvf dist/$(ls dist | grep .tar.gz | head -n 1)
- name: Rename RC source directory
if: steps.is_rc.outputs.is_rc == 1
working-directory: ./sdks/python
Expand Down
8 changes: 1 addition & 7 deletions .github/workflows/dask_runner_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,12 +44,9 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: 3.8
- name: Get build dependencies
working-directory: ./sdks/python
run: pip install pip setuptools --upgrade && pip install -r build-requirements.txt
- name: Build source
working-directory: ./sdks/python
run: python setup.py sdist
run: pip install -U build && python -m build --sdist
- name: Rename source file
working-directory: ./sdks/python/dist
run: mv $(ls | grep "apache-beam.*tar\.gz") apache-beam-source.tar.gz
Expand Down Expand Up @@ -78,9 +75,6 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.params.py_ver }}
- name: Get build dependencies
working-directory: ./sdks/python
run: pip install -r build-requirements.txt
- name: Install tox
run: pip install tox
- name: Install SDK with dask
Expand Down
5 changes: 1 addition & 4 deletions .github/workflows/python_dependency_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ on:
branches: ['master', 'release-*']
tags: 'v*'
# paths where Beam Python's dependencies are configured.
paths: ['sdks/python/setup.py', 'sdks/python/build-requirements.txt', 'sdks/python/container/base_image_requirements_manual.txt']
paths: ['sdks/python/setup.py', 'sdks/python/pyproject.toml', 'sdks/python/container/base_image_requirements_manual.txt']

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
Expand Down Expand Up @@ -38,9 +38,6 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.params.py_ver }}
- name: Install Build dependencies
working-directory: ./sdks/python
run: pip install -r build-requirements.txt
- name: Install base_image_requirements.txt
working-directory: ./sdks/python
run: pip install --no-deps -r container/${{ matrix.params.py_env }}/base_image_requirements.txt
Expand Down
16 changes: 2 additions & 14 deletions .github/workflows/python_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,12 +78,9 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: 3.8
- name: Get build dependencies
working-directory: ./sdks/python
run: pip install pip setuptools --upgrade && pip install -r build-requirements.txt
- name: Build source
working-directory: ./sdks/python
run: python setup.py sdist
run: pip install -U build && python -m build --sdist
- name: Rename source file
working-directory: ./sdks/python/dist
run: mv $(ls | grep "apache-beam.*tar\.gz") apache-beam-source.tar.gz
Expand All @@ -99,7 +96,7 @@ jobs:
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
os: [macos-latest, windows-latest]
params: [
{"py_ver": "3.8", "tox_env": "py38"},
{"py_ver": "3.9", "tox_env": "py39"},
Expand All @@ -113,9 +110,6 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.params.py_ver }}
- name: Get build dependencies
working-directory: ./sdks/python
run: pip install -r build-requirements.txt --use-pep517
- name: Install tox
run: pip install tox
- name: Run tests basic unix
Expand Down Expand Up @@ -148,9 +142,6 @@ jobs:
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python }}
- name: Get build dependencies
working-directory: ./sdks/python
run: pip install -r build-requirements.txt
- name: Install requirements
working-directory: ./sdks/python
run: pip install setuptools --upgrade && pip install -e .
Expand Down Expand Up @@ -192,9 +183,6 @@ jobs:
service_account_key: ${{ secrets.GCP_SA_KEY }}
project_id: ${{ secrets.GCP_PROJECT_ID }}
export_default_credentials: true
- name: Get build dependencies
working-directory: ./sdks/python
run: pip install -r build-requirements.txt
- name: Install requirements
working-directory: ./sdks/python
run: pip install setuptools --upgrade && pip install -e ".[gcp]"
Expand Down
3 changes: 0 additions & 3 deletions .github/workflows/run_perf_alert_tool.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,6 @@ jobs:
with:
service_account_key: ${{ secrets.GCP_SA_KEY }}
export_default_credentials: true
- name: Get Apache Beam Build dependencies
working-directory: ./sdks/python
run: pip install pip setuptools --upgrade && pip install -r build-requirements.txt
- name: Install Apache Beam
working-directory: ./sdks/python
run: pip install -e .[gcp,test]
Expand Down
7 changes: 1 addition & 6 deletions .github/workflows/typescript_tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,10 +89,8 @@ jobs:
- name: Setup Beam Python
working-directory: ./sdks/python
run: |
pip install pip setuptools --upgrade
pip install -r build-requirements.txt
pip install 'pandas>=1.0,<1.5'
python setup.py develop
pip install -e .
- run: npm ci
working-directory: ./sdks/typescript
- run: npm run build
Expand Down Expand Up @@ -146,10 +144,7 @@ jobs:
- name: Setup Beam Python
working-directory: ./sdks/python
run: |
pip install pip setuptools --upgrade
pip install -r build-requirements.txt
pip install 'pandas>=1.0,<1.5'
python setup.py develop
pip install -e ".[gcp]"
- name: Authenticate on GCP
uses: google-github-actions/setup-gcloud@v0
Expand Down
1 change: 1 addition & 0 deletions .test-infra/jenkins/job_PreCommit_Python.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ PrecommitJobBuilder builder = new PrecommitJobBuilder(
'^release/.*$',
],
gradleSwitches: [
'-PuseWheelDistribution',
'-Pposargs=\"--ignore=apache_beam/dataframe/ --ignore=apache_beam/examples/ --ignore=apache_beam/runners/ --ignore=apache_beam/transforms/\"' // All these tests are covered by different jobs.
],
numBuildsToRetain: 40
Expand Down
3 changes: 3 additions & 0 deletions .test-infra/jenkins/job_PreCommit_Python_Coverage.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@ PrecommitJobBuilder builder = new PrecommitJobBuilder(
scope: this,
nameBase: 'Python_Coverage',
gradleTask: ':sdks:python:test-suites:tox:py38:preCommitPyCoverage',
gradleSwitches: [
'-PuseWheelDistribution'
],
timeoutMins: 180,
triggerPathPatterns: [
'^model/.*$',
Expand Down
3 changes: 2 additions & 1 deletion .test-infra/jenkins/job_PreCommit_Python_Dataframes.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@ PrecommitJobBuilder builder = new PrecommitJobBuilder(
nameBase: 'Python_Dataframes',
gradleTask: ':pythonPreCommit',
gradleSwitches: [
'-Pposargs=apache_beam/dataframe/'
'-Pposargs=apache_beam/dataframe/',
'-PuseWheelDistribution'
],
timeoutMins: 180,
triggerPathPatterns: [
Expand Down
3 changes: 2 additions & 1 deletion .test-infra/jenkins/job_PreCommit_Python_Examples.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@ PrecommitJobBuilder builder = new PrecommitJobBuilder(
nameBase: 'Python_Examples',
gradleTask: ':pythonPreCommit',
gradleSwitches: [
'-Pposargs=apache_beam/examples/'
'-Pposargs=apache_beam/examples/',
'-PuseWheelDistribution'
],
timeoutMins: 180,
triggerPathPatterns: [
Expand Down
3 changes: 2 additions & 1 deletion .test-infra/jenkins/job_PreCommit_Python_Runners.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@ PrecommitJobBuilder builder = new PrecommitJobBuilder(
nameBase: 'Python_Runners',
gradleTask: ':pythonPreCommit',
gradleSwitches: [
'-Pposargs=apache_beam/runners/'
'-Pposargs=apache_beam/runners/',
'-PuseWheelDistribution'
],
timeoutMins: 180,
triggerPathPatterns: [
Expand Down
3 changes: 2 additions & 1 deletion .test-infra/jenkins/job_PreCommit_Python_Transforms.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,8 @@ PrecommitJobBuilder builder = new PrecommitJobBuilder(
nameBase: 'Python_Transforms',
gradleTask: ':pythonPreCommit',
gradleSwitches: [
'-Pposargs=apache_beam/transforms/'
'-Pposargs=apache_beam/transforms/',
'-PuseWheelDistribution'
],
timeoutMins: 180,
triggerPathPatterns: [
Expand Down
40 changes: 25 additions & 15 deletions buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2965,7 +2965,7 @@ class BeamModulePlugin implements Plugin<Project> {
executable 'sh'
args '-c', ". ${project.ext.envdir}/bin/activate && " +
"pip install --pre --retries 10 --upgrade pip && " +
"pip install --pre --retries 10 --upgrade tox -r ${project.rootDir}/sdks/python/build-requirements.txt"
"pip install --pre --retries 10 --upgrade tox"
}
}
// Gradle will delete outputs whenever it thinks they are stale. Putting a
Expand Down Expand Up @@ -3048,30 +3048,40 @@ class BeamModulePlugin implements Plugin<Project> {
}
return argList.join(' ')
}

project.ext.toxTask = { name, tox_env, posargs='' ->
project.tasks.register(name) {
dependsOn setupVirtualenv
dependsOn ':sdks:python:sdist'

doLast {
// Python source directory is also tox execution workspace, We want
// to isolate them per tox suite to avoid conflict when running
// multiple tox suites in parallel.
project.copy { from project.pythonSdkDeps; into copiedSrcRoot }

def copiedPyRoot = "${copiedSrcRoot}/sdks/python"
def distTarBall = "${pythonRootDir}/build/apache-beam.tar.gz"
project.exec {
executable 'sh'
args '-c', ". ${project.ext.envdir}/bin/activate && cd ${copiedPyRoot} && scripts/run_tox.sh $tox_env $distTarBall '$posargs'"
if (project.hasProperty('useWheelDistribution')) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need both modes?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Precommits running unittest will have this flag and we will use wheels to install apache beam for precommit tests. For tasks like pylint, docs, mypy, we don't need installation of wheels. Tox will take care of running these suits by installing required deps.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wdyt about automatically adding useWheelDistribution whenever platform is linux so that we don't have to add it in individual test suites?

def pythonVersionNumber = project.ext.pythonVersion.replace('.', '')
dependsOn ":sdks:python:bdistPy${pythonVersionNumber}linux"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we know the platform for the wheel? we have many different test suites.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All the tests that use this flag run on Linux.

doLast {
project.copy { from project.pythonSdkDeps; into copiedSrcRoot }
def copiedPyRoot = "${copiedSrcRoot}/sdks/python"
def collection = project.fileTree(project.project(':sdks:python').buildDir){
include "**/apache_beam-*cp${pythonVersionNumber}*manylinux*.whl"
}
String packageFilename = collection.singleFile.toString()
project.exec {
executable 'sh'
args '-c', ". ${project.ext.envdir}/bin/activate && cd ${copiedPyRoot} && scripts/run_tox.sh $tox_env ${packageFilename} '$posargs' "
}
}
} else {
// tox task will run in editable mode, which is configured in the tox.ini file.
doLast {
project.copy { from project.pythonSdkDeps; into copiedSrcRoot }
def copiedPyRoot = "${copiedSrcRoot}/sdks/python"
project.exec {
executable 'sh'
args '-c', ". ${project.ext.envdir}/bin/activate && cd ${copiedPyRoot} && scripts/run_tox.sh $tox_env '$posargs'"
}
}
}
inputs.files project.pythonSdkDeps
outputs.files project.fileTree(dir: "${pythonRootDir}/target/.tox/${tox_env}/log/")
}
}

// Run single or a set of integration tests with provided test options and pipeline options.
project.ext.enablePythonPerformanceTest = {

Expand Down
2 changes: 1 addition & 1 deletion release/src/main/scripts/build_release_candidate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -346,7 +346,7 @@ if [[ $confirmation = "y" ]]; then
cd ${BEAM_ROOT_DIR}
RELEASE_COMMIT=$(git rev-list -n 1 "tags/${RC_TAG}")
# TODO(https://github.com/apache/beam/issues/20209): Don't hardcode py version in this file.
cd sdks/python && pip install -r build-requirements.txt && tox -e py38-docs
cd sdks/python && tox -e py38-docs
GENERATED_PYDOC=~/${LOCAL_WEBSITE_UPDATE_DIR}/${LOCAL_PYTHON_DOC}/${BEAM_ROOT_DIR}/sdks/python/target/docs/_build
rm -rf ${GENERATED_PYDOC}/.doctrees

Expand Down
3 changes: 3 additions & 0 deletions sdks/python/apache_beam/coders/slow_coders_test.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@
from apache_beam.coders.coders_test_common import *


@unittest.skip(
'Remove non-cython tests.'
'https://github.com/apache/beam/issues/28307')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI, I talked with robertwb; we should be able to make an assumption beam will be installed either from wheels or with Cython at runtime, and remove the non-compiled path and respective tests. We can decouple this change in a separate PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sg. I will follow up in a different PR

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's update the comment. We should remove non-cython tests.
FYI I also reworded the issue.

class SlowCoders(unittest.TestCase):
def test_using_slow_impl(self):
try:
Expand Down
1 change: 0 additions & 1 deletion sdks/python/apache_beam/examples/inference/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ Some examples are also used in [our benchmarks](http://s.apache.org/beam-communi
You must have the latest (possibly unreleased) `apache-beam` or greater installed from the Beam repo in order to run these pipelines,
because some examples rely on the latest features that are actively in development. To install Beam, run the following from the `sdks/python` directory:
```
pip install -r build-requirements.txt
pip install -e .[gcp]
```

Expand Down
4 changes: 2 additions & 2 deletions sdks/python/apache_beam/examples/kafkataxi/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,9 +157,9 @@ Install Beam and dependencies and build a Beam distribution.

```sh
cd beam/sdks/python
pip install -r build-requirements.txt
pip install -e '.[gcp]'
python setup.py sdist
pip install -q build
tvalentyn marked this conversation as resolved.
Show resolved Hide resolved
python -m build --sdist
```

Run the Beam pipeline. You can either use the default Kafka topic name or specify
Expand Down
Loading
Loading