Skip to content

Commit

Permalink
1.1.4
Browse files Browse the repository at this point in the history
  • Loading branch information
tmtenbrink committed Apr 3, 2024
1 parent 76893a2 commit 2bee6ce
Show file tree
Hide file tree
Showing 12 changed files with 228 additions and 480 deletions.
239 changes: 85 additions & 154 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
test-linux:
strategy:
matrix:
py-version: ['3.7', '3.8', '3.9', '3.10']
py-version: ['3.9', '3.10', '3.11', '3.12']
runs-on: ubuntu-latest
steps:
- name: Checkout rustfrc
Expand All @@ -23,26 +23,26 @@ jobs:
- uses: actions/setup-python@v2
with:
python-version: ${{ matrix.py-version }}

- name: Rust latest
run: rustup update

# We install poetry as a dependency manager to read the pyproject.toml of our package.
- name: Poetry
run: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -
run: pipx install poetry==1.8.2

# GitHub has modified PATH behavior, so we use the below command to add poetry to our PATH
- name: Add poetry path
run: echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Maturin
uses: taiki-e/install-action@v2
with:
tool: maturin

# poetry update will install all dependencies for our package, as well as maturin, which we use as our build
# back-end
- name: Poetry update
run: poetry update
- name: Poetry install
run: poetry install --sync --no-root
working-directory: ${{ github.workspace }}

- name: maturin develop
run: poetry run maturin develop
run: |
. $(poetry env info --path)/bin/activate
maturin develop
working-directory: ${{ github.workspace }}

- name: cargo test
Expand All @@ -52,164 +52,95 @@ jobs:
- name: poetry pytest
run: poetry run pytest
working-directory: ${{ github.workspace }}
build-manylinux:

build-macos:
if: ${{ github.event_name == 'release' && github.event.action == 'published' }}
needs: [test-linux]
needs: [ test-linux ]
runs-on: macos-latest
strategy:
# Let individual matrix jobs go if one fails
matrix:
include:
# We are compatible with all supported Python versions. The 'cp37-cp37m' is the PEP 425 tag of that specific
# Python version, which is where the Python interpreters of the manylinux Docker container are located.
- py-version: '3.7'
py-pep-425: 'cp37-cp37m'
- py-version: '3.8'
py-pep-425: 'cp38-cp38'
- py-version: '3.9'
py-pep-425: 'cp39-cp39'
- py-version: '3.10'
py-pep-425: 'cp310-cp310'
runs-on: ubuntu-latest
py-version: [ '3.9', '3.10', '3.11', '3.12' ]
steps:
- name: Checkout rustfrc
uses: actions/checkout@v2
with:
path: ${{ github.repository }}
# Set up the repository with the Docker container action, which will allow us to build and run a manylinux Docker
# container that will ensure compatibility. See the action repository for more info.
- name: maturin-manylinux-wheels-action
uses: tmtenbrink/[email protected]
with:
py-version: "python${{ matrix.py-version }}"
py-pep-425: ${{ matrix.py-pep-425 }}
package-path: ${{ github.repository }}
# Set up the package repository in a sub-path, so it doesn't overwrite the Docker container action repo

# It now uploads all the wheel files to an artifact that we can later retrieve
- uses: actions/upload-artifact@v2
with:
name: wheels
path: ${{ github.repository }}/dist

build-mac:
if: ${{ github.event_name == 'release' && github.event.action == 'published' }}
needs: [test-linux]
runs-on: macos-11
strategy:
matrix:
# If we run it without this argument, it will generate only x86_64 wheels, which are compatible with more
# macOS and pip versions. Universal2 wheels are compatible with Apple Silicon (aarch64) Macs but also updated
# x86_64 Macs. In the future only universal2 wheels will be necessary.
target: ['--universal2', '']
py-version: ['3.7', '3.8', '3.9', '3.10']
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.py-version }}

- name: Rust latest
run: rustup update

# The x86_64 Rust toolchain is installed on GitHub runners, but since we compile also for Apple Silicon, we also
# need the correct Rust toolchain.
- name: Rustup install aarch64 target
run: rustup target add aarch64-apple-darwin
if: ${{ matrix.target == '--universal2' }}

# We install poetry as a dependency manager to read the pyproject.toml of our package.
- name: Poetry
run: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py | python -

# GitHub has modified PATH behavior, so we use the below command to add poetry to our PATH
- name: Add poetry path
run: echo "$HOME/.local/bin" >> $GITHUB_PATH

# poetry update will install all dependencies for our package, as well as maturin, which we use as our build
# back-end
- name: Poetry update
run: poetry update
working-directory: ${{ github.workspace }}

# delocate is the macOS version of auditwheel.
- name: Install delocate
run: pip install delocate
working-directory: ${{ github.workspace }}

# This command will actually compile the wheels. We use --release to make sure Rust properly optimizes the code.
# The ${{ matrix.target }} indicates whether we are building universal 2 wheels. Finally using -i python ensures
# only the Python we set up above will be used as target.
- name: maturin build release
run: poetry run maturin build --release ${{ matrix.target }} -i python
working-directory: ${{ github.workspace }}

# We now fix the wheels using delocate
- name: delocate fix wheels
run: find . -name "*.whl" -exec delocate-wheel -w wheelhouse --require-archs=universal2 {} \;
working-directory: ${{ github.workspace }}

- uses: actions/upload-artifact@v2
- name: Build wheels - universal2
uses: PyO3/maturin-action@v1
with:
target: universal2-apple-darwin
args: --release --out dist -i ${{ matrix.py-version }}
- name: Upload wheels
uses: actions/upload-artifact@v3
with:
name: wheels
path: ${{ github.workspace }}/wheelhouse
path: ${{ github.workspace }}/dist

build-windows:
if: ${{ github.event_name == 'release' && github.event.action == 'published' }}
needs: [test-linux]
runs-on: windows-latest
strategy:
matrix:
py-version: ['3.7', '3.8', '3.9', '3.10']
steps:
- uses: actions/checkout@v2
- uses: actions/setup-python@v2
with:
python-version: ${{ matrix.py-version }}

- name: Rust latest
run: rustup update

# Installing poetry requires a different command on Windows than on macOS/Linux
- name: Poetry
run: (Invoke-WebRequest -Uri https://raw.githubusercontent.com/python-poetry/poetry/master/install-poetry.py -UseBasicParsing).Content | python -

# PATH is also different
- name: Poetry path
run: echo "$env:APPDATA\Python\Scripts" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

- name: Poetry update
run: |
poetry update
working-directory: ${{ github.workspace }}

- name: maturin build release
run: poetry run maturin build --release -i python
working-directory: ${{ github.workspace }}
if: ${{ github.event_name == 'release' && github.event.action == 'published' }}
needs: [ test-linux ]
runs-on: windows-latest
strategy:
matrix:
py-version: ['3.9', '3.10', '3.11', '3.12']
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.py-version }}
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
target: x64
args: --release --out dist -i ${{ matrix.py-version }}
- name: Upload wheels
uses: actions/upload-artifact@v3
with:
name: wheels
path: ${{ github.workspace }}/dist

- uses: actions/upload-artifact@v2
with:
name: wheels
path: ${{ github.workspace }}/target/wheels
build-linux:
if: ${{ github.event_name == 'release' && github.event.action == 'published' }}
needs: [ test-linux ]
runs-on: ubuntu-latest
strategy:
matrix:
py-version: [ '3.9', '3.10', '3.11', '3.12' ]
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
with:
python-version: ${{ matrix.py-version }}
- name: Build wheels
uses: PyO3/maturin-action@v1
with:
rust-toolchain: stable
target: x86_64
manylinux: auto
args: --release --out dist -i ${{ matrix.py-version }}
- name: Upload wheels
uses: actions/upload-artifact@v3
with:
name: wheels
path: ${{ github.workspace }}/dist

wheel-publish:
needs: [build-manylinux, build-mac, build-windows]
runs-on: ubuntu-latest
steps:
- name: Download artifact
uses: actions/download-artifact@v2
with:
name: wheels
path: ${{ github.workspace }}/dist/
- name: Set up Python
uses: actions/setup-python@v2
with:
python-version: '3.9'
- name: Install wheel test
run: |
pip install ${{ github.workspace }}/dist/*39*manylinux*.whl
- name: Publish a Python distribution to PyPI
if: ${{ github.event_name == 'release' && github.event.action == 'published' }}
uses: pypa/gh-action-pypi-publish@release/v1
with:
user: __token__
password: ${{ secrets.PYPI_RUSTFRC_TOKEN }}
name: Release
runs-on: ubuntu-latest
needs: [ build-macos, build-windows, build-linux ]
steps:
- uses: actions/download-artifact@v3
with:
name: wheels
- name: Publish to PyPI
env:
MATURIN_PYPI_TOKEN: ${{ secrets.PYPI_RUSTFRC_TOKEN }}
uses: PyO3/maturin-action@v1
with:
command: upload
args: --skip-existing *
3 changes: 2 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,5 @@ Cargo.lock
/tests/.pytest_cache
**/__pycache__/**
**/*.pyd
*.so
*.so
.python-version
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Changelog

## 1.1.4 2024-04-03

* **Drop Python 3.7, 3.8 support**: Note that this is technically a breaking change, but 1.1.3 works just fine for older versions and this doesn't include much else. This is necessary to use NumPy 1.26, which is required for Python >=3.12.
* Update interal Rust crate to use new PyO3 bounds API
* `sqr_abs` no longer performs its computation using rayon, which improves performance for arrays at least up to 4000x4000
* Added benchmarks
* `pois_gen` no longer has a hard-coded limit for the amount of elements
2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[package]
name = "rustfrc"
version = "1.1.3"
version = "1.1.4"
edition = "2021"

[lib]
Expand Down
21 changes: 15 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,22 +3,32 @@
![GitHub release](https://flat.badgen.net/github/release/tmtenbrink/rustfrc)
![License](https://flat.badgen.net/github/license/tmtenbrink/rustfrc)

rustfrc is a Python package with some fast Rust functions that are using when performing Fourier Ring Correlation (FRC) computations for resolution determination in microscopy (specifically optical nanoscopy). It is in development for use in a Bachelor end project for the TU Delft in the period 2021-2022.
rustfrc is a Python package with some fast Rust functions that are useful when performing Fourier Ring Correlation (FRC) computations for resolution determination in microscopy (specifically optical nanoscopy). It was originally developed for use in a Bachelor end project for the TU Delft in the period 2021-2022. See the Python package [`frc`](https://github.com/tmtenbrink/frc) for examples of its usage. The `test_split.py` file in the `tests` directory also holds some examples.

Since rustfrc contains compiled (Rust) extensions and is not pure Python, it is not available for all platforms, but only for those with available compiled wheels or a Rust toolchain and `maturin` support (see below). They are available for Windows (x86_64), macOS (x86_64 and universal2, which includes Apple Silicon) and Linux (x86_64). However, since Rust and Python are supported on many platforms, it is not difficult to compile for other platforms (see below).

## Features

Currently, rustfrc does not have many features. The primary one is `binom_split(x: ndarray) -> ndarray` which samples binomial _(n, 0.5)_ with n as the array element value. The operation is fully parallelized and somewhere between 3-10x faster than sampling using NumPy.
rustfrc has only a few features. The primary one is `binom_split(x: ndarray) -> ndarray` which samples binomial _(n, 0.5)_ with n as the array element value. The operation is fully parallelized and somewhere between 3-10x faster than sampling using NumPy.

Furthermore, there are also (since version 1.1) `sqr_abs(a: ndarray) -> ndarray` and `pois_gen(lam: float, shape: tuple[int, ...]) -> ndarray`.

`sqr_abs` computes the element-wise norm and square of a complex array, while `pois_gen` generates an array of the specified size using the Poisson distribution and a single parameter λ.

## Requirements

* Python 3.7 or greater
* NumPy 1.18 or greater
* Python 3.8-3.12
* NumPy 1.18 or greater (tested last with v1.26)

## Performance

On an i7-8750H, a decently high performance 6-core chip from 2017, I measured the following speeds:

- `binom_split`: ~210 ms on a 4000x4000 array, with each element Poisson-generated with a mean of 200
- `pois_gen`: ~420 ms to generate a 4000x4000 array with mean 200
- `sqr_abs`: ~40 ms on a 4000x4000 array, where each element is a complex number with both the real and imaginary parts having a mean of 200

Take this with a grain of salt, but it should provide a decent order of magnitude for larger images.

## Installation

Expand All @@ -32,8 +42,7 @@ However, for an optimal Python experience, use [poetry](https://github.com/pytho

### From source (using maturin)

rustfrc uses [poetry](https://github.com/python-poetry/poetry) as its Python dependency manager. For best results, create a `poetry` virtualenv (be sure to install virtualenv as a systems package) with the `pyproject.toml` and run `poetry update` to install the required packages.
Installing [maturin](https://pypi.org/project/maturin/) manually should also work. It is also necessary to have a Rust toolchain installed on your computer. Rust can be easily installed using [rustup](https://rustup.rs/).
rustfrc uses [poetry](https://github.com/python-poetry/poetry) as its Python dependency manager. For best results, create a `poetry` virtualenv (be sure to install virtualenv as a systems package) with the `pyproject.toml` and run `poetry update` to install the required packages. I recommend installing [maturin](https://pypi.org/project/maturin/) as a global tool (e.g. with `cargo binstall`).

Build a wheel file like this (if using poetry, append `poetry run` before the command) from the project directory:

Expand Down
Loading

0 comments on commit 2bee6ce

Please sign in to comment.