Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SART algorithm stuck in "set_w" when the input image is large #496

Open
JiahaoPlus opened this issue Oct 7, 2023 · 13 comments
Open

SART algorithm stuck in "set_w" when the input image is large #496

JiahaoPlus opened this issue Oct 7, 2023 · 13 comments

Comments

@JiahaoPlus
Copy link

Behavior

When I ran iterative methods like SART on a CT scan (512×512×463, can be downloaded here), it was stuck in "set_w" (w=Ax(...)) when initializing IterativeReconAlg class (link to the code). Do you know the possible reason and how to fix it?

Smaller images (like 256×256×256) work well. Running FDK on the 512×512×463 CT also works well.

Code to reproduce the problem (If applicable)

image_pred = algs.sart(projections, geo, angles, niter=4)

where "projections, geo, angles" can be obtained from the CT pickle file I attached above.

Specifications

  • MATLAB/python version: python 3.9
  • OS: Linux
  • CUDA version: 11.3

(Data reference: Zha et. al, NAF)

Thanks!

@AnderBiguri
Copy link
Member

HI, @JiahaoPlus,
Can you clarify what you mean by "stuck"? How do you measure that?
What is your GPU?
Can you reproduce it using TIGRE code and data (headphantom)?

@caiyuanhao1998
Copy link

Hi, I also met this problem.
The stuck is like this:
截屏2023-10-07 下午2 36 31

@caiyuanhao1998
Copy link

I use RTX 8000 and use the eval.py to test.

@JiahaoPlus
Copy link
Author

Hi @AnderBiguri, by "stuck" I mean the algorithm keeps running in Line 222 and there is no response. I found it out by setting checkpoints and running line by line. I use A5000 GPU.
Will try TIGRE code and data (headphantom).

@JiahaoPlus
Copy link
Author

TIGRE demo TIGRE/Python/demos/d07_Algorithms02.py with head phantom data works well, but here the CT size is small (64x64x64).

@AnderBiguri
Copy link
Member

Hi @JiahaoPlus , you can change the head phantom to any size when you call it, can you try to make it the same size as your data?

I am just trying to figure out what is going on, as I have reconstructed images >2000^3, and while that call, Ax, takes long at big images, it shouldn't be more than few seconds at your image size

@JiahaoPlus
Copy link
Author

Hi @AnderBiguri, thanks for your reply! I changed the head phantom size to 512x512x512, and it worked well.

However, my data cannot work. I guess the problem comes from the "geometry" parameters. If I multiply the "geo.dVoxel" by 1000 (or 100, or 10) in my data, it will not get stuck, but the reconstructed image is wrong (because of the wrong geo.dVoxel).

Hereby I attach the minimal code to reproduce the problem (you can download the data here and change the path):

import tigre
import numpy as np
import tigre.algorithms as algs
import pickle

path = '/path/to/abdomen_50.pickle'

with open(path, "rb") as handle:
    data = pickle.load(handle)

#%% Geometry (unit: mm)
geo = tigre.geometry_default(high_resolution=False)
geo.DSD = data["DSD"] # Distance Source Detector
geo.DSO = data["DSO"]  # Distance Source Origin
# Detector parameters
geo.nDetector = np.array(data["nDetector"])  # number of pixels
geo.dDetector = np.array(data["dDetector"])  # size of each pixel
geo.sDetector = geo.nDetector * geo.dDetector  # total size of the detector
# Image parameters
geo.nVoxel = np.array(data["nVoxel"])  # number of voxels
geo.dVoxel = np.array(data["dVoxel"])  # size of each voxel
geo.sVoxel = geo.nVoxel * geo.dVoxel  # total size of the image
geo.nVoxel = np.flip(geo.nVoxel)
geo.dVoxel = np.flip(geo.dVoxel)
geo.sVoxel = np.flip(geo.sVoxel)

#%% Load image and projections
image = data["image"]
train_angles = data["train"]["angles"]
train_projs = data["train"]["projections"]

#%% Algorithms
imgSART = algs.sart(train_projs, geo, train_angles, niter=4)
print(imgSART.shape)

When I print "geo", I got:

ipdb> print(geo)
TIGRE parameters
-----
Geometry parameters
Distance from source to detector (DSD) = 1500.0 mm
Distance from source to origin (DSO)= 1000.0 mm
-----
Detector parameters
Number of pixels (nDetector) = [1024 1024]
Size of each pixel (dDetector) = [1. 1.] mm
Total size of the detector (sDetector) = [1024. 1024.] mm
-----
Image parameters
Number of voxels (nVoxel) = [463 512 512]
Total size of the image (sVoxel) = [463. 320. 320.] mm
Size of each voxel (dVoxel) = [1.    0.625 0.625] mm
-----
Offset correction parameters
Offset of image from origin (offOrigin) = [0 0 0] mm
Offset of detector (offDetector) = [0 0] mm
-----
Auxillary parameters
Samples per pixel of forward projection (accuracy) = 0.5
-----
Rotation of the Detector (rotDetector) = [0 0 0] rad

Thank you very much for looking into it!

@JiahaoPlus
Copy link
Author

Hi @AnderBiguri, if I change the tigre version to 2.3, everything works well! The same code could not work in the current version (2.5).

@AnderBiguri
Copy link
Member

Hum, interesting @JiahaoPlus.

Are you using a conda enviroment? Are you able to provide the versions of the libraries you are using with e.g. conda list ?

I ask because TIGRE 2.5 changed some things to account for updates that break compatibility by numpy and cython. It may be possible that your code is hanging in a weird way if you don't have the newest versions of these libraries. But if that is the case, the smart thing for TIGRE to do would be to catch it at e.g. install time and warn you about incompatibilities, instead of silently hanging sometimes.

It may be related to #492 , perhaps.

I ping @tsadakane just because they had a look at this in the past and may know whats going on, but no expectations of replying or trying to fix it, tsadakane!

@JiahaoPlus
Copy link
Author

Hi @AnderBiguri, yes I am using conda.

Here is the conda list with the newest TIGRE (2.5) which could not run well on my data (some packages are not relevant to TIGRE, but just in case, I put all of them here):

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
absl-py                   2.0.0                    pypi_0    pypi
argparse                  1.4.0                    pypi_0    pypi
asttokens                 2.4.0                    pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
blessed                   1.20.0                   pypi_0    pypi
ca-certificates           2023.08.22           h06a4308_0  
cachetools                5.3.1                    pypi_0    pypi
certifi                   2023.7.22                pypi_0    pypi
charset-normalizer        3.2.0                    pypi_0    pypi
contourpy                 1.1.1                    pypi_0    pypi
cycler                    0.11.0                   pypi_0    pypi
cython                    3.0.2                    pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
exceptiongroup            1.1.3                    pypi_0    pypi
executing                 1.2.0                    pypi_0    pypi
fonttools                 4.42.1                   pypi_0    pypi
google-auth               2.23.0                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
gpustat                   1.1.1                    pypi_0    pypi
grpcio                    1.58.0                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
imageio                   2.25.1                   pypi_0    pypi
importlib-metadata        6.8.0                    pypi_0    pypi
importlib-resources       6.0.1                    pypi_0    pypi
ipdb                      0.13.13                  pypi_0    pypi
ipython                   8.15.0                   pypi_0    pypi
jedi                      0.19.0                   pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
markdown                  3.4.4                    pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.7.0                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
networkx                  3.1                      pypi_0    pypi
ninja                     1.11.1                   pypi_0    pypi
numpy                     1.26.0                   pypi_0    pypi
nvidia-ml-py              12.535.108               pypi_0    pypi
nvitop                    1.3.0                    pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
opencv-python             4.7.0.68                 pypi_0    pypi
openssl                   3.0.10               h7f8727e_2  
packaging                 23.1                     pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    9.5.0                    pypi_0    pypi
pip                       23.2.1           py39h06a4308_0  
prompt-toolkit            3.0.39                   pypi_0    pypi
protobuf                  4.24.3                   pypi_0    pypi
psutil                    5.9.5                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pyasn1                    0.5.0                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pygments                  2.16.1                   pypi_0    pypi
pyparsing                 3.1.1                    pypi_0    pypi
python                    3.9.18               h955ad1f_0  
python-dateutil           2.8.2                    pypi_0    pypi
pytigre                   2.4.0                    pypi_0    pypi
pywavelets                1.4.1                    pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
readline                  8.2                  h5eee18b_0  
requests                  2.31.0                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scikit-image              0.19.3                   pypi_0    pypi
scipy                     1.10.0                   pypi_0    pypi
setuptools                68.0.0           py39h06a4308_0  
six                       1.16.0                   pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0  
stack-data                0.6.2                    pypi_0    pypi
tensorboard               2.12.0                   pypi_0    pypi
tensorboard-data-server   0.7.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
termcolor                 2.3.0                    pypi_0    pypi
tifffile                  2023.9.18                pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
tomli                     2.0.1                    pypi_0    pypi
torch                     1.11.0+cu113             pypi_0    pypi
torchaudio                0.11.0+cu113             pypi_0    pypi
torchvision               0.12.0+cu113             pypi_0    pypi
tqdm                      4.64.1                   pypi_0    pypi
traitlets                 5.10.0                   pypi_0    pypi
typing-extensions         4.8.0                    pypi_0    pypi
tzdata                    2023c                h04d1e81_0  
urllib3                   1.26.16                  pypi_0    pypi
wcwidth                   0.2.6                    pypi_0    pypi
werkzeug                  2.3.7                    pypi_0    pypi
wheel                     0.38.4           py39h06a4308_0  
xz                        5.4.2                h5eee18b_0  
zipp                      3.17.0                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0  

Here is the TIGRE 2.3 related conda list that can work well:

# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main  
_openmp_mutex             5.1                       1_gnu  
absl-py                   2.0.0                    pypi_0    pypi
argparse                  1.4.0                    pypi_0    pypi
asttokens                 2.4.0                    pypi_0    pypi
backcall                  0.2.0                    pypi_0    pypi
blessed                   1.20.0                   pypi_0    pypi
ca-certificates           2023.08.22           h06a4308_0  
cachetools                5.3.1                    pypi_0    pypi
certifi                   2023.7.22                pypi_0    pypi
charset-normalizer        3.3.0                    pypi_0    pypi
contourpy                 1.1.1                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
cython                    0.29.25                  pypi_0    pypi
decorator                 5.1.1                    pypi_0    pypi
exceptiongroup            1.1.3                    pypi_0    pypi
executing                 2.0.0                    pypi_0    pypi
fonttools                 4.43.1                   pypi_0    pypi
google-auth               2.23.2                   pypi_0    pypi
google-auth-oauthlib      0.4.6                    pypi_0    pypi
gpustat                   1.1.1                    pypi_0    pypi
grpcio                    1.59.0                   pypi_0    pypi
idna                      3.4                      pypi_0    pypi
imageio                   2.25.1                   pypi_0    pypi
importlib-metadata        6.8.0                    pypi_0    pypi
importlib-resources       6.1.0                    pypi_0    pypi
ipdb                      0.13.13                  pypi_0    pypi
ipython                   8.16.1                   pypi_0    pypi
jedi                      0.19.1                   pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
ld_impl_linux-64          2.38                 h1181459_1  
libffi                    3.4.4                h6a678d5_0  
libgcc-ng                 11.2.0               h1234567_1  
libgomp                   11.2.0               h1234567_1  
libstdcxx-ng              11.2.0               h1234567_1  
markdown                  3.5                      pypi_0    pypi
markupsafe                2.1.3                    pypi_0    pypi
matplotlib                3.7.0                    pypi_0    pypi
matplotlib-inline         0.1.6                    pypi_0    pypi
ncurses                   6.4                  h6a678d5_0  
networkx                  3.1                      pypi_0    pypi
ninja                     1.11.1                   pypi_0    pypi
numpy                     1.21.6                   pypi_0    pypi
nvidia-ml-py              12.535.108               pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
opencv-python             4.7.0.68                 pypi_0    pypi
openssl                   3.0.11               h7f8727e_2  
packaging                 23.2                     pypi_0    pypi
parso                     0.8.3                    pypi_0    pypi
pexpect                   4.8.0                    pypi_0    pypi
pickleshare               0.7.5                    pypi_0    pypi
pillow                    10.0.1                   pypi_0    pypi
pip                       23.2.1           py39h06a4308_0  
prompt-toolkit            3.0.39                   pypi_0    pypi
protobuf                  4.24.4                   pypi_0    pypi
psutil                    5.9.5                    pypi_0    pypi
ptyprocess                0.7.0                    pypi_0    pypi
pure-eval                 0.2.2                    pypi_0    pypi
pyasn1                    0.5.0                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pygments                  2.16.1                   pypi_0    pypi
pyparsing                 3.1.1                    pypi_0    pypi
python                    3.9.18               h955ad1f_0  
python-dateutil           2.8.2                    pypi_0    pypi
pytigre                   2.2.0                     dev_0    <develop>
pywavelets                1.4.1                    pypi_0    pypi
pyyaml                    6.0                      pypi_0    pypi
readline                  8.2                  h5eee18b_0  
requests                  2.31.0                   pypi_0    pypi
requests-oauthlib         1.3.1                    pypi_0    pypi
rsa                       4.9                      pypi_0    pypi
scikit-image              0.19.3                   pypi_0    pypi
scipy                     1.10.0                   pypi_0    pypi
setuptools                68.0.0           py39h06a4308_0  
six                       1.16.0                   pypi_0    pypi
sqlite                    3.41.2               h5eee18b_0  
stack-data                0.6.3                    pypi_0    pypi
tensorboard               2.12.0                   pypi_0    pypi
tensorboard-data-server   0.7.1                    pypi_0    pypi
tensorboard-plugin-wit    1.8.1                    pypi_0    pypi
tifffile                  2023.9.26                pypi_0    pypi
tk                        8.6.12               h1ccaba5_0  
tomli                     2.0.1                    pypi_0    pypi
torch                     1.11.0+cu113             pypi_0    pypi
torchaudio                0.11.0+cu113             pypi_0    pypi
torchvision               0.12.0+cu113             pypi_0    pypi
tqdm                      4.64.1                   pypi_0    pypi
traitlets                 5.11.2                   pypi_0    pypi
typing-extensions         4.8.0                    pypi_0    pypi
tzdata                    2023c                h04d1e81_0  
urllib3                   2.0.6                    pypi_0    pypi
wcwidth                   0.2.8                    pypi_0    pypi
werkzeug                  3.0.0                    pypi_0    pypi
wheel                     0.41.2           py39h06a4308_0  
xz                        5.4.2                h5eee18b_0  
zipp                      3.17.0                   pypi_0    pypi
zlib                      1.2.13               h5eee18b_0  

@JiahaoPlus
Copy link
Author

A brief summary, in my 2 conda environments:

With TIGRE 2.5,
pytigre==2.4.0, cython==3.0.2, numpy==1.26.0

With TIGRE 2.3,
pytigre==2.2.0, cython==0.29.25, numpy==1.21.6

@tsadakane
Copy link
Contributor

tsadakane commented Oct 9, 2023

@AnderBiguri , @JiahaoPlus

I made two fresh virtual environments of conda

  • test496_311
    • python: 3.11.6
    • cython: 3.0.3
    • numpy: 1.26.0
  • test496_39
    • python: 3.9.18
    • cython: 3.0.3
    • numpy: 1.26.0

and built tigre (4598aaa) by

pip install .

In both environments, the program given at #496 (comment) worked with the given data at the fist comment.

My PC's OS is Windows 10 + CUDA 11.8 and its GPUs are

0: NVIDIA GeForce RTX 2080 Ti
1: NVIDIA GeForce RTX 2080 Ti
2: NVIDIA GeForce GTX 1070

so I used 0 and 1 by modifying the program given at #496 (comment) by adding the following lines

listGpuNames = gpu.getGpuNames()
if len(listGpuNames) == 0:
    print("Error: No gpu found")
else:
    for id in range(len(listGpuNames)):
        print("{}: {}".format(id, listGpuNames[id]))

gpuids = gpu.getGpuIds(listGpuNames[0])
print(gpuids)
....
imgSART = algs.sart(train_projs, geo, train_angles, niter=4, gpuids=gpuids)  # add gpuids option

I also added lines for plots so that I can check the reconstructed volume.

The result of

conda list > conda_list.txt

in the virtual env. test496_39 after building tigre is as follows:

# packages in environment at C:\opt\CondaEnvs\test496_39:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h8ffe710_4    conda-forge
ca-certificates           2023.7.22            h56e8100_0    conda-forge
colorama                  0.4.6                    pypi_0    pypi
contourpy                 1.1.1                    pypi_0    pypi
cycler                    0.12.1                   pypi_0    pypi
cython                    3.0.3                    pypi_0    pypi
fonttools                 4.43.1                   pypi_0    pypi
importlib-resources       6.1.0                    pypi_0    pypi
kiwisolver                1.4.5                    pypi_0    pypi
libffi                    3.4.2                h8ffe710_5    conda-forge
libsqlite                 3.43.0               hcfcfb64_0    conda-forge
libzlib                   1.2.13               hcfcfb64_5    conda-forge
matplotlib                3.8.0                    pypi_0    pypi
numpy                     1.26.0                   pypi_0    pypi
openssl                   3.1.3                hcfcfb64_0    conda-forge
packaging                 23.2                     pypi_0    pypi
pillow                    10.0.1                   pypi_0    pypi
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.1.1                    pypi_0    pypi
python                    3.9.18          h4de0772_0_cpython    conda-forge
python-dateutil           2.8.2                    pypi_0    pypi
pytigre                   2.4.0                    pypi_0    pypi
scipy                     1.11.3                   pypi_0    pypi
setuptools                68.2.2             pyhd8ed1ab_0    conda-forge
six                       1.16.0                   pypi_0    pypi
tk                        8.6.13               hcfcfb64_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
ucrt                      10.0.22621.0         h57928b3_0    conda-forge
vc                        14.3                h64f974e_17    conda-forge
vc14_runtime              14.36.32532         hdcecf7f_17    conda-forge
vs2015_runtime            14.36.32532         h05e6639_17    conda-forge
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h8d14728_0    conda-forge
zipp                      3.17.0                   pypi_0    pypi

@AnderBiguri
Copy link
Member

Thanks a lot @tsadakane.
There is certainly somewhere something off and I really can't pinpoint it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants