Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dials.import: if lots of arguments passed e.g. 1000's of CBF files add a progress bar #2783

Open
graeme-winter opened this issue Oct 28, 2024 · 2 comments · May be fixed by cctbx/dxtbx#768
Open

dials.import: if lots of arguments passed e.g. 1000's of CBF files add a progress bar #2783

graeme-winter opened this issue Oct 28, 2024 · 2 comments · May be fixed by cctbx/dxtbx#768
Labels
workshop-feedback Usual side effect of people teaching at workshops

Comments

@graeme-winter
Copy link
Contributor

When running dials.import on a lot of CBF files nothing happens for quite some time - a tqdm progress bar or similar (for use on a tty) would be very useful
@graeme-winter graeme-winter added the workshop-feedback Usual side effect of people teaching at workshops label Oct 28, 2024
@graeme-winter
Copy link
Contributor Author

This may have been a particulary egregious example

  num images: 3600
  sequences:
    still:    0
    sweep:    36
  num stills: 0
--------------------------------------------------------------------------------
Writing experiments to imported.expt

real	1m53.006s
user	1m43.166s
sys	0m7.585s

@graeme-winter
Copy link
Contributor Author 

Ethics-Gradient test :( $ time dials.import ../data/*gz
DIALS (2018) Acta Cryst. D74, 85-97. https://doi.org/10.1107/S2059798317017235
DIALS 3.dev.1221-g17b179fb4
100%|███████████████████████████████████████| 3600/3600 [01:49<00:00, 32.84it/s]
The following parameters have been modified:

input {
  experiments = <image files>
}

--------------------------------------------------------------------------------
  format: <class 'dxtbx.format.FormatCBFFullPilatus.FormatCBFFullPilatus'>
  template: /Users/graeme/data/cows-pigs-people/data/CIX1_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX2_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX3_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX5_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX6_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX8_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX9_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX10_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX11_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX12_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX14_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/CIX15_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX5_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX6_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX7_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX8_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX9_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX10_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX11_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX12_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX13_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX14_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX15_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/PIX16_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X1_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X2_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X3_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X4_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X5_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X6_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X7_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X8_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X9_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X11_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X13_1_#####.cbf.gz:1:100
  template: /Users/graeme/data/cows-pigs-people/data/X14_1_#####.cbf.gz:1:100
  num images: 3600
  sequences:
    still:    0
    sweep:    36
  num stills: 0
--------------------------------------------------------------------------------
Writing experiments to imported.expt

real	1m52.310s
user	1m42.478s
sys	0m6.172s

graeme-winter added a commit to cctbx/dxtbx that referenced this issue Oct 31, 2024
@graeme-winter
Add tqdm progress bar to dials.import
c2a1e43 
Give the user some hint as to the progress i.e. the computer is doing
something when you impoet thousands of files.

Resolves dials/dials#2783
@graeme-winter graeme-winter linked a pull request Oct 31, 2024 that will close this issue
Open
graeme-winter added a commit to cctbx/dxtbx that referenced this issue Oct 31, 2024
@graeme-winter
Add tqdm progress bar to dials.import
fde4a2b 
Give the user some hint as to the progress i.e. the computer is doing
something when you impoet thousands of files.

Resolves dials/dials#2783

Only show progress bar if > 1 file and isatty
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
workshop-feedback Usual side effect of people teaching at workshops
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add tqdm progress bar to dials.import cctbx/dxtbx
1 participant
@graeme-winter