Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Suspected memory leak when processing large datasets with append=false #296

Open
nordam opened this issue Sep 19, 2024 · 2 comments · May be fixed by #328
Open

Suspected memory leak when processing large datasets with append=false #296

nordam opened this issue Sep 19, 2024 · 2 comments · May be fixed by #328
Assignees
Labels
bug Something isn't working

Comments

@nordam
Copy link
Collaborator

nordam commented Sep 19, 2024

Describe the bug

Processing a large dataset (300000 + images), and after about 16 hours, the memory use is now up to around 25 GB. Running with append=false in the toml file, and --num_chuncks 4 on the command line, pyopia version 2.5.5

@emlynjdavies emlynjdavies added the bug Something isn't working label Sep 19, 2024
@arsalanmstn
Copy link
Contributor

Can we try "gc.collect()" from garbage collection module at the end of the "def process_file_list(file_list, c)"??? So, it would release the unused memory (I used it for streaming mode of Vimba_python). The code can be modified as:

import gc
def process_file_list(file_list, c):
for ...:
try:
....
gc.collect()

@nepstad
Copy link
Collaborator

nepstad commented Sep 28, 2024

Can we try "gc.collect()" from garbage collection module at the end of the "def process_file_list(file_list, c)"??? So, it would release the unused memory (I used it for streaming mode of Vimba_python). The code can be modified as:

import gc def process_file_list(file_list, c): for ...: try: .... gc.collect()

Could you try this out and report back if you saw a decrease in memory use for long running processes? If so, please open a pull request with your changes.

@nepstad nepstad self-assigned this Oct 11, 2024
@nepstad nepstad linked a pull request Oct 11, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants