Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add N_DATA_WORKERS env variable to make dataloader num_workers configurable #680

Merged
merged 6 commits into from
Oct 14, 2024
Merged

Add N_DATA_WORKERS env variable to make dataloader num_workers configurable #680

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a177226
make num_worker configurable
nkaenzig Oct 11, 2024
52c5fcf
updated yaml files
nkaenzig Oct 11, 2024
1934942
update docs
nkaenzig Oct 11, 2024
26bc47c
lint
nkaenzig Oct 11, 2024
90b8ba1
update docs
nkaenzig Oct 11, 2024
4e33ae5
set default to 4
nkaenzig Oct 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions configs/vision/pathology/offline/classification/bach.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,8 +107,11 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 256}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
4 changes: 4 additions & 0 deletions configs/vision/pathology/offline/classification/camelyon16.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,14 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 32}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
4 changes: 4 additions & 0 deletions configs/vision/pathology/offline/classification/camelyon16_small.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,14 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 32}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
3 changes: 3 additions & 0 deletions configs/vision/pathology/offline/classification/crc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,8 +105,11 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 4096}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
3 changes: 3 additions & 0 deletions configs/vision/pathology/offline/classification/mhist.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -104,8 +104,11 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 256}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
4 changes: 4 additions & 0 deletions configs/vision/pathology/offline/classification/panda.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,10 +124,14 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 32}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
4 changes: 4 additions & 0 deletions configs/vision/pathology/offline/classification/panda_small.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,10 +124,14 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 32}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
4 changes: 4 additions & 0 deletions configs/vision/pathology/offline/classification/patch_camelyon.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,10 +119,14 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 4096}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
4 changes: 4 additions & 0 deletions configs/vision/pathology/offline/segmentation/bcss.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,10 +137,14 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
3 changes: 3 additions & 0 deletions configs/vision/pathology/offline/segmentation/consep.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,8 +125,11 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
3 changes: 3 additions & 0 deletions configs/vision/pathology/offline/segmentation/monusac.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,11 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
4 changes: 4 additions & 0 deletions configs/vision/pathology/offline/segmentation/total_segmentator_2d.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,10 +137,14 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
2 changes: 2 additions & 0 deletions configs/vision/pathology/online/classification/bach.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 256}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
2 changes: 2 additions & 0 deletions configs/vision/pathology/online/classification/crc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,8 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 4096}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
2 changes: 2 additions & 0 deletions configs/vision/pathology/online/classification/mhist.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,8 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 256}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
3 changes: 3 additions & 0 deletions configs/vision/pathology/online/classification/patch_camelyon.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,8 +92,11 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 4096}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
3 changes: 3 additions & 0 deletions configs/vision/pathology/online/segmentation/bcss.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,8 +117,11 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
2 changes: 2 additions & 0 deletions configs/vision/pathology/online/segmentation/consep.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,8 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
2 changes: 2 additions & 0 deletions configs/vision/pathology/online/segmentation/monusac.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,8 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
3 changes: 3 additions & 0 deletions configs/vision/pathology/online/segmentation/total_segmentator_2d.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,11 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
4 changes: 4 additions & 0 deletions configs/vision/radiology/offline/segmentation/lits.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,10 +132,14 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
4 changes: 4 additions & 0 deletions configs/vision/radiology/offline/segmentation/lits_balanced.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,10 +132,14 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
predict:
batch_size: &PREDICT_BATCH_SIZE ${oc.env:PREDICT_BATCH_SIZE, 64}
num_workers: *N_DATA_WORKERS
4 changes: 3 additions & 1 deletion configs/vision/radiology/online/segmentation/lits.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,11 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
shuffle: true
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
4 changes: 3 additions & 1 deletion configs/vision/radiology/online/segmentation/lits_balanced.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,9 +111,11 @@ data:
dataloaders:
train:
batch_size: &BATCH_SIZE ${oc.env:BATCH_SIZE, 64}
num_workers: &N_DATA_WORKERS ${oc.env:N_DATA_WORKERS, 4}
shuffle: true
val:
batch_size: *BATCH_SIZE
shuffle: true
num_workers: *N_DATA_WORKERS
test:
batch_size: *BATCH_SIZE
num_workers: *N_DATA_WORKERS
3 changes: 2 additions & 1 deletion docs/user-guide/getting-started/how_to_use.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,4 +58,5 @@ To customize runs, without the need of creating custom config-files, you can ove
| `MONITOR_METRIC` | `str` | The metric to monitor for early stopping and final model checkpoint loading |
| `MONITOR_METRIC_MODE` | `str` | "min" or "max", depending on the `MONITOR_METRIC` used |
| `REPO_OR_DIR` | `str` | GitHub repo with format containing model implementation, e.g. "facebookresearch/dino:main" |
| `TQDM_REFRESH_RATE` | `str` | Determines at which rate (in number of batches) the progress bars get updated. Set it to 0 to disable the progress bar. |
| `TQDM_REFRESH_RATE` | `str` | Determines at which rate (in number of batches) the progress bars get updated. Set it to 0 to disable the progress bar. |
| `N_DATA_WORKERS` | `str` | How many subprocesses to use for the torch dataloaders. Set to `null` to use the number of cpu cores. |
4 changes: 2 additions & 2 deletions src/eva/core/data/dataloaders/dataloader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ class DataLoader:
Mutually exclusive with `batch_size`, `shuffle`, `sampler` and `drop_last`.
"""

num_workers: int = multiprocessing.cpu_count()
num_workers: int | None = None
"""How many workers to use for loading the data.

By default, it will use the number of CPUs available.
Expand Down Expand Up @@ -71,7 +71,7 @@ def __call__(self, dataset: datasets.TorchDataset) -> dataloader.DataLoader:
shuffle=self.shuffle,
sampler=self.sampler,
batch_sampler=self.batch_sampler,
num_workers=self.num_workers,
num_workers=self.num_workers or multiprocessing.cpu_count(),
collate_fn=self.collate_fn,
pin_memory=self.pin_memory,
drop_last=self.drop_last,
Expand Down
5 changes: 3 additions & 2 deletions src/eva/vision/data/datasets/segmentation/lits.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,9 +96,10 @@ def configure(self) -> None:
@override
def validate(self) -> None:
for i in range(len(self._volume_files)):
if not os.path.exists(self._segmentation_file(i)):
seg_path = self._segmentation_file(i)
if not os.path.exists(seg_path):
raise FileNotFoundError(
f"Segmentation file not found for volume {self._volume_files[i]}."
f"Segmentation file {seg_path} not found for volume {self._volume_files[i]}."
)

_validators.check_dataset_integrity(
Expand Down