Skip to content
This repository has been archived by the owner on May 3, 2022. It is now read-only.

think about caching strategy for datasets #8

Open
sgratzl opened this issue Nov 15, 2016 · 1 comment
Open

think about caching strategy for datasets #8

sgratzl opened this issue Nov 15, 2016 · 1 comment
Labels
status: help wanted Extra attention is needed

Comments

@sgratzl
Copy link
Contributor

sgratzl commented Nov 15, 2016

In the current version all datasets are loaded on demand without any caching. However, most of the datasets could fit into the main memory.

see also https://kastnerkyle.github.io/posts/using-pytables-for-larger-than-ram-data-processing/

@ngehlenborg
Copy link

One thing to consider in this context is how to deal with situations in which many users are accessing the same server (even smallish data sets could clog main memory once there are too many that are being accessed).

Another situation is genome-wide sequencing data, which generally will not fit into main memory.

@mccalluc mccalluc added Layer: Server status: help wanted Extra attention is needed labels Nov 28, 2016
@sgratzl sgratzl assigned ghost Feb 7, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
status: help wanted Extra attention is needed
Projects
None yet
Development

No branches or pull requests

4 participants