Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address large files being given #2

Open
dmanno opened this issue Mar 9, 2016 · 1 comment
Open

Address large files being given #2

dmanno opened this issue Mar 9, 2016 · 1 comment

Comments

@dmanno
Copy link
Contributor

dmanno commented Mar 9, 2016

Currently large files being given will take a long time using cp. I'm not sure we want to encourage the reproduction of large files that are already stored, so this may be a non-issue.

We should at least check file size before "giving", maybe print an error and exit if the file is too large? Or let the user know the give is going to take too long. If we want give to allow large file sharing, we need to parallelize the copy or let the user know it could take a very long time.

@dayr @jennfshr thoughts?

@dmanno
Copy link
Contributor Author

dmanno commented Sep 28, 2017

At LANL, our givedirs reside on lustre, so an easy fix would be to check if the underlying FS is lustre (or another stripe-able FS) then based on filesize, setup an appropriate stripe count for the file being copied. It won't speed up the read, but it will speed up the write.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant