Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create DiskArchiver for SPS #4

Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Conversation

blinkdog
Copy link
Contributor

This PR creates a DiskArchiver component/module to replace the one in jadeite, for use at SPS.

Motivation:

  • The jadeite DiskArchiver works, but is relatively slow at finalizing disks, especially large disks.
  • This wasn't a big problem when the archival disks were ~2-6 TB, but now that we're deploying 16 TB disks, it could become an operational issue.
  • Moreover, the way metadata is created at the end puts all of the metadata in one large JSON object. Again, less of a problem when archival disks were ~2-6 TB, but now with 16 TB disks, it can become an operational issue. (Also; who wants to load a huge multi-GB single JSON object?)
  • The new DiskArchiver is intended to solve all of these issues, and also provide new data safety checks using hardware metadata.
  • In particular, we'd like to capture the hardware metadata at the time the disk is used for archival purposes. We'd also like to store that in a database, make it queryable, and ensure that a serial number is not reused "too soon" (i.e.: within the same retro disk season).

@blinkdog blinkdog self-assigned this Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant