re-archive hand remediated moabs that may need special treatment.

[jmartin-sul]

Some Moabs are/were invalid and need remediation; they were archived to cloud (because pres cat trusted that robots handed over good objects, or they went bad from hand edits later). should they be re-archived once they've been remediated?

remediation tickets by tag, moab_remediation online moab that may need remediation (e.g. missing files, extraneous files, corrupted content) (accurate as of 2020-04-23, but click through to see if anything is missing before closing this)

• checksum validation failure auditing moab for expunged object fk847cz0044 #1192
• extraneous versionMetadata.xml file in v1 of 504 moabs on storage root 2 #1393
• extraneous technicalMetadata.xml in 2 Moabs on storage root 2? #1394
• preservation copy of vq518cf9512 (service disk 8) has a number of issues #1395
• checksum mismatch on metadata file when auditing zx284xc2961 (services-disk10) #1397
• remediate moabs for mg599hw5271 and py880rz2144 (service disk 8) #1398
• fr360kt0172 seems to have a duplicate content file, maybe as a result of an scp mistake? (services-disk16) #1399
• expunged content in ds777pr3860 #1500
• checksum validation failure for vf742yx0561 #1520? -- should first confirm that this was modified, but it appears it was based on the comments on the ticket

Discussed F2F w/ @andrewjbtw and he said he prefers to re-create and re-push the archives for all remediated Moabs. Even in cases where the remediation was just to remove an extraneous datastream file (i.e. no expected content was missing/corrupt), the upside of that approach is that if we do ever have to retrieve the archives from the cloud, we won't be scratching our heads and wondering/worrying about why the archived Moabs don't fully validate. The downside in general of re-archiving is that we have to do the work of deleting the old archive versions and re-pushing new versions, and that we generally frown on messing with copies that are already considered archived (good archival practice would be that the thing is written and then left undisturbed forever if possible).

@andrewjbtw would also like @julianmorley's opinion on the difficulty of clearing the old archive copies (as well as any reservations he might have that we haven't thought of).

if we decide to re-archive, using the druid lists we've collected on the above linked issues should make it easy to trigger archiving for said druids using the existing pres cat code. [answered, see comments]

[jmartin-sul]

discussed in preservation planning meeting yesterday with @julianmorley, and he is ok with re-archiving these moabs. it will be a bit of a pain, since it may all be done via manual deletion via AWS/IBM web console (slight chance he'll script it -- he's mulling it over). and it'll result in some notification spam (because we get alerts for deleted archives, since we expect such deletions to be very rare). but we decided that these things are tolerable/worth it.

once julian has deleted the bad archives, we will need to:

• delete the ZippedMoabVersions and their associated ZipParts for the deleted archives (from pres cat's DB)
• feed those druids to ZipMaker for re-archiving of the remediated moabs.

this is on hold until the moabs are actually remediated.

[jmartin-sul]

if it's more manageable, it's fine to spin off subtickets from this for specific druid lists from this overall effort. it is possible that we won't get all the remediation done all at once, and then we may want to do the re-archiving as we go also. also fine to just do it all under this ticket. whatever is easiest for the people doing the re-archiving.

[jmartin-sul]

this is unblocked for the things that are linked.

once we have a process for this, it should be documented alongside moab remediation notes, and we should also instruct moab remediators to file re-archiving tickets for moabs that they remediate (since we expect there will be occasional remediations on an ongoing basis).

[ndushay]

Has Julian been asked to delete the archived copies from the cloud? Should that be done all in a lump as a first step?

[jmartin-sul]

Has Julian been asked to delete the archived copies from the cloud? Should that be done all in a lump as a first step?

i'd be hesitant to do that without a clear plan for pushing through what needs to be re-archived very shortly after that deletion, since for the most part, the mis-archived moabs are better than having no archived moabs (e.g. the hundreds with extraneous datastream xml).

as a first step, maybe the thing to do is to gather an explicit central list of druids to re-archive, based on the tickets linked from this one?

then a clear coordinated plan to delete and re-push, with the re-push following very shortly after the deletion.

we'll also need to delete pres cat's database records for the zips we're deleting from the cloud. my gut feeling (without having thought much about that part yet) is that we want to delete our DB records first.

if we wanted to be extra careful, we could delete and re-push without doing all cloud endpoints at once, so that we're not wiping all cloud archive copies for a moab simultaneously.

[ndushay]

I'm going to close this ticket because we now have audits that will find problems with both on prem storage and with replicated content. Plus, it's 2 1/2 years old.

[andrewjbtw]

That's not what this ticket is about. It's about specific moabs that were manipulated and need special treatment. It is unfortunate that it's stayed open for so long (I take a lot of the responsibility) but it's not done and has important information.

[andrewjbtw]

It might be superseded by what I'm doing with yet more moabs that need to be edited, so all could be re-archived under a new ticket that combines those with the ones identified here. I haven't reached the point of filing the new ticket because it's very difficult to edit moabs by hand to make changes that affect previous/all versions.

[ndushay]

@andrewjbtw JustiListtman has done a massive remediation of all Moabs with replication errors. If you would like us to check on specific ones as you work on them, perhaps list druids here?