Skip to content

Usage@Archival storage

Jan Tomášek edited this page Dec 18, 2020 · 5 revisions

Archival storage administration

Action Description
Cleanup Roll back/delete failed objects at all logical storages and changes their state to ROLLED_BACK/DELETED in DB. In order to succeed, all storages must be reachable. In the case of success object with states [ARCHIVAL_FAILURE, ROLLBACK_FAILURE] are set to ROLLED_BACK. Objects in state DELETION_FAILURE are set to DELETED. Method deletes failed files at logical storages but does not touch Archival Storage workspace or Producers' Transfer Area.
Cleanup - all Does the same as Cleanup method but also sets objects with states [PROCESSING, PRE_PROCESSING] to ROLLED_BACK. Also deletes the content of the Archival Storage workspace (the folder used for AIP import and DIP export, configured in arcstorage.tmpFolder property of the Archival Storage service). Does not touch the Producers' Transfer Area.
Set minimal storage count Set minimal storage count. This value is checked upon archival storage start and during removal of a storage. User is responded with error if the deletion of storage would cause breaking the minimal count limit.
Set read only state If true, the storage can perform only read requests. Archival storage is set to read only state automatically during initiation of attaching of new storage.
Set reachability check interval The interval (in minutes) used by periodic reachability check of all attached logical storages. The result of the reachability check (i.e. statement about logical storage reachability) is displayed in the table of logical storages in the Logical storage administration section.

Logical storage administration

This page allows administration of logical storages.

List of storages

Table description:

Attribute Description
Name Name of the storage used in the list of storages.
Host Address of the host of the running instance.
Port Port of the running instance.
Priority Value of type integer that determines the priority of the storage among other storage. The higher the priority, the more preferred the storage is.
Storage type Type of the storage, currently supported types: FS, ZFS, CEPH.
Configuration file JSON configuration file of the storage.
Mode Default is R/W. Storage is set to Write only mode during its initial synchronization, It may also be set to Write only mode manually by admin.
Reachable Boolean attribute, tested and updated by the system periodically and also during every write operation.

Actions description:

Action Description
Attach new storage User clicks Attach new storage, fills the form and clicks Submit. After the attachment, the storage is being synchronized. The synchronization ends when the storage contains all objects stored on other storages. See wiki of the Archival Storage for example storage configurations.
Edit storage User clicks on one the storages in the list, edits the desired attributes and clicks Save and close.
Continue with sync If the storage sync fails admin is expected to investigate the reason and, probably, manually solve it. Then the Continue with sync button should be used: User clicks on one the storages in the list. Then, if the red dot and text states that the synchronization has failed, user navigates to the Synchronization information section and clicks Continue with sync button. The system tries to continue with the synchronization from the timestamp of the last sync failure.
Delete storage User clicks on one the storages in the list and clicks Delete.
Check now Checks the reachability of all attached storages.

Storage detail

Colored dot on the top shows the state of the storage. Green dot means storage is ready and reachable. Orange dot means that storage is being synchronized. Red dot means that storage synchronization has failed (is stucked) or the storage is unreachable.

Storage

Storage attributes. Only some of them can be updated.

Synchronization information

Contains information about the storage synchronization. The synchronization is a one-time process executed after the storage attachment. If the synchronization fails, admin may try to solve the cause and then continue with the synchronization by clicking on the Continue with sync button.

Attribute Description
Synchronization started Timestamp of the synchronization start.
Information updated Timestamp of the last modification of the synchronization information. The information is updated after every copied object, during synchronization error etc.
Phase INIT: System waits for the 'quiet' time (without new writes) and prepares new storage (creates ARCLib dataspace etc.). COPYING_ARCHIVED_OBJECTS: Copying those objects which were already archived on other storages when the synchronization started. PROPAGATING_OPERATIONS: Propagating new remove/renew/delete operations which has occurred during the synchronization. POST_SYNC_CHECK: System waits for the 'quiet' time (without new writes), finishes the synchronization and checks state of all objects synced to the storage. DONE: Synchronization is completed.
Remains How many objects/operations remains in the current sync phase.
Done How many objects were copied / operations were propagated in the current sync phase. If the sync failed at some point of time and then it was continued by admin, this number does not include objects/operations processed before the sync failure.
Exception class The name of the exception class because of which the synchronization has failed. Shown only if the synchronization has failed.
Exception message The message of the exception because of which the synchronization has failed. The message may be the same as exception class name. Shown only if the synchronization has failed.

Storage state

This section contains the information which is typically obtained by calling commands specific to the technology of the logical storages:

FS

Attribute Description
available Free space of the FS (physical)
used Occupied space of the FS (physical)

ZFS

Contains data of zfs list and zpool list commands.

Attribute Description
Dataset > name Name of the ZFS dataset of the logical storage.
Dataset > used Occupied space of the ZFS dataset (logical value.. the physical space occupied is higher because of RAID settings etc.)
Dataset > available Free space of the ZFS dataset (logical, e.g. 500MB means that two 250MB objects may be stored)
Pool > name Name of the ZFS pool of the logical storage.
Pool > health Health of the pool
Pool > size Total space of the ZFS pool (physical)
Pool > alloc Occupied space of the ZFS pool (physical)
Pool > free Free space of the ZFS pool (physical)
Other pool information For example dedup, altroot, expandsz, frag, cap keys of zpool list command

Ceph S3

Contains data obtained through S3 API and the data of ceph -s and ceph df commands.

Attribute Description
accountId id of the ARCLib account
accountName name of the ARCLib account
usedBytes count of bytes used by the account
objectsCount count of RGW objects stored by the account
buckets list of buckets used by the account (possibly just one bucket)
ceph -s output of the ceph -s command
ceph df output of the ceph df command
Clone this wiki locally