verdi storage backup

docs/source/howto/faq.rst

@@ -111,3 +111,9 @@ When the SSH key pair expires, AiiDA will fail to connect to the remote computer
 This will cause all calculations submitted on that computer to pause.
 To restart them, one needs to generate a new SSH key pair and play the paused processes using ``verdi process play --all``.
 Typically, this is all one needs to do - AiiDA will re-establish the connection to the computer and will continue following the calculations.
+
+How to back up AiiDA data?
+=============================================================================
+
+The most convenient way to back up an AiiDA profile is to use the ``verdi --profile <name> storage backup`` command.
+For more information, see :ref:`how-to:installation:backup`.

docs/source/howto/installation.rst

@@ -547,20 +547,33 @@ See the :doc:`../reference/_changelog` for a list of breaking changes.
 
 .. _how-to:installation:backup:
 
-Backing up your installation
+Backing up your data
 ============================
 
-A full backup of an AiiDA instance and AiiDA managed data requires a backup of:
+General information
+-----------------------------------------
 
-* the AiiDA configuration folder, which is named ``.aiida``.
-  The location of the folder is shown in the output of ``verdi status``.
-  This folder contains, among other things, the ``config.json`` configuration file and log files.
+The most convenient way to back up the data of a single AiiDA profile is to use
 
-* the data stored for each profile.
-  Where the data is stored, depends on the storage backend used by each profile.
+.. code:: bash
 
-The panels below provide instructions for storage backends provided by ``aiida-core``.
-To determine what storage backend a profile uses, call ``verdi profile show``.
+    $ verdi --profile <profile_name> storage backup /path/to/destination
+
+This command automatically manages a subfolder structure of previous backups, and new backups are done in an efficient way (using ``rsync`` hard-link functionality to the previous backup).
+The command backs up everything that's needed to restore the profile later:
+
+* the AiiDA configuration file ``.aiida/config.json``, from which other profiles are removed (see ``verdi status`` for exact location);
+* all the data of the backed up profile (which depends on the storage backend).
+
+The specific procedure of the command and whether it even is implemented depends on the storage backend.
+
+.. note::
+    The ``verdi storage backup`` command is implemented in a way to be as safe as possible to use when AiiDA is running, meaning that it will most likely produce an uncorrupted backup even when data is being modified. However, the exact conditions depend on the specific storage backend and to err on the safe side, only perform a backup when the profile is not in use.
+
+Storage backend specific information
+-----------------------------------------
+
+Alternatively to the CLI command, one can also manually create a backup. This requires a backup of the configuration file  ``.aiida/config.json`` and the storage backend. The panels below provide instructions for storage backends provided by ``aiida-core``. To determine what storage backend a profile uses, call ``verdi profile show``.
 
 .. tip:: Before creating a backup, it is recommended to run ``verdi storage maintain``.
     This will optimize the storage which can significantly reduce the time required to create the backup.
@@ -605,44 +618,47 @@ To determine what storage backend a profile uses, call ``verdi profile show``.
 
 .. _how-to:installation:backup:restore:
 
-Restoring your installation
-===========================
+Restoring data from a backup
+==================================
 
-Restoring a backed up AiiDA installation requires:
+Restoring a backed up AiiDA profile requires:
 
-* restoring the backed up ``.aiida`` folder, with at the very least the ``config.json`` file it contains.
-  It should be placed in the path defined by the ``AIIDA_PATH`` environment variable.
-  To test the restoration worked, run ``verdi profile list`` to verify that all profiles are displayed.
+* restoring the profile information in the AiiDA ``config.json`` file. Simply copy the`profiles` entry from
+  the backed up `config.json`to the one of the running AiiDA instance (see `verdi status` for exact location).
+  Some information (e.g. the database parameters) might need to be updated.
 
-* restoring the data of each backed up profile.
+* restoring the data of of the backed up profile according to the ``config.json`` entry.
   Like the backup procedure, this is dependent on the storage backend used by the profile.
 
 The panels below provide instructions for storage backends provided by ``aiida-core``.
 To determine what storage backend a profile uses, call ``verdi profile show``.
+To test if the restoration worked, run ``verdi -p <profile-name> status`` to verify that AiiDA can successfully connect to the data storage.
 
 .. tab-set::
 
     .. tab-item:: psql_dos
 
-        To fully backup the data stored for a profile using the ``core.psql_dos`` backend, you should restore the associated database and file repository.
+        To restore the backed up data for a profile using the ``core.psql_dos`` backend, you should restore the associated database and file repository.
 
         **PostgreSQL database**
 
-        To restore the PostgreSQL database from the ``.psql`` file that was backed up, first you should create an empty database following the instructions described in :ref:`database <intro:install:database>` skipping the ``verdi setup`` phase.
+        To restore the PostgreSQL database from the ``db.psql`` file that was backed up, first you should create an empty database following the instructions described in :ref:`database <intro:install:database>` skipping the ``verdi setup`` phase.
         The backed up data can then be imported by calling:
 
         .. code-block:: console
 
-            psql -h <database_hostname> -p <database_port> -d <database_name> -W < aiida_backup.psql
+            psql -h <db_hostname> -p <db_port> - U <db_user> -d <db_name> -W < db.psql
+
+        where the parameters need to match with the corresponding AiiDA `config.json` profile entry.
 
         **File repository**
 
-        To restore the file repository, simply copy the directory that was backed up to the location indicated by the ``storage.config.repository_uri`` key returned by the ``verdi profile show`` command.
+        To restore the file repository, simply copy the directory that was backed up to the location indicated in AiiDA `config.json` (or the ``storage.config.repository_uri`` key returned by the ``verdi profile show`` command).
         Like the backing up process, we recommend using ``rsync`` for this:
 
         .. code-block:: console
 
-            rsync -arvz /some/path/aiida_backup <storage.config.repository_uri>
+            rsync -arvz /path/to/backup/container <storage.config.repository_uri>
 
 
 .. _how-to:installation:multi-user:

docs/source/nitpick-exceptions

@@ -150,6 +150,7 @@ py:class concurrent.futures._base.TimeoutError
 py:class concurrent.futures._base.Future
 
 py:class disk_objectstore.utils.LazyOpener
+py:class disk_objectstore.backup_utils.BackupManager
 
 py:class frozenset
 

docs/source/reference/command_line.rst

@@ -567,6 +567,7 @@ Below is a list with all available subcommands.
       --help  Show this message and exit.
 
     Commands:
+      backup     Backup the data storage of a profile.
       info       Summarise the contents of the storage.
       integrity  Checks for the integrity of the data storage.
       maintain   Performs maintenance tasks on the repository.

environment.yml

@@ -12,7 +12,7 @@ dependencies:
 - circus~=0.18.0
 - click-spinner~=0.1.8
 - click~=8.1
-- disk-objectstore~=1.0
+- disk-objectstore~=1.1
 - docstring_parser
 - get-annotations~=0.1
 - python-graphviz~=0.19

pyproject.toml

@@ -24,7 +24,7 @@ dependencies = [
   'circus~=0.18.0',
   'click-spinner~=0.1.8',
   'click~=8.1',
-  'disk-objectstore~=1.0',
+  'disk-objectstore~=1.1',
   'docstring-parser',
   'get-annotations~=0.1;python_version<"3.10"',
   'graphviz~=0.19',

requirements/requirements-py-3.10.txt

@@ -41,7 +41,7 @@ debugpy==1.6.7
 decorator==5.1.1
 defusedxml==0.7.1
 deprecation==2.1.0
-disk-objectstore==1.0.0
+disk-objectstore==1.1.0
 docstring-parser==0.15
 docutils==0.20.1
 emmet-core==0.57.1

requirements/requirements-py-3.11.txt

@@ -41,7 +41,7 @@ debugpy==1.6.7
 decorator==5.1.1
 defusedxml==0.7.1
 deprecation==2.1.0
-disk-objectstore==1.0.0
+disk-objectstore==1.1.0
 docstring-parser==0.15
 docutils==0.20.1
 emmet-core==0.57.1

requirements/requirements-py-3.12.txt

@@ -41,7 +41,7 @@ debugpy==1.8.0
 decorator==5.1.1
 defusedxml==0.7.1
 deprecation==2.1.0
-disk-objectstore==1.0.0
+disk-objectstore==1.1.0
 docstring-parser==0.15
 docutils==0.20.1
 executing==2.0.0

requirements/requirements-py-3.9.txt

@@ -41,7 +41,7 @@ debugpy==1.6.7
 decorator==5.1.1
 defusedxml==0.7.1
 deprecation==2.1.0
-disk-objectstore==1.0.0
+disk-objectstore==1.1.0
 docstring-parser==0.15
 docutils==0.20.1
 emmet-core==0.57.1

src/aiida/cmdline/commands/cmd_storage.py

@@ -165,3 +165,50 @@ def storage_maintain(ctx, full, no_repack, force, dry_run, compress):
     except LockingProfileError as exception:
         echo.echo_critical(str(exception))
     echo.echo_success('Requested maintenance procedures finished.')
+
+
+@verdi_storage.command('backup')
+@click.argument('dest', type=click.Path(file_okay=False), nargs=1)
+@click.option(
+    '--keep',
+    type=int,
+    required=False,
+    help=(
+        'Number of previous backups to keep in the destination, '
+        'if the storage backend supports it. If not set, keeps all previous backups.'
+    ),
+)
+@decorators.with_manager
+@click.pass_context
+def storage_backup(ctx, manager, dest: str, keep: int):
+    """Backup the data storage of a profile.
+
+    The backup is created in the destination `DEST`, in a subfolder that follows the naming convention
+    backup_<timestamp>_<randstr> and a symlink called `last-backup` is pointed to it.
+
+    Destination (DEST) can either be a local path, or a remote destination (reachable via ssh).
+    In the latter case, remote destination needs to have the following syntax:
+
+        [<remote_user>@]<remote_host>:<path>
+
+    i.e., contain the remote host name and the remote path, separated by a colon (and optionally the
+    remote user separated by an @ symbol). You can tune SSH parameters using the standard options given
+    by OpenSSH, such as adding configuration options to ~/.ssh/config (e.g. to allow for passwordless
+    login - recommended, since this script might ask multiple times for the password).
+
+    NOTE: 'rsync' and other UNIX-specific commands are called, thus the command will not work on
+    non-UNIX environments. What other executables are called, depend on the storage backend.
+    """
+
+    storage = manager.get_profile_storage()
+    profile = ctx.obj.profile
+    try:
+        storage.backup(dest, keep)
+    except NotImplementedError:
+        echo.echo_critical(
+            f'Profile {profile.name} uses the storage plugin `{profile.storage_backend}` which does not implement a '
+            'backup mechanism.'
+        )
+    except (ValueError, exceptions.StorageBackupError) as exception:
+        echo.echo_critical(str(exception))
+    echo.echo_success(f'Data storage of profile `{profile.name}` backed up to `{dest}`.')

src/aiida/common/exceptions.py

@@ -48,6 +48,7 @@
     'OutputParsingError',
     'HashingError',
     'StorageMigrationError',
+    'StorageBackupError',
     'LockedProfileError',
     'LockingProfileError',
     'ClosedStorage',
@@ -218,6 +219,10 @@ class StorageMigrationError(DatabaseMigrationError):
     """Raised if a critical error is encountered during a storage migration."""
 
 
+class StorageBackupError(AiidaException):
+    """Raised if a critical error is encountered during a storage backup."""
+
+
 class DbContentError(AiidaException):
     """Raised when the content of the DB is not valid.
     This should never happen if the user does not play directly

src/aiida/common/log.py

@@ -101,6 +101,11 @@ def get_logging_config():
                 'level': lambda: get_config_option('logging.verdi_loglevel'),
                 'propagate': False,
             },
+            'disk_objectstore': {
+                'handlers': ['console'],
+                'level': lambda: get_config_option('logging.disk_objectstore_loglevel'),
+                'propagate': False,
+            },
             'plumpy': {
                 'handlers': ['console'],
                 'level': lambda: get_config_option('logging.plumpy_loglevel'),
@@ -221,7 +226,7 @@ def configure_logging(with_orm=False, daemon=False, daemon_log_file=None):
     # can still configure those manually beforehand through the config options.
     if CLI_LOG_LEVEL is not None:
         for name, logger in config['loggers'].items():
-            if name in ['aiida', 'verdi']:
+            if name in ['aiida', 'verdi', 'disk_objectstore']:
                 logger['level'] = CLI_LOG_LEVEL
 
     # Add the `DbLogHandler` if `with_orm` is `True`

src/aiida/manage/configuration/config.py

@@ -80,6 +80,9 @@ class ProfileOptionsSchema(BaseModel, defer_build=True):
     logging__verdi_loglevel: LogLevels = Field(
         'REPORT', description='Minimum level to log to console when running a `verdi` command.'
     )
+    logging__disk_objectstore_loglevel: LogLevels = Field(
+        'INFO', description='Minimum level to log to daemon log and the `DbLog` table for `disk_objectstore` logger.'
+    )
     logging__db_loglevel: LogLevels = Field('REPORT', description='Minimum level to log to the DbLog table.')
     logging__plumpy_loglevel: LogLevels = Field(
         'WARNING', description='Minimum level to log to daemon log and the `DbLog` table for the `plumpy` logger.'