Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add remote borg backup support #4804

Open
wants to merge 21 commits into
base: main
Choose a base branch
from

Conversation

timdiels
Copy link
Collaborator

@timdiels timdiels commented Jun 9, 2024

This adds support for backing up directly to a remote borg repo. (I tried reaching out earlier)

Why?

I'd like to backup/restore via the UI, resist ransomware and store the backups remotely without storing a local (compressed) copy of the nextcloud data.

Alternatives (https://github.com/nextcloud/all-in-one?tab=readme-ov-file#are-remote-borg-backups-supported):

  • Mount a network file system: Ransomware can encrypt the backups on the network storage.
  • rclone: Ransomware and stores a local copy of the borg repo
  • borg backup the local repo to borgbase: Stores a local compressed copy of the nextcloud data as at least 1 borg backup
  • create your own solution: backups cannot be managed through the UI and others will have to reinvent my work

How to use it

Instead of entering a local repo path (which remains supported), you can choose to instead enter a remote borg repo url. (These screenshots are from a disaster recovery but it's analogous, just ignore the passphrase):

Screenshot from 2024-06-09 15-26-41

The first you try to initialise the repo, the backup container will create an ssh key and foolishly try to init the borg repo resulting in error because you have to authorise the ssh key it generated first. So, e.g. I copy paste the public key shown here to my borgbase repo:

Screenshot from 2024-06-09 15-27-01

Then I can try again as instructed and it should work. The following backup info is shown:

Screenshot from 2024-06-09 14-59-30

Changes

  • Support for storing backups directly at a remote borg repo (Replace commands to not assume the repo is local, e.g. use BORG_REPO env var, ...)
  • Exclude files from backups if we don't plan on restoring them anyway, such as the audit log
  • Restore with borg extract because, at least with a remote repo, borg mount is very slow (20 seconds vs 24 minutes restore for pretty much an empty nextcloud install).

FYI Oddly the original code never restores host-mounts.

Testing

I hacked the code til I could run local deployments of nextcloud-aio without any builds and manually tested:

  • Entering invalid stuff into forms I changed
  • Remote borg repo init
  • Creating and restoring a backup. I created some files in the master container volume; some in an ignored directory to make sure those are not deleted, some in a directory that is part of the backup, after the backup made some changes and made sure that once restored, the ignored files were left alone, new (non-ignored) files were deleted and changed files restored. I can't really open the user UI with my local test setup.
  • Checking a backup
  • Setting up a new nextcloud from backup

TODO

  • I did at one point test local backups, but I should redo that.
  • If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great
  • Update the readme, but I first want to hear your thoughts before putting more time in it

@szaimen szaimen added 3. to review Waiting for reviews enhancement New feature or request borg labels Jun 9, 2024
@szaimen szaimen modified the milestone: next Jun 9, 2024
Signed-off-by: Tim Diels <[email protected]>
@szaimen
Copy link
Collaborator

szaimen commented Jun 17, 2024

Hi, first of all thank you for your contribution!

I've thought a bit about this the last week and came to the conclusion that I do not want to maintain this since it requires an additional server for me for testing (with a not so easy to reproduce setup) and has the potential of many users needing help getting this to work (and thus only a limited usecase). So I would still only test local backup also in the future.

However I would be fine with merging this if you @timdiels would step in as the maintainer of this feature. That means if bug reports or any questions regarding this feature come in, I would ask you for help on the topic. If that is fine for you, I would continue with the review.

FYI Oddly the original code never restores host-mounts.

Yes, this is expected and documented.

TODO

  • I did at one point test local backups, but I should redo that.

Sounds good!

  • If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Yeah, we could create a dev instance from this if we proceed...

  • Update the readme, but I first want to hear your thoughts before putting more time in it

Yes

@szaimen szaimen added 2. developing Work in progress and removed 3. to review Waiting for reviews labels Jun 17, 2024
@timdiels
Copy link
Collaborator Author

timdiels commented Jun 22, 2024

Hi,

However I would be fine with merging this if you @timdiels would step in as the maintainer of this feature. That means if bug reports or any questions regarding this feature come in, I would ask you for help on the topic. If that is fine for you, I would continue with the review.

I'm willing to maintain it, mostly on weekends, for as long as I'm a nextcloud user (which I have been for a couple of years so far).

  • I did at one point test local backups, but I should redo that.

I will do it on the dev build after I've made the change you suggested.

  • If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Is there an easier way to develop than the hacks I did (in a different branch) for local testing? Not sure how you normally test AIO? If it's always via push to github, wait for build and deploy, I will probably keep my debug branch for later.

Note to self:

  • Update readme
  • Local repo should use borg mount and rsync
  • Test local and remote on dev build

@szaimen
Copy link
Collaborator

szaimen commented Jun 24, 2024

I'm willing to maintain it, mostly on weekends, for as long as I'm a nextcloud user (which I have been for a couple of years so far).

Cool, then we can go ahead with this PR. I've invited you to the repo for easier collaboration :)

  • I did at one point test local backups, but I should redo that.

I will do it on the dev build after I've made the change you suggested.

Great :)

  • If we could do a dev build that I could tmp upgrade my actual instance to for testing real backups, that would be great

Is there an easier way to develop than the hacks I did (in a different branch) for local testing? Not sure how you normally test AIO? If it's always via push to github, wait for build and deploy, I will probably keep my debug branch for later.

Usually it is indeed always via push to github, wait for build and deploy. So probably it is easier for you to keep your debug branch 👍

@gregjohnsonsaltaire
Copy link

Very much appreciate the PR guys!
I'm happy to help test informally ... I have 3 nextcloud-aio sites of 20GB, 40GB & 350GB doing borg backups to Hetzner Storageboxes that I'm prepared to convert to the PR ... Regards

@timdiels
Copy link
Collaborator Author

timdiels commented Jul 9, 2024

I tried out local backups on my machine and made all the requested changes I believe. Could we get a a dev build going?

@timdiels
Copy link
Collaborator Author

I manually tested my last change, it fixes the problem. Looks good to merge for me.

@szaimen szaimen modified the milestones: v9.6.0, next Sep 18, 2024
@szaimen szaimen modified the milestones: v9.7.0, next Oct 10, 2024
Copy link
Collaborator

@szaimen szaimen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for coming back so late to you.

I have some remarks. See below.
(After they are resolved, it is in a shape where we can merge this. As I said the plan is to include this in a release together with #5249)

Please also resolve the conflicts. Thanks!

Containers/borgbackup/backupscript.sh Outdated Show resolved Hide resolved
Containers/borgbackup/borg_excludes Show resolved Hide resolved
php/src/Data/ConfigurationManager.php Show resolved Hide resolved
@szaimen
Copy link
Collaborator

szaimen commented Oct 27, 2024

Hi @timdiels sorry for the ping but would you be able to look into my comments above? (I'd like to include this for the next major version of AIO which will likely come out as beta release in ~3 weeks. Until then the PR needs to be in a mergeable state. Thanks a lot already for all the work you've put into this!

@timdiels
Copy link
Collaborator Author

@szaimen Great! I'll have a look today.

@szaimen
Copy link
Collaborator

szaimen commented Oct 27, 2024

@timdiels one last task is then to resolve the conflicts after handling the comments.

Afterwards I will create a final image from the branch for a last testing round.

@timdiels
Copy link
Collaborator Author

@szaimen I believe the only remaining comment was to keep the excludes in sync. I've merged main into it. Could you create the new image?

@szaimen
Copy link
Collaborator

szaimen commented Oct 27, 2024

I believe the only remaining comment was to keep the excludes in sync.

Yes

I've merged main into it. Could you create the new image?

I created new images 👍

@@ -64,7 +64,7 @@ RUN set -ex; \
wget https://getcomposer.org/installer -O - | php -- --install-dir=/usr/local/bin --filename=composer; \
chmod +x /usr/local/bin/composer; \
cd /var/www/docker-aio; \
git clone https://github.com/nextcloud-releases/all-in-one.git --depth 1 .; \
git clone https://github.com/timdiels/nextcloud-aio.git --depth 1 --branch feature/remote-borg-backup3 .; \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last thing to revert before the merge

@szaimen
Copy link
Collaborator

szaimen commented Oct 27, 2024

@timdiels just a few suggestions to not print out borg info to not spam the logs...

@timdiels
Copy link
Collaborator Author

I retested the remote backup, still works.

Signed-off-by: Tim Diels <[email protected]>
@szaimen szaimen added 3. to review Waiting for reviews and removed 2. developing Work in progress labels Oct 27, 2024
@szaimen szaimen modified the milestones: v9.8.0, next Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3. to review Waiting for reviews borg enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants