Change directory for snap`nextcloud.export` exports

fun · August 8, 2022, 2:52am

The Nextcloud snap contains a utility nextcloud.export that automatically makes a copy of the Nextcloud instance. By default, the export location is /var/snap/nextcloud/common/backups/20190121-153921, where the date is the current date, as explained here:

This utility doesn’t offer a way to change the export directory, and I am trying to come up with a good workaround. For illustration purposes, let’s suppose I want to use /home/fun/nextcloud_backups/ as my backup directory instead.

My idea was to make /var/snap/nextcloud/common/backups/ into a symlink to my desired backup directory:

$ sudo -i
# mkdir /home/fun/nextcloud_backups/
# ln -s /home/fun/nextcloud_backups/ /var/snap/nextcloud/common/backups

(Note that at the time I ran these commands, I hadn’t used nextcloud.export, so the /var/snap/nextcloud/common/backups directory does not yet exist.)

However, trying to export gives me the following permissions error:

# nextcloud.export
WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

mkdir: cannot create directory '/var/snap/nextcloud/common/backups': Permission denied
chmod: cannot access '/var/snap/nextcloud/common/backups/20220808-024447': Permission denied
/snap/nextcloud/31222/bin/export-data: 134: /snap/nextcloud/31222/bin/export-data: cannot create /var/snap/nextcloud/common/backups/20220808-024447/format: Permission denied
Enabling maintenance mode... done
Exporting apps...
rsync: ERROR: cannot stat destination "/var/snap/nextcloud/common/backups/20220808-024447/apps": Permission denied (13)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(653) [Receiver=3.1.2]
Unable to export apps
Disabling maintenance mode... done

How do I give the Nextcloud snap permission to follow this symlink? Or is there a better way to change the export location?

Ultimately, my goal is to store the backups on a separate (internal) hard drive, and set up a cron job that does this on a weekly basis, so if my boot drive gets corrupted, I can wipe it, reinstall Nextcloud, and restore my instance from the other drive.
Therefore, using the export the utility as-is and then mving the exported directory doesn’t solve my problem, because it will require copying and then moving all of my Nextcloud data as opposed to just copying it once.

NaXal · August 8, 2022, 6:44am

Hello,

I am no expert on this, but I think it is a inherit limitation of snap. It works within its confinement. However, you can try sudo snap connect nextcloud:removable-media to allow snap access outside the confinement.

Check if that command helps you pushing the update out to external storage via system link.

That export command has arguments too, like -abcd, -abc flag will export apps, database and configuration while -d is for user data. So if you leave out the data flag, the actual backup size is really small.

User Data folder can be manually or automatically copied via rsync or cp command via scheduled corn job. Where as the export backups can be moved with scheduled mv command.

Thanks.

fun · August 8, 2022, 7:21am

Same problem, unfortunately:

# sudo snap connect nextcloud:removable-media
# nextcloud.export
WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

mkdir: cannot create directory '/var/snap/nextcloud/common/backups': Permission denied
chmod: cannot access '/var/snap/nextcloud/common/backups/20220808-065751': Permission denied
/snap/nextcloud/31222/bin/export-data: 134: /snap/nextcloud/31222/bin/export-data: cannot create /var/snap/nextcloud/common/backups/20220808-065751/format: Permission denied
Enabling maintenance mode... done
Exporting apps...
rsync: ERROR: cannot stat destination "/var/snap/nextcloud/common/backups/20220808-065751/apps": Permission denied (13)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(653) [Receiver=3.1.2]
Unable to export apps
Disabling maintenance mode... done

A weird quirk of snap, FWIW, is that the removable-media permission also allows snaps to access your home directory (see this Stack Exchange thread), so it “should have” worked for my toy example here where I am symlinking to my home directory.

Here is a similar question on the Snapcraft forums:

The solution there is to use a bind mount instead of a symlink. I need to look into this option a little bit more. I don’t want to give Nextcloud exclusive control over the extra HD, I just want it to use one subdirectory for its backups. And I’m concerned about making my filesystem too complicated for me to keep track of. But maybe the safe way is to make the bind mount a part of the cron job, and have it unmount when the backup is done.

But if you do it this way, you can’t restore from the backup in one line using nextcloud.import. You have to manually import the files, which kind of defeats the point of the nextcloud.import/export interface.

Compared to my user data folder (50 GB), the apps, database, and configuration take up essentially no space ( < 1 GB), so if I’m willing to accept that I’ll have to restore a backup piece by piece, I might as well skip nextcloud.export entirely and just rsync -a my entire /var/snap/nextcloud/ directory and not bother with nextcloud.export at all. This will add a little bit of overhead because it copies the Nextcloud binary and default apps, but practically speaking, if I can fit 50 GB on my backup drive, then I can fit 51 GB.

NaXal · August 8, 2022, 8:04am

Hello,

fun:

mkdir: cannot create directory '/var/snap/nextcloud/common/backups': Permission denied
chmod: cannot access '/var/snap/nextcloud/common/backups/20220808-065751': Permission denied
/snap/nextcloud/31222/bin/export-data: 134: /snap/nextcloud/31222/bin/export-data: cannot create /var/snap/nextcloud/common/backups/20220808-065751/format: Permission denied
Enabling maintenance mode... done

Just wondering, did you re-adjust your directory permissions?

sudo chown -R root:root /your/symlink/external/path
sudo chmod 0770 /your/symlink/external/path

Secondly,

That may not work with new installation

With manual method,

Copy user data to backup location
Run Snap export command with -bc flag
Install Snap Nextcloud then install related (same as before) apps
Copy data from backup location to new snap default user data directory
run snap import command with -bc
New installation should be up (you may still need to add trusted domain for new setup)

This is what works for me.

Thanks.

fun · August 8, 2022, 8:19am

I created the directories after sudo -i, so I’m not sure if the chown and chmod commands make a difference? Anyway, same error:

$ sudo -i
[sudo] password for fun:
# mkdir /home/fun/nextcloud_backups
# ln -s /home/fun/nextcloud_backups/ /var/snap/nextcloud/common/backups
# chown -R root:root /home/fun/nextcloud_backups/
# chmod 0770 /home/fun/nextcloud_backups/
# snap connect nextcloud:removable-media
# nextcloud.export
WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

mkdir: cannot create directory '/var/snap/nextcloud/common/backups': Permission denied
chmod: cannot access '/var/snap/nextcloud/common/backups/20220808-080928': Permission denied
/snap/nextcloud/31222/bin/export-data: 134: /snap/nextcloud/31222/bin/export-data: cannot create /var/snap/nextcloud/common/backups/20220808-080928/format: Permission denied
Enabling maintenance mode... done
Exporting apps...
rsync: ERROR: cannot stat destination "/var/snap/nextcloud/common/backups/20220808-080928/apps": Permission denied (13)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(653) [Receiver=3.1.2]
Unable to export apps
Disabling maintenance mode... done

My idea with the “manual method” is that if I ever have to restore from a backup, I will just run nextcloud.export on the default Nextcloud instance (with its sample files etc.) so that I can see where the app, database, config, and data files each end up. Then I will modify this “example backup” by replacing these four items with their counterparts from my rsynced folder and use nextcloud.import. I haven’t tested this, though. But if you read the source code for nextcloud.import on GitHub here you can see that it’s essentially doing the steps from Migrating to a different server — Nextcloud latest Administration Manual latest documentation but with the directory switched.

NaXal · August 8, 2022, 8:23am

Hello,

If you may please refer here → nextcloud.export fails when using a symlink for /var/snap/nextcloud/common/backups · Issue #1290 · nextcloud-snap/nextcloud-snap (github.com), that command somehow works for /mnt & /media only

If possible, try to mount your external drive under /mnt or /media directory and try the process again

Thanks.

fun · August 8, 2022, 8:27am

Huh, so these commands will work as-is if the symlink points to an external drive (mounted to /media/somewhere) but not if it points to a different location on the same drive (e.g. the home folder)? This seems like an issue unto itself, but I guess it’s probably related more to snap’s weird confinement rules than Nextcloud.

I’d like to try this out myself, but my current Nextcloud instance is hosted on a rented cloud server, so I don’t have any external drives to try to mount. In a few months, when I migrate to a physical server, I will give this a shot and let you know how it goes.

NaXal · August 8, 2022, 8:39am

Hello,

I have no idea, I am no expert of Linux but I guess snap is meant for people like me, who aren’t into such deep customizations and wants something plug n play, something which just works out of the box.

It is the snap’s greatest USP and at the same time, biggest weakness too. If you want some deep customizations, snap may not be the right deployment process choice !!

You can always try that in a VM with two vDisks !!

Thanks.

fun · August 8, 2022, 8:41am

Good points. I also chose the snap because I want to use a very “vanilla” setup that will be hard to break. I just also… want to be able to store backups on an external hard drive, haha.

Good idea about the VM. I might try that out this week if I have another slow day.

fun · August 9, 2022, 2:18am

OK, this is really weird. I tried setting things up in a virtual machine. The machine has a virtual boot drive and a virtual hard drive that automatically mounts (according to /etc/fstab) at /media/spare/. The goal is to back up files to /media/spare/nextcloud_backups/. So, the setup is identical in essence to the one discussed in this GitHub issue that @NaXal posted already.

I still get the same error:

fun@ubuntu-server:~$ sudo -i
root@ubuntu-server:~# ln -s /media/spare/nextcloud_backups/ /var/snap/nextcloud/common/backups
root@ubuntu-server:~# nextcloud.export
WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.
 
mkdir: cannot create directory '/var/snap/nextcloud/common/bnextcloud_backups/
root@ubuntu-server:/media/spare/nextcloud_backups# cd ..ackups': Permission denied
chmod: cannot access '/var/snap/nextcloud/common/backups/20220809-015144': Permission denied
/snap/nextcloud/31222/bin/export-data: 134: /snap/nextcloud/31222/bin/export-data: cannot create /var/snap/nextcloud/common/backups/20220809-015144/format: Permission denied
Enabling maintenance mode... done
Exporting apps...
rsync: ERROR: cannot stat destination "/var/snap/nextcloud/common/backups/20220809-015144/apps": Permission denied (13)
rsync error: errors selecting input/output files, dirs (code 3) at main.c(653) [Receiver=3.1.2]
Unable to export apps
Disabling maintenance mode... done

The permissions on the backup directory are as follows:

root@ubuntu-server:~# cd /media/spare/
root@ubuntu-server:/media/spare# ls -l
total 20
-rw-rw-r-- 1 fun  fun      0 Aug  9 01:19 a
-rw-rw-r-- 1 fun  fun      0 Aug  9 01:19 b
-rw-rw-r-- 1 fun  fun      0 Aug  9 01:19 c
drwx------ 2 root root 16384 Aug  9 01:09 lost+found
drwxrwxr-x 2 fun  fun   4096 Aug  9 01:19 nextcloud_backups

The NC snap has the removable-media connection enabled:

root@ubuntu-server:/media/spare# snap connections nextcloud
Interface        Plug                       Slot              Notes
network          nextcloud:network          :network          -
network-bind     nextcloud:network-bind     :network-bind     -
network-observe  nextcloud:network-observe  -                 -
removable-media  nextcloud:removable-media  :removable-media  manual

And modifying the file permissions as suggested above with chown and chmod has no effect.

Note that this is a brand-new Nextcloud instance, not the one that created the issues described in my earlier post.

Here are the logs from the issue debugger script: Pastebin link (Discourse won’t let me paste the full logs because they are too long.)

The only weird thing is at line 395, where there is a warning Nextcloud is not installed - only a limited number of commands are available that repeats every second. However, I think this applies to the interim period where I had installed the Nextcloud snap but hadn’t yet run nextcloud.manual-install user password to actually set it up—i.e. it isn’t relevant.

kyrofa · August 9, 2022, 5:53pm

That’s right-- the confinement model is smart enough to follow symlinks. For the most part, the snap can only write to /var/snap/nextcloud/current/, /var/snap/nextcloud/common, and /media/* or /mnt/* if you connect the removable-media interface. There are a few subdirectories of $HOME as well, but this particular snap doesn’t generally use those.

The best way for accomplishing what you desire is, I’d say, partnering the export with another command to move it where you like. If you don’t rely on the snap to move the export, you can move it wherever you like.

fun · August 9, 2022, 10:34pm

Right, but as my example above demonstrates, there is still a permission issue even when the target directory is in /media/. I think the next thing I’ll try is a bind mount. Or what if I just set /var/snap/nextcloud/common/backups as the permanent mount point of my other hard drive—is this a bad idea?

The downside of this is that, if you are going to back up your data files as well as your config and DB, then you essentially need to have your boot drive at < 50% full at all times. Or you can back up the files with a separate command, but then you can’t use nextcloud.import to restore from the backup.

NaXal · August 10, 2022, 11:57am

Hello,

@kyrofa is the snap build maintainer so I guess he is the right and best person so shade some light into it

But I see import command works with -abcd flags. So it is possible to manually backup the files and use snap export command with -bc flags to generate db and config backup only and same can be reversed with import command while manually copying the user files back into /var/snap/nextcloud/common/nextcloud/data directory !!

I have done it multiple times to change servers, it works for me flawlessly !!

Thanks.

fun · August 11, 2022, 6:26am

Aha, I didn’t realize that @kyrofa was the maintainer of the snap. Thanks for your hard work.

I’m glad to hear that it’s possible to import the files separately, and I will use this method if the symlink doesn’t work. However—and I don’t mean to be combative, but—this is a workaround, not a solution, and there are numerous mentions on other forums and in GitHub issues of the possibility of redirecting backups to an external drive by enabling the removable-media socket and symlinking /var/snap/nextcloud/common/backups to the target. I really want to figure out why this recommended approach is not working, even on the minimal VM installation described above. (Can anyone reproduce my issue? Did it used to work and then undergo a regression in a recent update?)

Moreover, I have now tried using a bind mount instead of a symlink as suggested here again on a brand-new installation in an Ubuntu Server 22.04 virtual machine, and still get them same cannot create directory error, even after snap connect nextcloud:removable-media and a reboot. Here’s /etc/fstab:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/ubuntu-vg/ubuntu-lv during curtin installation
/dev/disk/by-id/dm-uuid-LVM-Yw6VlRKFv95WbMTPmxmTE8n2gVk0FuxZXBRwrcT23Ow8QPqTQxh8gRCo34grLgOW / ext4 defaults 0 1
# /boot was on /dev/sda2 during curtin installation
/dev/disk/by-uuid/e48cd0e6-1f5c-40d1-877f-d06b35272446 /boot ext4 defaults 0 1
/dev/disk/by-uuid/5975cc07-e438-4b7a-b5c3-9fe5e4c94716 /mnt/spare auto nosuid,nodev,nofail,x-gvfs-show 0 0
/mnt/spare/nextcloud_backups /var/snap/nextcloud/common/backups none bind,nobootwait 0 0

fun · August 11, 2022, 6:33am

Let me be a little more explicit about the steps to reproduce this issue:

In VirtualBox, create a machine with two hard drives.
Install Ubuntu Server 22.04 to one of the hard drives, sudo apt update && sudo apt upgrade to get it current.
Format the other drive as ext4 and add something like
/dev/disk/by-uuid/5975cc07-e438-4b7a-b5c3-9fe5e4c94716 /mnt/spare auto nosuid,nodev,nofail,x-gvfs-show 0 0 to /etc/fstab to get this drive to mount on startup to /mnt/spare.
sudo snap install nextcloud, then nextcloud.manual-install to set it up.
mkdir /mnt/spare/nextcloud_backups
ln -s /mnt/spare/nextcloud_backups /var/snap/nextcloud/common/backups
sudo snap connect nextcloud:removable-media
Reboot.
sudo nextcloud.export produces the permissions error.

Note that according to the closure note on the GitHub issue linked, this should have worked. But the issue was closed more than 2 years ago.

devnull · August 11, 2022, 6:42am

I agree to @NaXal. Nextcloud Snap is Nextcloud Snap and maybe not all is possible.

OT

offtopic (but still frustrated from Snap Nextcloud):
A few years ago i wanted to soft migrate a Snap Nextcloud. For this i would have needed two different (sub)domain names for the same nextcloud, each with its own SSL certificate. The goal was actually only to migrate the name gently. Could not. Asked in this forum. With a normal installation this takes a few minutes and it is standard. Until today I have not found a solution for this. Doesn’t seem to be necessary for snap installations. Reverse proxy was no option and is also not needed (normally).

fun · August 11, 2022, 6:50am

I don’t want this thread to be a general gripe session about the Nextcloud snap. It’s a great package for what it is—a secure container for an NC server that is easy to install and set up. I just want to know what is and isn’t supposed to be possible with the NC snap. The official docs say nothing about backing up (directly) to external media, but the GH issues I have linked to make this out to be a possibility. If it’s not possible, then perhaps a comment on the GH issues would be nice to warn Googlers that this feature no longer exists. If it is possible, then this is a useful feature, and IMO it would be nice to add it to the snap documentation to ensure that its functionality is maintained going forward.

kyrofa · August 11, 2022, 6:52am

I just ran through this to check:

$ sudo mkdir /mnt/foo

$ sudo snap install nextcloud
nextcloud 24.0.3snap1 from Nextcloud✓ installed

$ sudo nextcloud.manual-install admin admin
Nextcloud was successfully installed

$ sudo ln -s /mnt/foo /var/snap/nextcloud/common/backups

$ sudo snap connect nextcloud:removable-media 

$ sudo nextcloud.export
WARNING: This functionality is still experimental and under
development, use at your own risk. Note that the CLI interface is
unstable, so beware if using from within scripts.

Enabling maintenance mode... done
Exporting apps...
              0 100%    0.00kB/s    0:00:00 (xfr#0, to-chk=0/1)
Exporting database...
Exporting config...
Exporting data...
         24.25M 100%  220.28MB/s    0:00:00 (xfr#36, to-chk=0/45) 

Successfully exported /var/snap/nextcloud/common/backups/20220810-234726
Disabling maintenance mode... done

$ sudo ls /mnt/foo/20220810-234726/
apps  config.php  data	database.sql  format

Can you confirm that both /mnt and /mnt/spare are owned by root?

Yes, sadly the only way to avoid trolls is to log issues and have discussions in github instead of here, which you’re always welcome and encouraged to do. As a bonus, I’m generally a bit more responsive there.

I appreciate your attitude, no combat taken . We’ll get to the bottom of what’s happening here. I realize, of course, that this UX is not great. I’m a bit bound by confinement in this case-- there’s no way for the snap to know what plugs are connected, and even with removable-media the number of places that the snap can actually write a backup are so limited that making it a parameter seemed like even more of a footgun. I wonder if just printing a tarfile to stdout which could be redirected would be a better approach, but it also feels like a wonky UX. Maybe the command could just grow that as an option. Not sure off the top of my head how to do that without first needing to copy everything into one place though, which sort of defeats the whole purpose. Research needed.

fun · August 11, 2022, 7:30am

OK, this got me a little further: I had these files owned by root, but /mnt/spare/nextcloud_backups was owned by my user account! By correcting this, I was able to get the export to work just like yours, so I’ll mark this as solved.

However, even after checking permissions, I was not able to get the output to go to my home folder (which, I now realize, was the setup that was failing in my post above).

In summary, after setting ownership to root and adding removable media:

Symlink from /var/snap/nextcloud/common/backups to /mnt/something works
Symlink from /var/snap/nextcloud/common/backups to /home/fun/something does not work
Bind mount from /var/snap/nextcloud/common/backups to /mnt/something does not work

The second case surprised me because my impression from this (possibly outdated) SE question was that the removable-media plug also included the home folder. But upon closer reading, there appears to be a separate plug called home. But the nextcloud snap doesn’t have this plug:

$ sudo snap connect nextcloud:removable-media
error: snap "nextcloud" has no plug named "home"

The third case surprises me because I thought bind mounts were supposed to be opaque to snaps—i.e. that the snap would follow it as through the external drive were a part of its own snap directory.

I would LOVE this interface. It’s common in all sorts of Unix core utils to be able to direct things to stdout, even ugly binary files, so that the user can compose the output to other places. Maybe it could be a nondefault flag, something like -o, where we can use sudo nextcloud.export -abcd --stdout > ~/todays_backup.tar or something. I think (but I don’t know a lot of shell) that there is a way to check if the output is a terminal and refuse to print it if so. In fact, this is what the tar utility itself does:

$ touch a b c d
$ tar -cv a b c d
tar: Refusing to write archive contents to terminal (missing -f option?)
tar: Error is not recoverable: exiting now
$ tar -cv a b c d > res.tar
a
b
c
d

So, my main feature requests for the Nextcloud snap at this point (which you are free to disregard) are

nextcloud:home plug in the snap
Interface to redirect output from nextcloud.export to stdout

kyrofa · August 11, 2022, 7:49am

Excellent.

This is expected, because:

I’ve never seen that SE question/answer, but it’s a misunderstanding (and dare I suggest you may be misunderstanding the original misunderstanding), as my colleague Zygmunt attempted to explain in the comments and got shot down pretty quickly. The utility to which the answer links is setting up a pivot root for execution, and isn’t responsible for the actual plug-based confinement model at all. The removable-media plug does not include the home directory, that would indeed be silly. This is probably more detail than you care about, but you can easily see what the plug actually does in the component that IS responsible for confinement: snapd.

tl;dr; the nextcloud snap does not, by design, have access to any part of your home directory other than ~/snap/nextcloud/{current,common}.

This is actually probably due to the pivot root I mentioned above. That is setup by snapd/systemd, and it won’t inherit any mounting you do unless you happen to get it in place before the snapd-related systemd units run. This is hacky: I don’t recommend it.

This would be tremendously dangerous, and isn’t something I’m willing to consider with the current state of snap tech. The home interface is automatically connected, which is something one cannot opt out of. Having a web server listening on the open internet with default access to everyone’s home directories is not a power/responsibility I desire.

That, I agree, would be worthwhile. Though like I said, doing it without requiring double the space anyway might prove impossible. Any thoughts you have on the matter would be appreciated, but would perhaps be better suited to an issue on github.