NFS data dir from inside LXD container

brmiller · March 5, 2018, 3:19am

I’ve read

but still can’t figure out how to accomplish what I’m after. I’ve run OC 8 (!) under QEMU/KVM for 3 years or so, and it’s been grand. I’m now getting into linux containers, and want to run a new NC 13 instance under LXD and ultimately migrate my data from OC8 to NC13. (I tried the upgrade-upgrade-upgrade approach, but it’s too much work, so I’ll just start over.)

One of the things about VMs and containers is that it’s hard to forecast how much data storage you need, and a little difficult to add later (not really, of course). But I’ve already invested into separate file storage for non-NC purposes with redundancy, proper backup strategy, etc. and it would be super convenient to only consider the nextcloud software itself in the container and “pretend” that my data directory “disk” was infinite.

To that end, I tried to mount my storage export over NFS on the LXD host, and then attach it to my container using

lxc config device add nextcloud nextcloud-data disk source=/data/nextcloud path=/home/nextcloud/data

First I shutdown nextcloud and moved /home/nextcloud/data to /home/nextcloud/data.x with the idea that if it worked, I’d just copy from data.x to the new data mount, restart nextcloud and be off to the races. But life in containers is not that easy - I had permission / write issues.

The nextcloud data directory looks to want to be owned by www-data:www-data, but in the container this is some large uid/gid on the host which has absolutely no correlation to anything on the actual NFS server, and I’m not sure if that matters. I’ve gotten around multi-system file-sharing permissions issues in the past by simply creating users on both server and client with the same user id - this works if you can manage it and there aren’t too many users to create.

But in container-land, I’m a little lost. Perhaps this is a @JasonBayton question (), but I’d appreciate some pointers to getting this accomplished. It’s not ostensibly a nextcloud issue I suppose, but this seems like a vibrant forum and certainly someone is doing something similar.

Thanks,
brmiller

JasonBayton · March 7, 2018, 11:18am

Hey

There’s still no clean way of doing this, ultimately you grant ownership of the whole external folder you push into the container to a user in the container (www-data) from outside of the container. So, if www-data in the container is UID 00110, which you can see by:

ls -l /var/lib/lxd/containers/containername/rootfs/var/www/

Then outside of the container, on the host, you’d run:

sudo chown -R 00110:00110 /my/local/share/folder

After which you can add the folder in to the container as a new device:

lxc config device add CONTAINERNAME FOLDERNAME disk source=/my/local/share/folder path=/my/lxd-share/mount/folder

Any interaction outside of the container will need sudo, and you’ll need to make sure you update permissions of anything you put in the share to reflect the UID otherwise it won’t be usable in the container.

brmiller · March 11, 2018, 10:47pm

Thanks @JasonBayton - this is what I thought, but reading you say it made all the difference. For the [complete] solution in case someone else has to do this (or if I forget), here is my complete winning recipe.

On the NFS server (NAS, big storage, etc), I add the following to /etc/exports:

# access to LXD host for container passthru
/data/nextcloud 10.1.0.40(rw,sync,no_root_squash,no_subtree_check)

10.1.0.40 is the IP of the Ubuntu 16.04 box running LXD and hosting my containers. Don’t forget to exportfs -a afterward or restart nfsd.

On the LXD host, create the /data/nextcloud mountpoint add to /etc/fstab

nfsserver:/data/nextcloud /data/nextcloud nfs rsize=8192,wsize=8192,timeo=14,intr      0       0

This will cause the fileshare mount to remount on reboot. After adding this, mount the share from the NFS server on the LXD host using

mount /data/nextcloud

I’m sure there are other parameters I could add here, but this sufficed for today. In the nextcloud container, find where nextcloud is storing the data. For me it was /home/nextcloud/data. Shutdown apache and move this directory aside and create the new mount point (do this in the container):

cd /home/nextcloud
mv data data.x
mkdir data

Then, on the LXD host, add a new disk device to the container using what @JasonBayton provided above:

lxc config device add nextcloud nextcloud-data disk source=/data/nextcloud path=/home/nextcloud/data

Back in the container, cp from the old data directory to the new mounted one:

rsync -avz /home/nextcloud/data.x/* /home/nextcloud/data.x/.* /home/nextcloud/data

Finally, fixup all the permissions. Do this by looking for what host user maps to the container’s www-data user as @JasonBayton provided. Do this on the LXD host:

ls -l /var/lib/lxd/containers/nextcloud/rootfs/var/www

Look for the nextcloud folder - ordinarily it should be owned by the www-data user. Mine was 231105. On the LXD host,

chown -R 231105:231105 /data/nextcloud

(I think I may have also done this on the NFS server at some point.)

Again, I realize this is more of an LXD question than a nextcloud one, but this is a great forum. Thanks again!

hi-ko · September 20, 2018, 1:22pm

interestingly I came up with the same solution with my first nextcloud install. Only point to add: root will never behave as usual on that nfs share since the containers root will be mapped to the id defined in /etc/subuid. e.g. it will not work to rsync -a as root because the mapped id is not allowed to chown. As long the use case only supports only one user in the container, it’s OK.
More Details in Stéphane Graber’s Blog about uid mapping