Loolwsd and unprivileged LXC Containers

I’ve installed Collabora Online on a debian buster LXC unprivilegend container mostly following NextCloud info in https://nextcloud.com/collaboraonline/; loolwsd start as expected, Nextclud connect to the instance but when i try to open a document i got:

Sep 25 15:27:42 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: mount failed remount [/opt/lool/child-roots/ridzJ5vsTwBcah6P] readonly: Permission denied.
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:42.797451 [ kit_spare_002 ] ERR  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/ridzJ5vsTwBcah6P/] readonly.| common/JailUtil.cpp:59
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:42.797520 [ kit_spare_002 ] WRN  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/ridzJ5vsTwBcah6P/], will link/copy contents.| kit/Kit.cpp:2149
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/ridzJ5vsTwBcah6P/tmp] failed: Permission denied.
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:42.827965 [ kit_spare_002 ] ERR  Failed to unmount [/opt/lool/child-roots/ridzJ5vsTwBcah6P/tmp].| common/JailUtil.cpp:70
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/ridzJ5vsTwBcah6P/lo] failed: Permission denied.
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:42.847363 [ kit_spare_002 ] ERR  Failed to unmount [/opt/lool/child-roots/ridzJ5vsTwBcah6P/lo].| common/JailUtil.cpp:70
Sep 25 15:27:42 vnclpb1 systemd[15367]: opt-lool-child\x2droots-ridzJ5vsTwBcah6P.mount: Succeeded.
Sep 25 15:27:42 vnclpb1 systemd[1]: opt-lool-child\x2droots-ridzJ5vsTwBcah6P.mount: Succeeded.
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/ridzJ5vsTwBcah6P] failed: Permission denied.
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:42.880200 [ kit_spare_002 ] ERR  Failed to unmount [/opt/lool/child-roots/ridzJ5vsTwBcah6P/].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:43.283542 [ kit_spare_002 ] ERR  mknod(/opt/lool/child-roots/ridzJ5vsTwBcah6P//tmp/dev/random) failed. Mount must not use nodev flag. (EPERM: Operation not permitted)| common/JailUtil.cpp:228
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:43.283625 [ kit_spare_002 ] ERR  mknod(/opt/lool/child-roots/ridzJ5vsTwBcah6P//tmp/dev/urandom) failed. Mount must not use nodev flag. (EPERM: Operation not permitted)| common/JailUtil.cpp:240
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: mount failed remount [/opt/lool/child-roots/Il1oS2dgPsdODGa9] readonly: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15573-14256 2020-09-25 13:27:43.557557 [ kit_spare_003 ] ERR  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/Il1oS2dgPsdODGa9/] readonly.|
+common/JailUtil.cpp:59
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15573-14256 2020-09-25 13:27:43.557623 [ kit_spare_003 ] WRN  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/Il1oS2dgPsdODGa9/], will link/copy
+contents.| kit/Kit.cpp:2149
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: mount failed remount [/opt/lool/child-roots/XuuVkTQOzdi6lfl4] readonly: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15574-14256 2020-09-25 13:27:43.564909 [ kit_spare_004 ] ERR  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/] readonly.|
+common/JailUtil.cpp:59
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15574-14256 2020-09-25 13:27:43.564977 [ kit_spare_004 ] WRN  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/], will link/copy
+contents.| kit/Kit.cpp:2149
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/Il1oS2dgPsdODGa9/tmp] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15573-14256 2020-09-25 13:27:43.600571 [ kit_spare_003 ] ERR  Failed to unmount [/opt/lool/child-roots/Il1oS2dgPsdODGa9/tmp].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/tmp] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15574-14256 2020-09-25 13:27:43.603914 [ kit_spare_004 ] ERR  Failed to unmount [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/tmp].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/Il1oS2dgPsdODGa9/lo] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15573-14256 2020-09-25 13:27:43.627610 [ kit_spare_003 ] ERR  Failed to unmount [/opt/lool/child-roots/Il1oS2dgPsdODGa9/lo].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/lo] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15574-14256 2020-09-25 13:27:43.642396 [ kit_spare_004 ] ERR  Failed to unmount [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/lo].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 systemd[15367]: opt-lool-child\x2droots-Il1oS2dgPsdODGa9.mount: Succeeded.
Sep 25 15:27:43 vnclpb1 systemd[1]: opt-lool-child\x2droots-Il1oS2dgPsdODGa9.mount: Succeeded.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/Il1oS2dgPsdODGa9] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15573-14256 2020-09-25 13:27:43.661583 [ kit_spare_003 ] ERR  Failed to unmount [/opt/lool/child-roots/Il1oS2dgPsdODGa9/].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 systemd[15367]: opt-lool-child\x2droots-XuuVkTQOzdi6lfl4.mount: Succeeded.
Sep 25 15:27:43 vnclpb1 systemd[1]: opt-lool-child\x2droots-XuuVkTQOzdi6lfl4.mount: Succeeded.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/XuuVkTQOzdi6lfl4] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15574-14256 2020-09-25 13:27:43.697419 [ kit_spare_004 ] ERR  Failed to unmount [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/].| common/JailUtil.cpp:70

and in the host system (Proxmox VE 6):

Sep 25 15:27:42 ino kernel: [433028.908691] audit: type=1400 audit(1601040462.792:24): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-104_</var/lib/lxc>" name="/opt/lool/child-roots/ridzJ5vsTwBcah6P/" pid=3673 comm="loolmount" flags="ro, nosuid, nodev, remount, noatime, rbind, silent"
Sep 25 15:27:43 ino kernel: [433029.669132] audit: type=1400 audit(1601040463.552:25): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-104_</var/lib/lxc>" name="/opt/lool/child-roots/Il1oS2dgPsdODGa9/" pid=3813 comm="loolmount" flags="ro, nosuid, nodev, remount, noatime, rbind, silent"
Sep 25 15:27:43 ino kernel: [433029.676506] audit: type=1400 audit(1601040463.560:26): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-104_</var/lib/lxc>" name="/opt/lool/child-roots/XuuVkTQOzdi6lfl4/" pid=3814 comm="loolmount" flags="ro, nosuid, nodev, remount, noatime, rbind, silent"

I’ve tried to disable options like ‘mount_jail_tree’ and ‘capabilities’ with no luck.

Collabora Online is incompatible with unprivileged containers?!

Thanks.

(also on https://github.com/CollaboraOnline/richdocumentscode/issues/72)

that means there’s nothing we can do here on the forum… I feel sorry for you.

No, sorry, i’m still me. :wink:
I’ve simply posted as a bug there, and as a question here. :wink:

It simply seems strange to me that no one have tried to run Collabora Online in a unprivileged LXC container…

1 Like

You are not the only one who has tried to run in an unprivileged container, I tried too using Proxmox just like you. But unprivileged containers should remain as they are, I don’t like to loosen security further than needed, just for the sake of the lower footprint.
So I ended up with a KVM VM instead of a container. Some more overhead , but more secure and I’m happy with it.

…but the ‘default’ distribution for Collabora Online is the docker container, and i don’t think that a docker container and an LXC container differ too much…

So, i’m a bit confused…

Not every container is the same. You could say that docker is more made for applications containers and lxc (in proxmox) is made for system containers:
Quote form https://pve.proxmox.com/wiki/Linux_Container

Our primary goal is to offer an environment that provides the benefits of using a VM, but without the additional overhead. This means that Proxmox Containers can be categorized as “System Containers”, rather than “Application Containers”.

[image] If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox Qemu VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isn’t possible with containers.

So in the end docker and lxc differ.

1 Like

Surely docker and LXC container are different things, but AFAIK they are based on the same things, eg CGroups and so on.

In Proxmox there’s a way to ‘relax’ capability and security feature built around containerization. But looking at logs, it is not clear what loolwsd do. Surely it call ‘loolmount’ to (bind)mount some dir, but apart that…

There’s some techincal docs that explain what loolwsd do? Or someone can point me to a piece of code where i can get this information?

It is hard to understand and so, possibly, solve a problem if there’s little or no clue…

Thanks.

Although I’m not in favor of weakening the default container security, I’m happy to guide you a bit.
I’m not sure if this discussion should take place here, because the issues you are having are not Nextcloud related, rather Collabora/Libreoffice Online in combination with Proxmox/LXC. So perhaps it’s better to discuss these technical details in their communities and when interesting linking them here.

Guidance for Proxmox:
I saw that you posted the question on the proxmox mailing list, maybe you should just post it in the proxmox forum [1]. In my opinion the people on the pve-user mailing list are mostly people using it in production for their business, and to be honest this topic tends a bit to tinkering, so maybe you won’t get an answer there.
In the forums you will find a lot of people tinkering, so I think you will get advice there soon enough. (proxmox staff is on the forums too).
I don’t have experience myself with dropping linux capabilities inside the containers, but probably that could be the way to get it working, although you mention that you already tried some with caps. Perhaps this [2] is an example to drop the security far enough, but be aware what that means for your host and other containers/vm’s.
man lxc.container.conf
man capabilities

Guidance for Collabora/Libreoffice online:
You could try their mailing lists or IRC here [3] and ask them what the requirements are to get it running inside lxc.
Also you could look at the code, the official repo is here [4], or a github mirror for easier search here [5]

Your clue is /usr/bin/loolmount
When searching for loolmount I found these files [6] and [7] interesting.

[1] https://forum.proxmox.com/
[2] https://forum.proxmox.com/threads/docker-not-working-on-alpine-linux-lxc.49216/post-329765
[3] https://www.libreoffice.org/get-help/community-support/
[4] https://git.libreoffice.org/online/
[5] https://github.com/LibreOffice/online
[6] https://github.com/LibreOffice/online/blob/aa779549a77cdf6b0382f2256515c2165df5308c/debian/loolwsd.postinst.in#L7-L8
[7] https://github.com/LibreOffice/online/blob/5c9988f2e345ca82e7bb5f5e9bf66a30b82a0446/loolwsd.spec.in#L143-L144

Surely it is a LOO/PVE trouble, but (also looking at your link [3] I’ve not found specific support (forum, mailing list, …) for LOO. ;-(

Thanks for the links. I’ve effectively tried (ithink most but surely not all) combination of caps, but for the most container does not start, for other nothing change (eg, LOO does not work).
The thing that seems have some sort of effect are settings in loolwds.xml file false, meaning that setting that at least the phisical PVE server does not complain anymore that the container are doing something nasty, but does not work (same errors about mounting).

So i’m still here, seeking some feedback…

Thanks.

If you add

lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:

to the /etc/pve/lxc/{containerid}.conf you can run the Code container.

1 Like