Loolwsd and unprivileged LXC Containers

Iā€™ve installed Collabora Online on a debian buster LXC unprivilegend container mostly following NextCloud info in https://nextcloud.com/collaboraonline/; loolwsd start as expected, Nextclud connect to the instance but when i try to open a document i got:

Sep 25 15:27:42 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: mount failed remount [/opt/lool/child-roots/ridzJ5vsTwBcah6P] readonly: Permission denied.
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:42.797451 [ kit_spare_002 ] ERR  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/ridzJ5vsTwBcah6P/] readonly.| common/JailUtil.cpp:59
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:42.797520 [ kit_spare_002 ] WRN  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/ridzJ5vsTwBcah6P/], will link/copy contents.| kit/Kit.cpp:2149
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/ridzJ5vsTwBcah6P/tmp] failed: Permission denied.
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:42.827965 [ kit_spare_002 ] ERR  Failed to unmount [/opt/lool/child-roots/ridzJ5vsTwBcah6P/tmp].| common/JailUtil.cpp:70
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/ridzJ5vsTwBcah6P/lo] failed: Permission denied.
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:42.847363 [ kit_spare_002 ] ERR  Failed to unmount [/opt/lool/child-roots/ridzJ5vsTwBcah6P/lo].| common/JailUtil.cpp:70
Sep 25 15:27:42 vnclpb1 systemd[15367]: opt-lool-child\x2droots-ridzJ5vsTwBcah6P.mount: Succeeded.
Sep 25 15:27:42 vnclpb1 systemd[1]: opt-lool-child\x2droots-ridzJ5vsTwBcah6P.mount: Succeeded.
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/ridzJ5vsTwBcah6P] failed: Permission denied.
Sep 25 15:27:42 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:42.880200 [ kit_spare_002 ] ERR  Failed to unmount [/opt/lool/child-roots/ridzJ5vsTwBcah6P/].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:43.283542 [ kit_spare_002 ] ERR  mknod(/opt/lool/child-roots/ridzJ5vsTwBcah6P//tmp/dev/random) failed. Mount must not use nodev flag. (EPERM: Operation not permitted)| common/JailUtil.cpp:228
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15540-14256 2020-09-25 13:27:43.283625 [ kit_spare_002 ] ERR  mknod(/opt/lool/child-roots/ridzJ5vsTwBcah6P//tmp/dev/urandom) failed. Mount must not use nodev flag. (EPERM: Operation not permitted)| common/JailUtil.cpp:240
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: mount failed remount [/opt/lool/child-roots/Il1oS2dgPsdODGa9] readonly: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15573-14256 2020-09-25 13:27:43.557557 [ kit_spare_003 ] ERR  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/Il1oS2dgPsdODGa9/] readonly.|
+common/JailUtil.cpp:59
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15573-14256 2020-09-25 13:27:43.557623 [ kit_spare_003 ] WRN  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/Il1oS2dgPsdODGa9/], will link/copy
+contents.| kit/Kit.cpp:2149
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: mount failed remount [/opt/lool/child-roots/XuuVkTQOzdi6lfl4] readonly: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15574-14256 2020-09-25 13:27:43.564909 [ kit_spare_004 ] ERR  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/] readonly.|
+common/JailUtil.cpp:59
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15574-14256 2020-09-25 13:27:43.564977 [ kit_spare_004 ] WRN  Failed to mount [/opt/lool/systemplate] -> [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/], will link/copy
+contents.| kit/Kit.cpp:2149
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/Il1oS2dgPsdODGa9/tmp] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15573-14256 2020-09-25 13:27:43.600571 [ kit_spare_003 ] ERR  Failed to unmount [/opt/lool/child-roots/Il1oS2dgPsdODGa9/tmp].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/tmp] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15574-14256 2020-09-25 13:27:43.603914 [ kit_spare_004 ] ERR  Failed to unmount [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/tmp].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/Il1oS2dgPsdODGa9/lo] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15573-14256 2020-09-25 13:27:43.627610 [ kit_spare_003 ] ERR  Failed to unmount [/opt/lool/child-roots/Il1oS2dgPsdODGa9/lo].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/lo] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15574-14256 2020-09-25 13:27:43.642396 [ kit_spare_004 ] ERR  Failed to unmount [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/lo].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 systemd[15367]: opt-lool-child\x2droots-Il1oS2dgPsdODGa9.mount: Succeeded.
Sep 25 15:27:43 vnclpb1 systemd[1]: opt-lool-child\x2droots-Il1oS2dgPsdODGa9.mount: Succeeded.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/Il1oS2dgPsdODGa9] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15573-14256 2020-09-25 13:27:43.661583 [ kit_spare_003 ] ERR  Failed to unmount [/opt/lool/child-roots/Il1oS2dgPsdODGa9/].| common/JailUtil.cpp:70
Sep 25 15:27:43 vnclpb1 systemd[15367]: opt-lool-child\x2droots-XuuVkTQOzdi6lfl4.mount: Succeeded.
Sep 25 15:27:43 vnclpb1 systemd[1]: opt-lool-child\x2droots-XuuVkTQOzdi6lfl4.mount: Succeeded.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: /usr/bin/loolmount: forced unmount of [/opt/lool/child-roots/XuuVkTQOzdi6lfl4] failed: Permission denied.
Sep 25 15:27:43 vnclpb1 loolwsd[14250]: kit-15574-14256 2020-09-25 13:27:43.697419 [ kit_spare_004 ] ERR  Failed to unmount [/opt/lool/child-roots/XuuVkTQOzdi6lfl4/].| common/JailUtil.cpp:70

and in the host system (Proxmox VE 6):

Sep 25 15:27:42 ino kernel: [433028.908691] audit: type=1400 audit(1601040462.792:24): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-104_</var/lib/lxc>" name="/opt/lool/child-roots/ridzJ5vsTwBcah6P/" pid=3673 comm="loolmount" flags="ro, nosuid, nodev, remount, noatime, rbind, silent"
Sep 25 15:27:43 ino kernel: [433029.669132] audit: type=1400 audit(1601040463.552:25): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-104_</var/lib/lxc>" name="/opt/lool/child-roots/Il1oS2dgPsdODGa9/" pid=3813 comm="loolmount" flags="ro, nosuid, nodev, remount, noatime, rbind, silent"
Sep 25 15:27:43 ino kernel: [433029.676506] audit: type=1400 audit(1601040463.560:26): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-104_</var/lib/lxc>" name="/opt/lool/child-roots/XuuVkTQOzdi6lfl4/" pid=3814 comm="loolmount" flags="ro, nosuid, nodev, remount, noatime, rbind, silent"

Iā€™ve tried to disable options like ā€˜mount_jail_treeā€™ and ā€˜capabilitiesā€™ with no luck.

Collabora Online is incompatible with unprivileged containers?!

Thanks.

(also on https://github.com/CollaboraOnline/richdocumentscode/issues/72)

that means thereā€™s nothing we can do here on the forumā€¦ I feel sorry for you.

No, sorry, iā€™m still me. :wink:
Iā€™ve simply posted as a bug there, and as a question here. :wink:

It simply seems strange to me that no one have tried to run Collabora Online in a unprivileged LXC containerā€¦

1 Like

You are not the only one who has tried to run in an unprivileged container, I tried too using Proxmox just like you. But unprivileged containers should remain as they are, I donā€™t like to loosen security further than needed, just for the sake of the lower footprint.
So I ended up with a KVM VM instead of a container. Some more overhead , but more secure and Iā€™m happy with it.

ā€¦but the ā€˜defaultā€™ distribution for Collabora Online is the docker container, and i donā€™t think that a docker container and an LXC container differ too muchā€¦

So, iā€™m a bit confusedā€¦

Not every container is the same. You could say that docker is more made for applications containers and lxc (in proxmox) is made for system containers:
Quote form Linux Container - Proxmox VE

Our primary goal is to offer an environment that provides the benefits of using a VM, but without the additional overhead. This means that Proxmox Containers can be categorized as ā€œSystem Containersā€, rather than ā€œApplication Containersā€.

[image] If you want to run application containers, for example, Docker images, it is recommended that you run them inside a Proxmox Qemu VM. This will give you all the advantages of application containerization, while also providing the benefits that VMs offer, such as strong isolation from the host and the ability to live-migrate, which otherwise isnā€™t possible with containers.

So in the end docker and lxc differ.

1 Like

Surely docker and LXC container are different things, but AFAIK they are based on the same things, eg CGroups and so on.

In Proxmox thereā€™s a way to ā€˜relaxā€™ capability and security feature built around containerization. But looking at logs, it is not clear what loolwsd do. Surely it call ā€˜loolmountā€™ to (bind)mount some dir, but apart thatā€¦

Thereā€™s some techincal docs that explain what loolwsd do? Or someone can point me to a piece of code where i can get this information?

It is hard to understand and so, possibly, solve a problem if thereā€™s little or no clueā€¦

Thanks.

Although Iā€™m not in favor of weakening the default container security, Iā€™m happy to guide you a bit.
Iā€™m not sure if this discussion should take place here, because the issues you are having are not Nextcloud related, rather Collabora/Libreoffice Online in combination with Proxmox/LXC. So perhaps itā€™s better to discuss these technical details in their communities and when interesting linking them here.

Guidance for Proxmox:
I saw that you posted the question on the proxmox mailing list, maybe you should just post it in the proxmox forum [1]. In my opinion the people on the pve-user mailing list are mostly people using it in production for their business, and to be honest this topic tends a bit to tinkering, so maybe you wonā€™t get an answer there.
In the forums you will find a lot of people tinkering, so I think you will get advice there soon enough. (proxmox staff is on the forums too).
I donā€™t have experience myself with dropping linux capabilities inside the containers, but probably that could be the way to get it working, although you mention that you already tried some with caps. Perhaps this [2] is an example to drop the security far enough, but be aware what that means for your host and other containers/vmā€™s.
man lxc.container.conf
man capabilities

Guidance for Collabora/Libreoffice online:
You could try their mailing lists or IRC here [3] and ask them what the requirements are to get it running inside lxc.
Also you could look at the code, the official repo is here [4], or a github mirror for easier search here [5]

Your clue is /usr/bin/loolmount
When searching for loolmount I found these files [6] and [7] interesting.

[1] https://forum.proxmox.com/
[2] https://forum.proxmox.com/threads/docker-not-working-on-alpine-linux-lxc.49216/post-329765
[3] Community Assistance | LibreOffice - Free Office Suite - Based on OpenOffice - Compatible with Microsoft
[4] online - Gitiles
[5] GitHub - LibreOffice/online: Read-only Mirror - no pull request (use https://gerrit.libreoffice.org instead)
[6] online/debian/loolwsd.postinst.in at aa779549a77cdf6b0382f2256515c2165df5308c Ā· LibreOffice/online Ā· GitHub
[7] online/loolwsd.spec.in at 5c9988f2e345ca82e7bb5f5e9bf66a30b82a0446 Ā· LibreOffice/online Ā· GitHub

Surely it is a LOO/PVE trouble, but (also looking at your link [3] Iā€™ve not found specific support (forum, mailing list, ā€¦) for LOO. ;-(

Thanks for the links. Iā€™ve effectively tried (ithink most but surely not all) combination of caps, but for the most container does not start, for other nothing change (eg, LOO does not work).
The thing that seems have some sort of effect are settings in loolwds.xml file false, meaning that setting that at least the phisical PVE server does not complain anymore that the container are doing something nasty, but does not work (same errors about mounting).

So iā€™m still here, seeking some feedbackā€¦

Thanks.

If you add

lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:

to the /etc/pve/lxc/{containerid}.conf you can run the Code container.

1 Like

Hi @nextnoci , I am running LXD/LXC installed via snap so all of my config items for LXD are in /nsap/lxd/ā€¦ so I am having trouble trying out your solution.

It seems like we are supposed to be editing LXD Container configurations using the

lxc config edit <container_name> command with brings up the config in an editor and does a validation check when saving (much like visudo error checks before saving).

I have tried editing in many of these suggestions but they donā€™t seem to stick.

Have you ever used the ubuntu snap environment?

My nextcloud container config looks like this when I use the edit suggested command

  GNU nano 4.8                                            /tmp/lxd_editor_3106367835.yaml                                                      ### This is a YAML representation of the configuration.
### Any line starting with a '# will be ignored.
###
### A sample configuration looks like:
### name: instance1
### profiles:
### - default
### config:
###   volatile.eth0.hwaddr: 00:16:3e:e9:f8:7f
### devices:
###   homedir:
###     path: /extra
###     source: /home/user
###     type: disk
### ephemeral: false
###
### Note that the name is shown but cannot be changed

architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 22.04 LTS amd64 (release) (20230107)
  image.label: release
  image.os: ubuntu
  image.release: jammy
  image.serial: "20230107"
  image.type: squashfs
  image.version: "22.04"
  volatile.base_image: ed7509d7e83f29104ff6caa207140619a8b235f66b5997f1ed6c5e462617fb71
  volatile.cloud-init.instance-id: d6d4773f-cf7a-4ef3-8cc3-15b3a40b31cb
  volatile.eth0.hwaddr: 00:16:3e:52:54:dd
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":>  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":100>  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: bcd84bdb-461c-46bb-94a5-1e2a9fec28bc
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""
created_at: 2023-01-19T13:49:01.102494553Z
name: nextcloud
status: Stopped
status_code: 102
last_used_at: 2023-01-20T15:17:29.808026539Z
location: none
type: container
project: nextcloud

@HeneryH Try this:
lxc config set container_name_goes_here security.syscalls.intercept.mknod true
lxc restart container_name_goes_here
See Printing or downloading pdf not working Ā· Issue #2041 Ā· CollaboraOnline/online Ā· GitHub

Proxmox has a different way to describe lxc containers. Snap is not used and is different again.