Mounting external volume in container permission denied where user can write

misacek007 · March 21, 2021, 2:46pm

Nextcloud version (eg, 20.0.5): 20.0.8 (nextcloud:stable-fpm container)
Operating system and version (eg, Ubuntu 20.04): Ubuntu 20.04
Apache or nginx version (eg, Apache 2.4.25): nginx-1.19.8
PHP version (eg, 7.4): `n/a``

The issue you are facing:

I’m trying to mount external storage (/syncthing/fotky) in the container. The
OS directory is not owned by the user running nextcloud container (nextcloud)
but user nextcloud is in the group that can access and write that directory.

Is this the first time you’ve seen this error? (Y/N): y

Steps to replicate it:

See below for the exact description.

The output of your Nextcloud log in Admin > Logging:

Error   PHP     Error: opendir(/syncthing/fotky/test/): failed to open dir: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#132        23 minutes ago
Error   PHP     Error: opendir(/syncthing/fotky/rodinka/): failed to open dir: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#132     an hour ago
Error   PHP     Error: opendir(/syncthing/fotky/rodinka/rok.osmy): failed to open dir: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#132              2 hours ago
Error   PHP     Error: file_put_contents(/syncthing/fotky/rodinka/rok.osmy/test.md): failed to open stream: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#276         2 hours ago
Error   PHP     Error: copy(//syncthing/fotky/rodinka/rok.osmy/test.md): failed to open stream: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#350             2 hours ago
Error   PHP     Error: fopen(/syncthing/fotky/rodinka/rok.osmy/test.md): failed to open stream: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#355             2 hours ago
Error   PHP     Error: file_put_contents(/syncthing/fotky/rodinka/rok.osmy/test.md): failed to open stream: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#276         2 hours ago
Error   PHP     Error: copy(//syncthing/fotky/rodinka/rok.osmy/test.md): failed to open stream: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#350             2 hours ago
Error   PHP     Error: file_get_contents(/syncthing/fotky/rodinka/rok.osmy/test.md): failed to open stream: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#272         2 hours ago
Error   PHP     Error: file_get_contents(/syncthing/fotky/rodinka/rok.osmy/test.md): failed to open stream: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#272         2 hours ago

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$ podman exec -ti -u www-data nextcloud cat config/config.php
<?php
$CONFIG = array (
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'passwordsalt' => 'u+iHvc7zmz9yDqFo5lqGafbPWXQ6Sg',
  'secret' => '5xAy1BkJFojqyuzVLsXGuVV1VN2j5X7dxis5T8f7oUfHBSak',
  'trusted_domains' =>
  array (
    0 => 'cloud.misackovo.eu',
    1 => '172.17.0.2',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '20.0.8.1',
  'overwrite.cli.url' => 'https://cloud.misackovo.eu',
  'dbname' => 'nextcloud',
  'dbhost' => '127.0.0.1',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'nextcloud',
  'installed' => true,
  'instanceid' => 'ocxmh1y5sb2l',
  'check_data_directory_permissions' => false,
);


$ podman exec -ti -u www-data nextcloud cat /usr/local/etc/php-fpm.d/www.conf
[www]
user = www-data
group  = nogroup
; pid /var/html/html/data/php-fpm.pid

listen = 127.0.0.1:9000

pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3

The output of your Apache/nginx/system log in /var/log/____:

IMHO irelevant to the issue.

I’m trying to run rootless nextcloud:fpm-stable + mariadb in a container using
podman. I’m using nginx as a proxy.

I’m trying to mount external storage (/syncthing/fotky) in the container. The
OS directory is not owned by the user running nextcloud container (nextcloud)
but user nextcloud is in the group that can access and write that directory.

Container is running as privileged and it’s www-data user running php-fpm group
is changed to “nogroup”. User www-data can create files in
/syncthing/fotky/test directory as it is writable by group “nogroup”. However
nextcloud is not able to do the same and says “Permission denied”.

My best guess would be that it is not able for some reason to write to
directory owned by the group it is running. Should it work?

More precisely, if php-fpm is running under group “nogroup” should it be able
to write to the directory writable by group “nogroup”? At the moment it seems
that it is not.

Changing volume directory permissions to “777” all seems to work fine which
seems to point towards that hypothesis as well.

See below for the exact commands I’m trying.

–

** System unit that runs the container.**

nextcloud@test-pi-priv:~$ systemctl --user cat container-nextcloud
# /home/nextcloud/.config/systemd/user/container-nextcloud.service
# container-nextcloud.service
# autogenerated by Podman 3.0.1
# Sun Mar 21 14:51:02 CET 2021

[Unit]
Description=Podman container-nextcloud.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStartPre=/bin/rm -f %t/container-nextcloud.pid %t/container-nextcloud.ctr-id
ExecStart=/usr/bin/podman container run --conmon-pidfile %t/container-nextcloud.pid --cidfile %t/container-nextcloud.ctr-id --cgroups=no-conmon --replace --name nextcloud --privileged=True --an
notation run.oci.keep_original_groups=1 --env MYSQL_HOST=127.0.0.1 --env MYSQL_DATABASE=nextcloud --env MYSQL_USER=nextcloud --env NEXTCLOUD_ADMIN_USER=admin --env-file /home/nextcloud/containe
rs/conf/nextcloud/.env --pod nextcloud-pod --volume data_nextcloud:/var/www/html --volume /home/syncthing/syncthing:/syncthing:rw --volume /home/nextcloud/containers/conf/nextcloud/www.conf:/us
r/local/etc/php-fpm.d/www.conf --detach=True quay.io/misacek/nextcloud-custom:stable-fpm
ExecStop=/usr/bin/podman container stop --ignore --cidfile %t/container-nextcloud.ctr-id -t 10
ExecStopPost=/usr/bin/podman container rm --ignore -f --cidfile %t/container-nextcloud.ctr-id
PIDFile=%t/container-nextcloud.pid
Type=forking

[Install]
WantedBy=multi-user.target default.target

# /home/nextcloud/.config/systemd/user/container-nextcloud.service.d/override.conf
[Service]
After=container-mariadb network-online.target
# Requires=container-mariadb
BindsTo=container-mariadb

OS view of processes

$ ps axfu
nextclo+   12574  0.0  0.1  21704  9224 ?        Ss   08:11   0:02 /lib/systemd/systemd --user
nextclo+   12575  0.0  0.0 106240  2692 ?        S    08:11   0:00  \_ (sd-pam)
nextclo+   12712  0.0  0.0   7108  3336 ?        Ss   08:11   0:01  \_ /usr/bin/dbus-daemon --session --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
nextclo+   13995  0.0  0.0   6048  2468 ?        Ss   08:12   0:24  \_ /usr/bin/fuse-overlayfs -o ,lowerdir=/home/nextcloud/.local/share/containers/storage/overlay/l/BFY6B3MPUZWVAHV7AP7PWKMUES:
nextclo+   13997  0.0  0.0  80516  2044 ?        Ssl  08:12   0:00  \_ /usr/libexec/podman/conmon --api-version 1 -c 5bc48f680b13e7b5fb93033a880c75516e5c64394a6fe743b79bf83582eaf252 -u 5bc48f68
166534     14000  0.2  1.3 1895376 112092 ?      Ssl  08:12   0:56  |   \_ mysqld
nextclo+   63359  0.0  0.0   7040  3388 ?        Ss   14:51   0:00  \_ /usr/bin/fuse-overlayfs -o ,lowerdir=/home/nextcloud/.local/share/containers/storage/overlay/l/RWGCJ36AB3KQWACFQK2TYCDTOX:
nextclo+   63361  0.0  0.0  80516  2108 ?        Ssl  14:51   0:00  \_ /usr/libexec/podman/conmon --api-version 1 -c 77f176cb5ab6a7d054a3ed4ae28c3283cde8d32afd8308d6fc8d92ab508b7798 -u 77f176cb
nextclo+   63364  0.0  0.4 276292 33660 ?        Ss   14:51   0:00      \_ php-fpm: master process (/usr/local/etc/php-fpm.conf)
165568     63387  0.2  0.6 310984 54960 ?        S    14:51   0:02          \_ php-fpm: pool www
165568     63388  0.2  0.7 313080 59704 ?        S    14:51   0:01          \_ php-fpm: pool www
165568     66228  0.2  0.4 282708 39372 ?        S    14:52   0:01          \_ php-fpm: pool www

Container view list of processes:

$ podman exec -ti nextcloud ps axfu
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         115  0.0  0.0   7636  2780 pts/0    Rs+  14:04   0:00 ps axfu
root           1  0.0  0.4 276292 33660 ?        Ss   13:51   0:00 php-fpm: master process (/usr/local/etc/php-fpm.conf)
www-data      14  0.2  0.6 310984 54960 ?        S    13:51   0:02 php-fpm: pool www
www-data      15  0.2  0.7 313080 59704 ?        S    13:51   0:01 php-fpm: pool www
www-data     109  0.2  0.4 282708 39372 ?        S    13:52   0:01 php-fpm: pool www

** Container process is running as www-data/nogroup**
$ podman exec -ti -u www-data nextcloud stat -c ‘%u %g’ /proc/14
33 65534

$ podman exec -ti -u www-data nextcloud grep nogroup /etc/group
nogroup:x:65534:www-data

Container www-data user is in group nogroup:
$ podman exec -ti -u www-data nextcloud id
uid=33(www-data) gid=33(www-data) groups=33(www-data),0(root),65534(nogroup)

OS view of data directory:
$ ls -la /home/syncthing/syncthing
total 16
drwxrwsrwx+ 4 syncthing-user syncthing 4096 Mar 21 15:03 .
drwxrws–x+ 5 syncthing-user syncthing 4096 Mar 21 14:21 …
drwxrws—+ 2 syncthing-user syncthing 4096 Mar 21 15:03 test

Container view of data directory
$ podman exec -ti -u www-data nextcloud ls -la /syncthing/fotky
total 16
drwxrwsrwx+ 4 nobody nogroup 4096 Mar 21 14:03 .
drwxrws–x+ 5 nobody nogroup 4096 Mar 21 13:21 …
drwxrws—+ 2 nobody nogroup 4096 Mar 21 14:03 test

Adding the directory as an external storage it can be addded and verified:

$ podman exec -u www-data nextcloud /var/www/html/occ files_external:list
+----------+-------------+---------+---------------------+-------------------------------------+---------+------------------+-------------------+
| Mount ID | Mount Point | Storage | Authentication Type | Configuration                       | Options | Applicable Users | Applicable Groups |
+----------+-------------+---------+---------------------+-------------------------------------+---------+------------------+-------------------+
| 7        | /fotky      | Local   | None                | datadir: "\/syncthing\/fotky\/test" |         | admin            | admini            |
+----------+-------------+---------+---------------------+-------------------------------------+---------+------------------+-------------------+
$ podman exec -u www-data nextcloud /var/www/html/occ files_external:verify 7
  - status: ok
  - code: 0
  - message

At this point all seemed good to me but trying to mount the External storage
tick as admin user I got red exclamation mark and I see the following in the
log:

Error: opendir(/syncthing/fotky/test/): failed to open dir: Permission denied at /var/www/html/lib/private/Files/Storage/Local.php#132

Trying to create file as www-data user works well

$ podman exec -ti -u www-data nextcloud ls -l /syncthing/fotky/test/test.file
ls: cannot access '/syncthing/fotky/test/test.file': No such file or directory

$ podman exec -ti -u www-data nextcloud touch /syncthing/fotky/test/test.file

$ podman exec -ti -u www-data nextcloud ls -l /syncthing/fotky/test/test.file
-rw-rw---- 1 www-data nogroup 0 Mar 21 14:22 /syncthing/fotky/test/test.file

Reiner_Nippes · March 21, 2021, 6:58pm

You posted here credentials of your installation. hope that it’s just a test installation or the credentials are fake.

if you want to use a folder on your host you do that by using -v /path/on/your/host:/path/inside/container as a docker run parameter.

please describe what you mean by mount in the container?

the /path/on/your/host and all files inside must have the same user id than the user running php/webserver inside the container. most likely it’s not the user nextcloud but www-data.

grafik

but beware the the user www-data may have another id on your host and inside the container.

Reiner_Nippes · March 21, 2021, 7:04pm

podman exec -ti -u www-data nextcloud id

would be interesting.

misacek007 · March 22, 2021, 4:58am

You posted here credentials of your installation. hope that it’s just a test installation or the credentials are fake.

Indeed Thank you for pointing that out.

if you want to use a folder on your host you do that by using -v /path/on/your/host:/path/inside/container as a docker run parameter.

please describe what you mean by mount in the container?

By mounting I mean connecting host directory to container using -v. I believe I’m doing it right. For some more context, I’m having a directory on the host that I want to be owned by host user “syncthing” and writable by both user and group but not by others. Connecting this host directory to the rootless container it’s ids got remapped to nouser:nogroup in the container. As php-fpm processes are running inside the container as www-data:www-data and it cannot write to that directory.

My idea of solving this was to make the directory writable for a group and run php-fpm process with that group. From your answer I believe this is not something that is supposed to work and the process group is ignored, right?

misacek007 · March 22, 2021, 5:01am

I thought it might be of interest and it is in the mess of the first entry:
$ podman exec -ti -u www-data nextcloud id
uid=33(www-data) gid=33(www-data) groups=33(www-data),0(root),65534(nogroup)

misacek007 · April 2, 2021, 4:52pm

After digging some more into the issue I’ll probably solve the problem differently.

It seems that sharing data to nextcloud container is best done via NFS volume or directory volume mount from host. In these cases it seems that Nextcloud should be able to detect changes in these directories made outside of Nextcloud instantly when filesystem_check_changes=1 is set .n config.php.

I’ll report back once I got this working (or not).