Nextcloud/NFS results in hang; normal mount works fine

ℹ️ Support
1

Nextcloud version : 20.0.1 (Docker)
Operating system and version: Ubuntu 20.04
Apache or nginx version: n/a
PHP version: whatever is included in the nextcloud image

The issue you are facing:
I’m trying to move my Nextcloud 20.0.1 instance to use an NFS mount instead of a filesystem mount. (Note: I’m running this on Kubernetes using the nextcloud Docker image).

When I change the /var/www/html mount to point to the NFS share (I’ve already rsynced the data from one to the other) and update the deployment, I can see it mounts the NFS volume correctly, and I can cat the files in there so I know the filesystem has mounted and is readable. Permissions are correct also.

However, whenever I try to access the server through my web browser, it hangs, and eventually times out. Also, if I try to execute e.g., ./occ status it also hangs. I’m assuming from this that there’s something different in the NFS semantics, possibly having to do with how PHP locks files?

The mount itself looks like this:

10.1.2.3:/export/[redacted] on /var/www/html type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.1.2.3,mountvers=3,mountport=20048,mountproto=udp,local_lock=none,addr=10.1.2.3)

I’ve tried moving nextcloud.log to /tmp to see if that would fix it; no joy.

I should mention that I have other Apache/PHP applications (Rainloop) that I have successfully migrated to NFS without issue.

Is there anything wrong with that mount that would cause these symptoms? I can’t seem to find an answer.

Is this the first time you’ve seen this error? Y

Steps to replicate it:

  1. Mount /var/www/html from an NFS volume
  2. Do anything involving PHP (serve a web page, run occ)
  3. Watch it hang.

The output of your Nextcloud log in Admin > Logging:

None

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):


<?php
$CONFIG = array (
  'instanceid' => 'nope',
  'passwordsalt' => 'nope',
  'trusted_domains' =>
  array (
    0 => 'nope',
    1 => 'nope',
  ),
  'datadirectory' => '/var/www/html/data',
  'objectstore' =>
  array (
    'class' => 'OC\\Files\\ObjectStore\\S3',
    'arguments' =>
    array (
      'bucket' => 'nope',
      'autocreate' => true,
      'key' => 'nope',
      'secret' => 'nope',
      'region' => 'nope',
      'hostname' => 'nope',
      'port' => 9000,
      'use_ssl' => false,
      'use_path_style' => true,
    ),
  ),
  'version' => '20.0.1.1',
  'dbtype' => 'pgsql',
  'dbhost' => 'nope',
  'dbname' => 'nope',
  'dbuser' => 'nope',
  'dbpassword' => 'nope',
  'dbtableprefix' => 'oc_',
  'installed' => true,
  'default_language' => 'en',
  'defaultapp' => 'files',
  'knowledgebaseenabled' => true,
  'enable_avatars' => true,
  'allow_user_to_change_display_name' => true,
  'remember_login_cookie_lifetime' => 1296000,
  'session_lifetime' => 86400,
  'session_keepalive' => true,
  'skeletondirectory' => '',
  'lost_password_link' => '',
  'mail_domain' => 'nope',
  'mail_from_address' => 'nope',
  'mail_smtpdebug' => false,
  'mail_smtpmode' => 'smtp',
  'mail_smtphost' => 'nope',
  'mail_smtpport' => '587',
  'mail_smtptimeout' => 10,
  'mail_smtpauth' => 1,
  'mail_smtpauthtype' => 'PLAIN',
  'mail_smtpname' => 'nope',
  'mail_smtppassword' => 'nope',
  'overwritehost' => 'cloud.koehn.com',
  'overwriteprotocol' => 'https',
  'overwritewebroot' => '/',
  'overwritecondaddr' => '',
  'overwrite.cli.url' => 'https://cloud.koehn.com',
  'htaccess.RewriteBase' => '/',
  'trashbin_retention_obligation' => 'auto',
  'versions_retention_obligation' => 'auto',
  'appcodechecker' => false,
  'updatechecker' => true,
  'updater.server.url' => 'https://updates.nextcloud.com/updater_server/',
  'has_internet_connection' => true,
  'check_for_working_webdav' => true,
  'check_for_working_wellknown_setup' => true,
  'check_for_working_htaccess' => true,
  'config_is_read_only' => false,
  'log_type' => 'owncloud',
  'logfile' => '/var/www/html/data/nextcloud.log',
  'loglevel' => 2,
  'syslog_tag' => 'Nextcloud',
  'log.condition' =>
  array (
    'shared_secret' => 'nope',
    'users' =>
    array (
      0 => 'sample-user',
    ),
    'apps' =>
    array (
      0 => 'files',
    ),
  ),
  'logdateformat' => 'F d, Y H:i:s',
  'logtimezone' => 'Europe/Berlin',
  'log_query' => false,
  'cron_log' => true,
  'log_rotate_size' => 10485760,
  'customclient_desktop' => 'https://nextcloud.com/install/',
  'customclient_android' => 'https://play.google.com/store/apps/details?id=com.nextcloud.client',
  'customclient_ios' => 'https://itunes.apple.com/us/app/nextcloud/id1125420102?mt=8',
  'appstoreenabled' => true,
  'appstore.experimental.enabled' => true,
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'enable_previews' => true,
  'preview_max_x' => 2048,
  'preview_max_y' => 2048,
  'preview_max_scale_factor' => 10,
  'preview_max_filesize_image' => 50,
  'preview_libreoffice_path' => '/usr/bin/libreoffice',
  'preview_office_cl_parameters' => ' --headless --nologo --nofirststartwizard --invisible --norestore --convert-to pdf --outdir ',
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\PNG',
    1 => 'OC\\Preview\\JPEG',
    2 => 'OC\\Preview\\GIF',
    3 => 'OC\\Preview\\BMP',
    4 => 'OC\\Preview\\XBitmap',
    5 => 'OC\\Preview\\MP3',
    6 => 'OC\\Preview\\TXT',
    7 => 'OC\\Preview\\MarkDown',
    8 => 'OC\\Preview\\PDF',
  ),
  'ldapUserCleanupInterval' => 51,
  'comments.managerFactory' => '\\OC\\Comments\\ManagerFactory',
  'systemtags.managerFactory' => '\\OC\\SystemTag\\ManagerFactory',
  'maintenance' => false,
  'singleuser' => false,
  'enable_certificate_management' => false,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'redis',
    'port' => 6379,
    'timeout' => 0.0,
  ),
  'memcached_servers' =>
  array (
    0 =>
    array (
      0 => 'memcached',
      1 => 11211,
    ),
  ),
  'memcached_options' =>
  array (
    14 => 50,
    15 => 50,
    19 => 50,
    20 => 50,
    8 => 50,
    -1001 => true,
    16 => true,
    18 => true,
  ),
  'cache_path' => '',
  'cache_chunk_gc_ttl' => 86400,
  'sharing.managerFactory' => '\\OC\\Share20\\ProviderFactory',
  'sqlite.journal_mode' => 'DELETE',
  'supportedDatabases' =>
  array (
    0 => 'sqlite',
    1 => 'mysql',
    2 => 'pgsql',
    3 => 'oci',
  ),
  'tempdirectory' => '/tmp',
  'hashingCost' => 10,
  'blacklisted_files' =>
  array (
    0 => '.htaccess',
  ),
  'share_folder' => '/',
  'theme' => '',
  'cipher' => 'AES-256-CFB',
  'minimum.supported.desktop.version' => '1.7.0',
  'quota_include_external_storage' => false,
  'filesystem_check_changes' => 0,
  'part_file_in_storage' => true,
  'mount_file' => '/var/www/nextcloud/data/mount.json',
  'filesystem_cache_readonly' => false,
  'secret' => '',
  'trusted_proxies' =>
  array (
    0 => 'nope',
    1 => 'nope',
  ),
  'forwarded_for_headers' =>
  array (
    0 => 'HTTP_X_FORWARDED',
    1 => 'HTTP_FORWARDED_FOR',
  ),
  'max_filesize_animated_gifs_public_sharing' => 10,
  'filelocking.enabled' => true,
  'filelocking.ttl' => 3600,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'upgrade.disable-web' => false,
  'debug' => false,
  'data-fingerprint' => 'nope',
  'updater.release.channel' => 'stable',
  'mail_smtpsecure' => 'tls',
  'mail_sendmailmode' => 'smtp',
  'app_install_overwrite' =>
  array (
    0 => 'apporder',
    1 => 'calendar',
    2 => 'drawio',
  ),
);

The output of your Apache/nginx/system log in /var/log/____:

[Fri Oct 30 15:13:19.157849 2020] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.4.11 configured -- resuming normal operations
[Fri Oct 30 15:13:19.157894 2020] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
10.244.4.182 - - [30/Oct/2020:15:13:19 +0000] "GET / HTTP/1.1" 200 622 "-" "curl/7.64.1"
10.244.4.182 - x [30/Oct/2020:15:13:19 +0000] "PROPFIND /remote.php/dav/files/bkoehn/ HTTP/1.1" 200 646 "-" "Mozilla/5.0 (Macintosh) mirall/3.0.1git (build 3355) (Nextcloud)"
10.244.4.182 - y [30/Oct/2020:15:13:19 +0000] "GET /status.php HTTP/1.1" 500 639 "-" "Mozilla/5.0 (Macintosh) mirall/2.6.5stable (build 20200710) (Nextcloud)"
10.244.4.182 - - [30/Oct/2020:15:13:19 +0000] "GET / HTTP/1.1" 200 608 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:82.0) Gecko/20100101 Firefox/82.0"
10.244.4.182 - y [30/Oct/2020:15:13:38 +0000] "GET /ocs/v2.php/core/navigation/apps?absolute=true&format=json HTTP/1.1" 200 763 "-" "Mozilla/5.0 (Macintosh) mirall/2.6.5stable (build 20200710) (Nextcloud)"
10.244.4.182 - y [30/Oct/2020:15:13:41 +0000] "GET /status.php HTTP/1.1" 500 639 "-" "Mozilla/5.0 (Macintosh) mirall/2.6.5stable (build 20200710) (Nextcloud)"
127.0.0.1 - - [30/Oct/2020:15:14:42 +0000] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.38 (Debian) PHP/7.4.11 (internal dummy connection)"
10.244.4.182 - - [30/Oct/2020:15:13:46 +0000] "GET / HTTP/1.1" 200 608 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:82.0) Gecko/20100101 Firefox/82.0"
10.244.4.182 - x [30/Oct/2020:15:13:51 +0000] "PROPFIND /remote.php/dav/files/bkoehn/ HTTP/1.1" 200 646 "-" "Mozilla/5.0 (Macintosh) mirall/3.0.1git (build 3355) (Nextcloud)"
10.244.4.182 - - [30/Oct/2020:15:13:51 +0000] "GET / HTTP/1.1" 200 608 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:82.0) Gecko/20100101 Firefox/82.0"
10.244.4.182 - y [30/Oct/2020:15:14:10 +0000] "GET /ocs/v2.php/core/navigation/apps?absolute=true&format=json HTTP/1.1" 200 763 "-" "Mozilla/5.0 (Macintosh) mirall/2.6.5stable (build 20200710) (Nextcloud)"
10.244.4.182 - y [30/Oct/2020:15:14:13 +0000] "GET /status.php HTTP/1.1" 500 639 "-" "Mozilla/5.0 (Macintosh) mirall/2.6.5stable (build 20200710) (Nextcloud)"
10.244.4.182 - y [30/Oct/2020:15:14:42 +0000] "GET /ocs/v2.php/core/navigation/apps?absolute=true&format=json HTTP/1.1" 200 763 "-" "Mozilla/5.0 (Macintosh) mirall/2.6.5stable (build 20200710) (Nextcloud)"
10.244.4.182 - y [30/Oct/2020:15:14:45 +0000] "GET /status.php HTTP/1.1" 500 639 "-" "Mozilla/5.0 (Macintosh) mirall/2.6.5stable (build 20200710) (Nextcloud)"
10.244.4.182 - - [30/Oct/2020:15:14:51 +0000] "GET / HTTP/1.1" 200 608 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.16; rv:82.0) Gecko/20100101 Firefox/82.0"
10.244.4.182 - x [30/Oct/2020:15:14:53 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications?format=json HTTP/1.1" 200 763 "-" "Mozilla/5.0 (Macintosh) mirall/3.0.1git (build 3355) (Nextcloud)"
2

OK, it looks like this is definitely an NFS problem. The issue is that /lib/private/Config.php:212 attempts to grab a shared lock on the /conf/config.php file, and that’s where it’s getting hung up. It seems that this is a known issue with NFS, and Nextcloud’s use of filesystem locks won’t ever be supported, which is kind of sad. I’d like to see Nextcloud use a locking mechanism compatible with NFS (e.g., .lock files).

I’d love to be able to run my server under NFS, or at least with some kind of shared filesystem, but that’s not going to happen any time soon.