Snap server stops working after system reboot (can't see page)

snap

#1

Hey guys,

Background
So I installed nextcloud using the snap option (on an openstack cloud computer) and I was immediately impressed with how well everything worked out of the box. It was a bit of a pain trying to change the data directory, but I eventually learned how to do that as well (resigning myself to having to use the /media directory, i.e. https://askubuntu.com/questions/882625/nextcloud-snap-with-data-directory-on-external-harddrive/882696#882696).

Following the instructions linked above, I mounted my external volume in the media directory and everything (creating users, syncing, installing apps, file-sharing etc.) worked. So far so good.

Problem
After a week or so however, I rebooted the server machine and, to my great surprise, I could no longer access the nextcloud server at all. By this, I mean that typing the ip address into firefox or vivaldi results in “This site cannot be reached” or “Unable to connect” respectively. I cannot reach the nextcloud server page, or sync files; I can only ssh into the server machine.

I’ve spent quite a bit of time on this and can’t seem to make any headway. My only progress so far has been that I realised that the mountpoint of my datadirectory changed on restart. I thought this was the problem, but, having fixed this now, I still can’t login.

Various system information is attached below; any comments would be much appreciated.

Thanks very much.

System information
Nextcloud version: 12.0.4
OS: Ubuntu 16.04.3 LTS (xenial)

config.php (removing identifying information):

<?php
$CONFIG = array (
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/snap/nextcloud/current/htdocs/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/snap/nextcloud/current/nextcloud/extra-apps',
      'url' => '/extra-apps',
      'writable' => true,
    ),
  ),
  'supportedDatabases' => 
  array (
    0 => 'mysql',
  ),
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '/tmp/sockets/redis.sock',
    'port' => 0,
  ),

  'trusted_domains' => 
  array (
    0 => '---serveripaddress---',
  ),
  'datadirectory' => '/media/servermount/nextcloud/data',
  'overwrite.cli.url' => '---serveripaddress---',
  'dbtype' => 'mysql',
  'version' => '12.0.4.3',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost:/tmp/sockets/mysql.sock',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',

  'installed' => true,
  'maintenance' => false,
);

The result of sudo snap info nextcloud:

name:      nextcloud
summary:   Nextcloud Server - A safe home for all your data
publisher: nextcloud
contact:   https://github.com/nextcloud/nextcloud-snap
description: |
  Access, share and protect your files, calendars, contacts, communication and
  more at home and in your enterprise.
snap-id: njObIbGQEaVx1H4nyWxchk1i8opy4h54
commands:
  - nextcloud.disable-https
  - nextcloud.enable-https
  - nextcloud.manual-install
  - nextcloud.mysql-client
  - nextcloud.mysqldump
  - nextcloud.occ
services:
  nextcloud.apache:         simple, enabled, inactive
  nextcloud.mdns-publisher: simple, enabled, active
  nextcloud.mysql:          simple, enabled, active
  nextcloud.nextcloud-cron: simple, enabled, active
  nextcloud.php-fpm:        simple, enabled, active
  nextcloud.redis-server:   simple, enabled, active
  nextcloud.renew-certs:    simple, enabled, active
tracking:                   stable
installed:                  12.0.4snap1 (4371) 192MB -
refreshed:                  2017-12-12 23:52:42 +0000 UTC
channels:
  stable:                   12.0.4snap1       (4371) 192MB -
  candidate:                12.0.4snap2-rc1   (4887) 193MB -
  beta:                     12.0.4snap1       (4880) 193MB -
  edge:                     master-2018-01-28 (4937) 193MB -
  11/stable:                11.0.6snap1       (4724) 191MB -
  11/candidate:             11.0.6snap1       (4724) 191MB -
  11/beta:                  11.0.6snap1       (4724) 191MB -
  11/edge:                  11-2018-01-28     (4938) 192MB -
  12/stable:                –
  12/candidate:             –
  12/beta:                  –
  12/edge:                  12-2018-01-28     (4947) 193MB -
  13/stable:                –
  13/candidate:             13.0.0-rc2snap1   (4813) 192MB -
  13/beta:                  ↑
  13/edge:                  ↑

apache-log-error_log:

[Fri Jan 26 12:57:13.254338 2018] [proxy_fcgi:error] [pid 15573:tid 140011512956672] [client 148.6.26.34:60420] AH01067: Failed to read FastCGI header
[Fri Jan 26 12:57:13.254554 2018] [proxy_fcgi:error] [pid 15573:tid 140011512956672] (104)Connection reset by peer: [client 148.6.26.34:60420] AH01075: Error dispatching request to :
[Fri Jan 26 12:57:13.259985 2018] [proxy:error] [pid 15571:tid 140011529742080] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /tmp/sockets/php-fpm.sock (*) failed
[Fri Jan 26 12:57:13.260126 2018] [proxy_fcgi:error] [pid 15571:tid 140011529742080] [client 148.6.26.34:60468] AH01079: failed to make connection to backend: httpd-UDS
[Fri Jan 26 12:57:18.237008 2018] [proxy:error] [pid 15571:tid 140011571705600] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /tmp/sockets/php-fpm.sock (*) failed
[Fri Jan 26 12:57:18.237076 2018] [proxy_fcgi:error] [pid 15571:tid 140011571705600] [client 148.6.26.34:60470] AH01079: failed to make connection to backend: httpd-UDS
[Fri Jan 26 12:57:20.220405 2018] [proxy:error] [pid 15571:tid 140011689203456] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /tmp/sockets/php-fpm.sock (*) failed
[Fri Jan 26 12:57:20.220490 2018] [proxy_fcgi:error] [pid 15571:tid 140011689203456] [client 148.6.26.34:60474] AH01079: failed to make connection to backend: httpd-UDS
[Fri Jan 26 12:57:22.198866 2018] [proxy:error] [pid 15571:tid 140011672418048] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /tmp/sockets/php-fpm.sock (*) failed
[Fri Jan 26 12:57:22.198964 2018] [proxy_fcgi:error] [pid 15571:tid 140011672418048] [client 148.6.26.34:60478] AH01079: failed to make connection to backend: httpd-UDS
[Fri Jan 26 12:57:23.316623 2018] [proxy:error] [pid 15571:tid 140011613669120] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /tmp/sockets/php-fpm.sock (*) failed
[Fri Jan 26 12:57:23.316686 2018] [proxy_fcgi:error] [pid 15571:tid 140011613669120] [client 148.6.26.8:41606] AH01079: failed to make connection to backend: httpd-UDS
[Fri Jan 26 12:57:24.318912 2018] [proxy:error] [pid 15571:tid 140011554920192] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /tmp/sockets/php-fpm.sock (*) failed
[Fri Jan 26 12:57:24.318967 2018] [proxy_fcgi:error] [pid 15571:tid 140011554920192] [client 148.6.26.8:41608] AH01079: failed to make connection to backend: httpd-UDS
[Fri Jan 26 12:57:26.331047 2018] [proxy:error] [pid 15571:tid 140011697596160] (2)No such file or directory: AH02454: FCGI: attempt to connect to Unix domain socket /tmp/sockets/php-fpm.sock (*) failed
[Fri Jan 26 12:57:26.331148 2018] [proxy_fcgi:error] [pid 15571:tid 140011697596160] [client 148.6.26.8:41610] AH01079: failed to make connection to backend: httpd-UDS
[Fri Jan 26 12:57:43.331963 2018] [mpm_event:notice] [pid 15570:tid 140011865941888] AH00491: caught SIGTERM, shutting down

Apache logs php_errors.log

[25-Jan-2018 15:49:50 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/snap/nextcloud/4371/nextcloud/extra-apps/mail/vendor/pear-pear.horde.org/Horde_Imap_Client/Horde/Imap/Client/Socket.php on line 4875
[25-Jan-2018 15:49:50 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/snap/nextcloud/4371/nextcloud/extra-apps/mail/vendor/pear-pear.horde.org/Horde_Imap_Client/Horde/Imap/Client/Socket.php on line 4875
[25-Jan-2018 15:49:56 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/snap/nextcloud/4371/nextcloud/extra-apps/mail/vendor/pear-pear.horde.org/Horde_Stream/Horde/Stream.php on line 402
[25-Jan-2018 15:50:02 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/snap/nextcloud/4371/nextcloud/extra-apps/mail/vendor/pear-pear.horde.org/Horde_Imap_Client/Horde/Imap/Client/Socket.php on line 4875
[25-Jan-2018 15:50:02 UTC] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/snap/nextcloud/4371/nextcloud/extra-apps/mail/vendor/pear-pear.horde.org/Horde_Imap_Client/Horde/Imap/Client/Socket.php on line 4875
[26-Jan-2018 12:58:06 UTC] PHP Fatal error:  Uncaught Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] No such file or directory in /snap/nextcloud/4371/htdocs/lib/private/DB/Connection.php:61
Stack trace:
#0 /snap/nextcloud/4371/htdocs/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /snap/nextcloud/4371/htdocs/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /snap/nextcloud/4371/htdocs/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /snap/nextcloud/4371/htdocs/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /snap/nextcloud/4371/htdocs/lib/private/DB/Connection.php(148): Doctrine\DBAL\Connection->setTransactionIsolation(2)
#5 /snap/nextcloud/4371/htdocs/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DriverManager.php(172): OC\DB\Connection->__con in /snap/nextcloud/4371/htdocs/lib/private/DB/Connection.php on line 61
[26-Jan-2018 12:58:07 UTC] PHP Warning:  fileperms(): stat failed for /media/servermount/nextcloud/data/nextcloud.log in /snap/nextcloud/4371/htdocs/lib/private/Log/File.php on line 133
[26-Jan-2018 12:58:07 UTC] {"reqId":"0XWi2mPmc1D11cIhT1Oo","level":3,"time":"2018-01-26T12:58:07+00:00","remoteAddr":"","user":"--","app":"PHP","method":"--","url":"--","message":"fileperms(): stat failed for \/media\/servermount\/nextcloud\/data\/nextcloud.log at \/snap\/nextcloud\/4371\/htdocs\/lib\/private\/Log\/File.php#133","userAgent":"--","version":"12.0.4.3"}
[26-Jan-2018 12:58:07 UTC] {"reqId":"0XWi2mPmc1D11cIhT1Oo","level":3,"time":"2018-01-26T12:58:06+00:00","remoteAddr":"","user":"--","app":"PHP","method":"--","url":"--","message":"Uncaught Doctrine\\DBAL\\DBALException: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] No such file or directory in \/snap\/nextcloud\/4371\/htdocs\/lib\/private\/DB\/Connection.php:61\nStack trace:\n#0 \/snap\/nextcloud\/4371\/htdocs\/3rdparty\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Connection.php(429): OC\\DB\\Connection->connect()\n#1 \/snap\/nextcloud\/4371\/htdocs\/3rdparty\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Connection.php(389): Doctrine\\DBAL\\Connection->getDatabasePlatformVersion()\n#2 \/snap\/nextcloud\/4371\/htdocs\/3rdparty\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Connection.php(328): Doctrine\\DBAL\\Connection->detectDatabasePlatform()\n#3 \/snap\/nextcloud\/4371\/htdocs\/3rdparty\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Connection.php(623): Doctrine\\DBAL\\Connection->getDatabasePlatform()\n#4 \/snap\/nextcloud\/4371\/htdocs\/lib\/private\/DB\/Connection.php(148): Doctrine\\DBAL\\Connection->setTransactionIsolation(2)\n#5 \/snap\/nextcloud\/4371\/htdocs\/3rdparty\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/DriverManager.php(172): OC\\DB\\Connection->__con at \/snap\/nextcloud\/4371\/htdocs\/lib\/private\/DB\/Connection.php#61","userAgent":"--","version":"12.0.4.3"}
[26-Jan-2018 13:44:34 UTC] PHP Fatal error:  Uncaught Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] No such file or directory in /snap/nextcloud/4371/htdocs/lib/private/DB/Connection.php:61
"apache-logs-php_errors.log" 266L, 42541C   

The result of running ps aux | grep snap:

root      1157  0.1  0.1 728344 34184 ?        Ssl  15:03   0:04 /usr/lib/snapd/snapd
root      1161  0.0  0.0   4504  1692 ?        Ss   15:03   0:00 /bin/sh /snap/nextcloud/4371/bin/renew-certs
root      1167  0.0  0.0   4504  1732 ?        Ss   15:03   0:00 /bin/sh /snap/nextcloud/4371/bin/start_mysql
root      1198  0.0  0.0   4504   704 ?        Ss   15:03   0:00 /bin/sh /snap/nextcloud/4371/bin/delay-on-failure mdns-publisher nextcloud
root      1215  0.0  0.0   4504   784 ?        Ss   15:03   0:00 /bin/sh /snap/nextcloud/4371/bin/start-redis-server
root      1254  0.0  0.0   4504  1600 ?        Ss   15:03   0:00 /bin/sh /snap/nextcloud/4371/bin/start-php-fpm
root      1256  0.0  0.0   4504  1752 ?        Ss   15:03   0:00 /bin/sh /snap/nextcloud/4371/bin/nextcloud-cron
root      2031  0.0  0.0   4504  1784 ?        S    15:03   0:00 /bin/sh /snap/nextcloud/4371/bin/mysqld_safe --datadir=/var/snap/nextcloud/4371/mysql --pid-file=/tmp/pids/mysql.pid --lc-messages-dir=/snap/nextcloud/4371/share --socket=/tmp/sockets/mysql.sock
root      2204  0.4  0.5 1336924 167948 ?      Sl   15:03   0:11 /snap/nextcloud/4371/bin/mysqld --basedir=/snap/nextcloud/4371 --datadir=/var/snap/nextcloud/4371/mysql --plugin-dir=/snap/nextcloud/4371/lib/plugin --lc-messages-dir=/snap/nextcloud/4371/share --log-error=/var/snap/nextcloud/4371/mysql/wal-server.err --pid-file=/tmp/pids/mysql.pid --socket=/tmp/sockets/mysql.sock
root      2263  0.0  0.1 314440 39088 ?        Ss   15:03   0:00 php-fpm: master process (/snap/nextcloud/4371/config/php/php-fpm.conf)
root      9382  0.0  0.0   4504   796 ?        Ss   15:51   0:00 /bin/sh /snap/nextcloud/4371/bin/run-httpd -k stop
root      9399  0.0  0.0   4504  1700 ?        S    15:51   0:00 /bin/sh /snap/nextcloud/4371/bin/setup_nextcloud
root      9418  0.0  0.0   4504  1676 ?        S    15:51   0:00 /bin/sh /snap/nextcloud/4371/bin/occ config:system:set redis host --value=/tmp/sockets/redis.sock --type=string
root      9432  0.0  0.1 303644 39904 ?        R    15:51   0:00 php -c /snap/nextcloud/4371/config/php /snap/nextcloud/4371/htdocs/occ config:system:set redis host --value=/tmp/sockets/redis.sock --type=string
ubuntu    9435  0.0  0.0  12944  1088 pts/1    S+   15:51   0:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn snap

#2

Same problem after reboot for me with NC 13 stable & NC 14 rc1 via snap install on ubuntu bionic server. snap.nextcloud.apache / mysql services not running even after restart. Site not reachable.


#3

Just for future reference, this was a ports problem for me. What happens is that on reboot another process grabs the ports before nextcloud can. If you kill that process and restart nextcloud it should work.

I wrote this up on a github page, but I can’t find it at the moment and I no-longer use a snap installation.