MySQL server has gone away on Nextcloud Box

ℹ️ Support
1

Hi, I have a Nextcloud Box up and running since two weeks and it worked fine so far. Since today I can’t upload larger files any more. I get an error message that my SQL connection went away:

Fehler beim Hochladen der Datei "StMartin.mp4": An exception occurred while executing 'SELECTfileid,storage,path,parent,name,mimetype,mimepart,size,mtime,storage_mtime,encrypted,etag,permissions,checksumFROMoc_filecacheWHEREstorage= ? ANDpath_hash= ?' with params ["3", "870dc9bfc71ef14d08c125b4d2388ca0"]: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away

First I tried it with Nextcloud Client, then directly through the browser, but same result. This file is 800 MB, a smaller file (50 MB) worked fine.

Regards,
Sascha

2

Hi Sascha,

I’m sorry to hear about your experience with the Nextcloud Box. What happens, if you try to run

sudo nextcloud.mysql-client

over SSH or directly on the box? Have you already tried to restart the box?

3

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/snap/nextcloud/48/mysql/mysql.sock' (2)

Let me restart it and see what happens …

Ok, same error after restart :frowning:

4

I see this again and again in syslog:

Nov  5 14:10:22 ubuntu-standard systemd[1]: snap.nextcloud.mysql.service: State 'stop-sigterm' timed out. Killing.
Nov  5 14:10:22 ubuntu-standard rsyslogd-2007: action 'action 9' suspended, next retry is Sat Nov  5 14:10:52 2016 [v8.16.0 try http://www.rsyslog.com/e/2007 ]
Nov  5 14:10:22 ubuntu-standard systemd[1]: snap.nextcloud.mysql.service: Unit entered failed state.
Nov  5 14:10:22 ubuntu-standard systemd[1]: snap.nextcloud.mysql.service: Failed with result 'timeout'.
Nov  5 14:10:22 ubuntu-standard systemd[1]: snap.nextcloud.mysql.service: Service hold-off time over, scheduling restart.
Nov  5 14:10:22 ubuntu-standard systemd[1]: Stopped Service for snap application nextcloud.mysql.
Nov  5 14:10:22 ubuntu-standard systemd[1]: Started Service for snap application nextcloud.mysql.
Nov  5 14:10:22 ubuntu-standard /usr/lib/snapd/snapd[1006]: daemon.go:174: DEBUG: uid=0;@ GET /v2/snaps 19.160337ms 200
Nov  5 14:10:22 ubuntu-standard kernel: [  163.039030] audit_printk_skb: 367 callbacks suppressed
Nov  5 14:10:22 ubuntu-standard kernel: [  163.039062] audit: type=1400 audit(1478355022.506:651): apparmor="DENIED" operation="open" profile="/usr/lib/snapd/snap-confine" name="/etc/ld.so.preload" pid=2495 comm="ubuntu-core-lau" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov  5 14:10:22 ubuntu-standard kernel: [  163.064423] audit: type=1400 audit(1478355022.530:652): apparmor="DENIED" operation="open" profile="snap.nextcloud.mysql" name="/etc/ld.so.preload" pid=2495 comm="snap-exec" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov  5 14:10:22 ubuntu-standard kernel: [  163.092356] audit: type=1400 audit(1478355022.558:653): apparmor="DENIED" operation="open" profile="snap.nextcloud.mysql" name="/etc/ld.so.preload" pid=2495 comm="command-mysql.w" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov  5 14:10:22 ubuntu-standard kernel: [  163.098423] audit: type=1400 audit(1478355022.566:654): apparmor="DENIED" operation="open" profile="snap.nextcloud.mysql" name="/etc/ld.so.preload" pid=2495 comm="start_mysql" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov  5 14:10:22 ubuntu-standard kernel: [  163.105000] audit: type=1400 audit(1478355022.570:655): apparmor="DENIED" operation="open" profile="snap.nextcloud.mysql" name="/etc/ld.so.preload" pid=2508 comm="hostname" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov  5 14:10:22 ubuntu-standard kernel: [  163.112015] audit: type=1400 audit(1478355022.578:656): apparmor="DENIED" operation="open" profile="snap.nextcloud.mysql" name="/etc/ld.so.preload" pid=2509 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov  5 14:10:22 ubuntu-standard snap[2495]: 2016-11-05T14:10:22.606191Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
Nov  5 14:10:22 ubuntu-standard snap[2495]: 2016-11-05T14:10:22.606717Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
Nov  5 14:10:22 ubuntu-standard snap[2495]: 2016-11-05T14:10:22.607817Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
Nov  5 14:10:22 ubuntu-standard snap[2495]: 2016-11-05T14:10:22.614899Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
Nov  5 14:10:22 ubuntu-standard snap[2495]: 2016-11-05T14:10:22.615123Z 0 [ERROR] Aborting
Nov  5 14:10:22 ubuntu-standard kernel: [  163.151059] audit: type=1400 audit(1478355022.618:657): apparmor="DENIED" operation="open" profile="snap.nextcloud.mysql" name="/etc/ld.so.preload" pid=2510 comm="mysql.server" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov  5 14:10:22 ubuntu-standard kernel: [  163.160409] audit: type=1400 audit(1478355022.626:658): apparmor="DENIED" operation="exec" profile="snap.nextcloud.mysql" name="/bin/run-parts" pid=2511 comm="mysql.server" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov  5 14:10:22 ubuntu-standard kernel: [  163.169616] audit: type=1400 audit(1478355022.634:659): apparmor="DENIED" operation="open" profile="snap.nextcloud.mysql" name="/etc/ld.so.preload" pid=2514 comm="my_print_defaul" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov  5 14:10:22 ubuntu-standard kernel: [  163.184228] audit: type=1400 audit(1478355022.650:660): apparmor="DENIED" operation="open" profile="snap.nextcloud.mysql" name="/etc/ld.so.preload" pid=2515 comm="hostname" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov  5 14:10:22 ubuntu-standard snap[2495]: Starting MySQL
Nov  5 14:10:22 ubuntu-standard snap[2495]: .
Nov  5 14:10:22 ubuntu-standard snap[2495]: Waiting for server...
Nov  5 14:10:22 ubuntu-standard snap[2495]: Checking/upgrading mysql tables if necessary...
Nov  5 14:10:22 ubuntu-standard snap[2495]: mysql_upgrade: Got error: 2002: Can't connect to local MySQL server through socket '/var/snap/nextcloud/48/mysql/mysql.sock' (2) while connecting to the MySQL server
Nov  5 14:10:22 ubuntu-standard snap[2495]: Upgrade process encountered error and will not continue.
Nov  5 14:10:22 ubuntu-standard /usr/lib/snapd/snapd[1006]: daemon.go:174: DEBUG: uid=0;@ GET /v2/snaps 11.561409ms 200
Nov  5 14:10:22 ubuntu-standard snap[2530]: MySQL server process #2461 is not running! ... failed!

Every 30 seconds …

5

May I ask you for the Version of Nextcloud (Snap) running on your box? Have you done any configuration changes in the past? :thinking:

6

Hi Marius, thanks for your help. I’m running 10.0.1 (stable). The last thing I did was a apt-get upgrade last night and a reboot after that. But I think it was still running earlier this morning.

If there are any log files I can provide just let me know.

Regards,
Sascha

7

Just found an issue about that on GitHub:

Could you provide us the output of $ journalctl -u snap.nextcloud.mysql.service :slight_smile: THX!

8

It tries to start MySQL, waits for some time and then fails. Again and again.

Nov 05 17:30:19 ubuntu-standard snap[2580]: Starting MySQL
Nov 05 17:30:19 ubuntu-standard snap[2580]: .
Nov 05 17:30:19 ubuntu-standard snap[2580]: Waiting for server...
Nov 05 17:30:19 ubuntu-standard snap[2580]: Checking/upgrading mysql tables if necessary...
Nov 05 17:30:19 ubuntu-standard snap[2580]: mysql_upgrade: Got error: 2002: Can't connect to local MySQL server through socket '/var/snap/nextcloud/48/mysql/mysql.sock' (2) while connecting to the MySQL server
Nov 05 17:30:19 ubuntu-standard snap[2580]: Upgrade process encountered error and will not continue.
Nov 05 17:30:19 ubuntu-standard snap[2615]: MySQL server process #2494 is not running! ... failed!
Nov 05 17:30:50 ubuntu-standard systemd[1]: snap.nextcloud.mysql.service: State 'stop-sigterm' timed out. Killing.
Nov 05 17:30:50 ubuntu-standard systemd[1]: snap.nextcloud.mysql.service: Unit entered failed state.
Nov 05 17:30:50 ubuntu-standard systemd[1]: snap.nextcloud.mysql.service: Failed with result 'timeout'.
Nov 05 17:30:50 ubuntu-standard systemd[1]: snap.nextcloud.mysql.service: Service hold-off time over, scheduling restart.
Nov 05 17:30:50 ubuntu-standard systemd[1]: Stopped Service for snap application nextcloud.mysql.
Nov 05 17:30:50 ubuntu-standard systemd[1]: Started Service for snap application nextcloud.mysql.
Nov 05 17:30:50 ubuntu-standard snap[2829]: 2016-11-05T17:30:50.568750Z 0 [Warning] Changed limits: max_open_files: 1024 (requested 5000)
Nov 05 17:30:50 ubuntu-standard snap[2829]: 2016-11-05T17:30:50.569486Z 0 [Warning] Changed limits: table_open_cache: 431 (requested 2000)
Nov 05 17:30:50 ubuntu-standard snap[2829]: 2016-11-05T17:30:50.570946Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
Nov 05 17:30:50 ubuntu-standard snap[2829]: 2016-11-05T17:30:50.580355Z 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
Nov 05 17:30:50 ubuntu-standard snap[2829]: 2016-11-05T17:30:50.580615Z 0 [ERROR] Aborting
Nov 05 17:30:50 ubuntu-standard snap[2829]: Starting MySQL
Nov 05 17:30:50 ubuntu-standard snap[2829]: .
Nov 05 17:30:50 ubuntu-standard snap[2829]: Waiting for server...
Nov 05 17:30:50 ubuntu-standard snap[2829]: Checking/upgrading mysql tables if necessary...
Nov 05 17:30:50 ubuntu-standard snap[2829]: mysql_upgrade: Got error: 2002: Can't connect to local MySQL server through socket '/var/snap/nextcloud/48/mysql/mysql.sock' (2) while connecting to the MySQL server
Nov 05 17:30:50 ubuntu-standard snap[2829]: Upgrade process encountered error and will not continue.
Nov 05 17:30:50 ubuntu-standard snap[2864]: MySQL server process #2794 is not running! ... failed!

I can provide the whole output as file if you wish. So far we only have 800 files (maybe 2 GB in total and less than 10 users on the Nextcloud Box.

Regards,
Sascha

9

Thanks :thumbsup: I have posted this in the Issue :wink: Let’s see what the snap guys say :sweat_smile: … I hope we will find a solution for your problem soon :smiley_cat:

cc @kyrofa @oparoz

10

Ok, hopefully we can get it back up and running.

This looks a bit weird to me:
sudo nextcloud.mysql-client
ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/var/snap/nextcloud/48/mysql/mysql.sock’ (2)

If I look at /var/snap/nextcloud I see a couple of directories, and their is a more recent one:

ubuntu@ubuntu-standard:/var/snap/nextcloud$ ls -rtl
total 16
drwxr-xr-x 3 root root 4096 Aug 24 04:05 common
drwxr-xr-x 8 root root 4096 Aug 24 04:05 48
drwxr-xr-x 8 root root 4096 Aug 24 04:05 136
drwxr-xr-x 8 root root 4096 Aug 24 04:05 128
lrwxrwxrwx 1 root root    3 Oct 17 17:10 current -> 136

Maybe there is something else broken?