Internal Server Error by uploading files over 500mb

ℹ️ Support
1

Nextcloud version 11.0.3
Operating system and version Ubuntu 16.04 LTS
Apache 2.4.18 and nginx 1.10.0
PHP version 7.0
Is this the first time you’ve seen this error?: YES

Can you reliably replicate it? (If so, please outline steps):

The issue you are facing:
I cant upload files which are bigger than 550mb

The output of your Nextcloud log in Admin > Logging:
Fatal webdav Doctrine\DBAL\Exception\DriverException: An exception occurred while executing ‘UPDATE oc_file_locks SET lock = -1, ttl = ? WHERE key = ? AND lock = 1’ with params [1494606071, “files/44af3aa4f63dc8cb2b87b85a7e98be93”]: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away

/var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/DBALException.php - line 128: Doctrine\DBAL\Driver\AbstractMySQLDriver->convertException('An exception oc...', Object(Doctrine\DBAL\Driver\PDOException))
/var/www/html/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php - line 996: Doctrine\DBAL\DBALException driverExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDOMySql\Driver), Object(Doctrine\DBAL\Driver\PDOException), 'UPDATE `oc_file...', Array)
/var/www/html/lib/private/DB/Connection.php - line 215: Doctrine\DBAL\Connection->executeUpdate('UPDATE `oc_file...', Array, Array)
/var/www/html/lib/private/Lock/DBLockingProvider.php - line 229: OC\DB\Connection->executeUpdate('UPDATE `oc_file...', Array)
/var/www/html/lib/private/Files/Storage/Common.php - line 698: OC\Lock\DBLockingProvider->changeLock('files/44af3aa4f...', 2)
/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php - line 610: OC\Files\Storage\Common->changeLock('files/Betriebss...', 2, Object(OC\Lock\DBLockingProvider))
/var/www/html/lib/private/Files/Storage/Wrapper/Wrapper.php - line 610: OC\Files\Storage\Wrapper\Wrapper->changeLock('files/Betriebss...', 2, Object(OC\Lock\DBLockingProvider))
/var/www/html/lib/private/Files/View.php - line 1935: OC\Files\Storage\Wrapper\Wrapper->changeLock('files/Betriebss...', 2, Object(OC\Lock\DBLockingProvider))
/var/www/html/apps/dav/lib/Connector/Sabre/Node.php - line 349: OC\Files\View->changeLock('/Betriebssystem...', 2)
/var/www/html/apps/dav/lib/Connector/Sabre/File.php - line 170: OCA\DAV\Connector\Sabre\Node->changeLock(2)
/var/www/html/apps/dav/lib/Connector/Sabre/Directory.php - line 137: OCA\DAV\Connector\Sabre\File->put(Resource id #86)
/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php - line 1072: OCA\DAV\Connector\Sabre\Directory->createFile('antergos-2015.0...', Resource id #86)
/var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php - line 525: Sabre\DAV\Server->createFile('Betriebssysteme...', Resource id #86, NULL)
[internal function] Sabre\DAV\CorePlugin->httpPut(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
/var/www/html/3rdparty/sabre/event/lib/EventEmitterTrait.php - line 105: call_user_func_array(Array, Array)
/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php - line 479: Sabre\Event\EventEmitter->emit('method PUT', Array)
/var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php - line 254: Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
/var/www/html/apps/dav/appinfo/v1/webdav.php - line 60: Sabre\DAV\Server->exec()
/var/www/html/remote.php - line 165: require_once('/var/www/html/a...')
{main}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php $CONFIG = array ( 'instanceid' => 'ocqacny65bj9', 'passwordsalt' => 'xxxxxxxxxxxxxxxxxx', 'secret' => 'xxxxxxxx', 'trusted_domains' => array ( 0 => 'xxxx.xxxx.xxxx', 1 => 'xxxx.xx', ), 'enable_avatars' => true, 'auth.bruteforce.protection.enabled' => true, 'updater.server.url' => 'https://updates.nextcloud.com/updater_server/', 'datadirectory' => '/var/www/html/data', 'dbtype' => 'mysql', 'version' => '11.0.3.2', 'dbname' => 'xxxxxxx', 'dbhost' => 'xxxxx', 'dbport' => '', 'dbtableprefix' => 'oc_', 'dbuser' => 'xxxxxx', 'dbpassword' => 'xxxxxxx', 'logtimezone' => 'UTC', 'installed' => true, 'forcessl' => true, 'forceSSLforSubdomains' => true, 'htaccess.RewriteBase' => '/', 'forwarded_for_headers' => array ( 0 => 'HTTP_X_FORWARDED', 1 => 'HTTP_FORWARDED_FOR', ), 'trusted_proxies' => array ( 0 => 'x.x.x.x.x', 1 => '127.0.0.1', ), 'overwrite.cli.url' => 'https://xxxx.xx/', 'overwriteprotocol' => 'https', 'updater.secret' => 'xxxxxxxxxxxxxxxxxxxxxx', 'maintenance' => false, 'theme' => '', 'loglevel' => 2, ); The output of your Apache/nginx/system log in `/var/log/____`: Apache2: [Fri May 12 17:21:46.630536 2017] [authz_core:error] [pid 3426] [client xxxxxx:61677] AH01630: client denied by server configuration: /var/www/html2/data/.ocdata [Fri May 12 17:21:53.301953 2017] [authz_core:error] [pid 3417] [client xxxxxx:61690] AH01630: client denied by server configuration: /var/www/html2/data/.ocdata [Fri May 12 17:22:16.217974 2017] [authz_core:error] [pid 3419] [client xxxxxx:42224] AH01630: client denied by server configuration: /var/www/html/data/.ocdata [Fri May 12 17:22:46.394908 2017] [:error] [pid 3426] [client xxxxx:61691] PHP Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in /var/www/html2/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:91\nStack trace:\n#0 /var/www/html2/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(91): PDOStatement->execute(NULL)\n#1 /var/www/html2/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(987): Doctrine\\DBAL\\Driver\\PDOStatement->execute()\n#2 /var/www/html2/lib/private/DB/Connection.php(215): Doctrine\\DBAL\\Connection->executeUpdate('UPDATE `oc_file...', Array, Array)\n#3 /var/www/html2/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php(208): OC\\DB\\Connection->executeUpdate('UPDATE `oc_file...', Array, Array)\n#4 /var/www/html2/lib/private/DB/QueryBuilder/QueryBuilder.php(177): Doctrine\\DBAL\\Query\\QueryBuilder->execute()\n#5 /var/www/html2/lib/private/Lock/DBLockingProvider.php(278): OC\\DB\\QueryBuilder\\QueryBuilder->execute()\n#6 [internal function]: OC\\Lock\\DBLockingProvider->releaseAll()\n#7 {main}\n\nNext Doctrine\\DBA in /var/www/html2/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php on line 116 [Fri May 12 17:25:23.751118 2017] [:error] [pid 3477] [client xxxxxxx:61708] PHP Fatal error: Uncaught PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away in /var/www/html2/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php:91\nStack trace:\n#0 /var/www/html2/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php(91): PDOStatement->execute(NULL)\n#1 /var/www/html2/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(987): Doctrine\\DBAL\\Driver\\PDOStatement->execute()\n#2 /var/www/html2/lib/private/DB/Connection.php(215): Doctrine\\DBAL\\Connection->executeUpdate('UPDATE `oc_file...', Array, Array)\n#3 /var/www/html2/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php(208): OC\\DB\\Connection->executeUpdate('UPDATE `oc_file...', Array, Array)\n#4 /var/www/html2/lib/private/DB/QueryBuilder/QueryBuilder.php(177): Doctrine\\DBAL\\Query\\QueryBuilder->execute()\n#5 /var/www/html2/lib/private/Lock/DBLockingProvider.php(278): OC\\DB\\QueryBuilder\\QueryBuilder->execute()\n#6 [internal function]: OC\\Lock\\DBLockingProvider->releaseAll()\n#7 {main}\n\nNext Doctrine\\DBA in /var/www/html2/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Driver/AbstractMySQLDriver.php on line 116 [Fri May 12 17:27:08.844835 2017] [authz_core:error] [pid 3438] [client xxxxxx:42328] AH01630: client denied by server configuration: /var/www/html/data/.ocdata [Fri May 12 17:29:20.279489 2017] [authz_core:error] [pid 3419] [client xxxxxx:42378] AH01630: client denied by server configuration: /var/www/html/data/.ocdata Nginx: 2017/05/12 16:46:20 [error] 23717#23717: *9 connect() failed (111: Connection refused) while connecting to upstream, client: xxxx, server: xxxx, request: "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1", upstream: "http://xxxx:80/ocs/v2.php/apps/notifications/api/v2/notifications", host: "xxxx" 2017/05/12 16:46:21 [error] 23717#23717: *7 connect() failed (111: Connection refused) while connecting to upstream, client: xxxx, server: xxxx, request: "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1", upstream: "http://xxxx:80/ocs/v2.php/apps/notifications/api/v2/notifications", host: "xxxx" 2017/05/12 16:46:25 [error] 23717#23717: *59 connect() failed (111: Connection refused) while connecting to upstream, client: xxxx, server: xxxx, request: "GET /status.php HTTP/1.1", upstream: "http://xxxx:80/status.php", host: "xxxx" 2017/05/12 16:46:26 [error] 23717#23717: *62 connect() failed (111: Connection refused) while connecting to upstream, client: xxxx, server: xxxx, request: "GET /status.php HTTP/1.1", upstream: "http://xxxx:80/status.php", host: "xxxx" --- Apache2 Config: ServerName xxxxx ServerAdmin admin@xxxxxx DocumentRoot /var/www/html Options +FollowSymlinks AllowOverride All Dav off SetEnv HOME /var/www/html SetEnv HTTP_HOME /var/www/html Header always set Strict-Transport-Security "max-age=15768000; preload" # keep the host Collabora Office ProxyPreserveHost On # static html, js, images, etc. served from loolwsd # loleaflet is the client part of LibreOffice Online ProxyPass /loleaflet https://127.0.0.1:9980/loleaflet retry=0 ProxyPassReverse /loleaflet https://127.0.0.1:9980/loleaflet # WOPI discovery URL ProxyPass /hosting/discovery https://127.0.0.1:9980/hosting/discovery retry=0 ProxyPassReverse /hosting/discovery https://127.0.0.1:9980/hosting/discovery # Main websocket ProxyPassMatch "/lool/(.*)/ws$" wss://127.0.0.1:9980/lool/$1/ws nocanon # Admin Console websocket ProxyPass /lool/adminws wss://127.0.0.1:9980/lool/adminws # Download as, Fullscreen presentation and Image upload operations ProxyPass /lool https://127.0.0.1:9980/lool ProxyPassReverse /lool https://127.0.0.1:9980/lool NGINX rProxy config: upstream mywebapp2 { server xxxxxxxx:80; } server { listen 80; listen 443 ssl; server_name xxxx; ssl on; ssl_certificate /etc/nginx/ssl/hum/server.pem; ssl_certificate_key /etc/nginx/ssl/hum/private.key; #ssl_trusted_certificate /etc/nginx/ssl/ca-bundle.crt; ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers ECDHE-RSA-AES256-SHA384:AES256-SHA256:RC4:HIGH:!MD5:!aNULL:!eNULL:!NULL:!DH:!EDH:!AESGCM; ssl_session_cache shared:SSL:5m; ssl_session_timeout 10m; ssl_prefer_server_ciphers on; location / { proxy_pass http://mywebapp2; client_max_body_size 20000M; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect off; proxy_buffers 64 4k; } location ~ /.well-known { allow all; proxy_pass http://localhost; } } ---- I also tried to rewrite the php.ini but that doesn't help I modified in the backend of NC the max upload size, but still same issue It would be very nice, if someone could help
2

Looks like you sql-connection is not kept up long enough, it could be mysql itself which closes the connection, or in the php config (mysql-section) defines a lower timeout.

3

How can I fix this?

4

Before anything else, I see you didn’t set any cache settings. This takes some load away from the database server, especially redis as file locking cache:
https://docs.nextcloud.com/server/11/admin_manual/configuration_server/caching_configuration.html?highlight=cache

perhaps this already solves most of your problems, if you need to continue:

your php.ini has a variable mysql.connect_timeout.

To reduce the number of components to consider, I would connect directly to my Nextcloud server (without proxy) and fix the issue (try higher timeout settings).

Another thought: If you run more or less the standard settings on your database, it might be worth optimizing them. Tuning-primer.sh or mysqltuner can help you to identify possible improvements. Increasing some cache-settings and lowering other non needed, can help to speed up the performance a lot and it is less likely to run into timeouts. Some of my improvements (https://github.com/owncloud/core/issues/20967#issuecomment-205474772):

innodb_buffer_pool_size = 512M
innodb_buffer_pool_instance = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 32M
innodb_max_dirty_pages_pct = 90

These were some starting parameters, I increased e.g. the buffer_pool_size to 4 GB. Please have a look at the variables and what they mean (especially innodb_flush_log_at_trx_commit). The difficulty is that there are no universal settings and finding the best ones is a continuing process.

5

I know now the failure.
I was using nextcloud in a Cluster Database. For this reason I used haproxy for loadbalancing the Database connections. So there I made a timeout after 5 Seconds. Thanks for your help and support.