Nextcloud version: 18.0.4
Operating system and version: Ubuntu 18.04
Nginx version: 1.14.0
PHP version: 7.2
The issue you are facing:
Many messages “Unknown error” (in russian locale “Произошла неизвестная ошибка”) appear while uploading large amount of small files (jpeg photos mostly) to a folder in my nextcloud. As a result few files are missing by the end of uploading.
In this example Im uploading 42GB in 6333 files. Uploading process is moving quite fast regarding to data’s size. But there is almost 100% of cpu load on server.
I have a small server - HP ML10G9 with G4400 3.3Ghz 4GB. It is in my home 1gbit LAN.
It works well processing small batches but almost dies under such tasks.
So my question is how to avoid this bottleneck? Is there any way to control the bandwidth of incoming traffic? I’ve tried limit php queries in nginx but it doesnt stacks with NXC and generate even more errors.
All checks of NXC are passed.
/mnt/data - is the mdadm array of 2 HDD.
Is this the first time you’ve seen this error?: N
Steps to replicate it:
- Server’s entry level cpu
- Upload 5000 files other the fast connection
The output of your Nextcloud log in Admin > Logging:
cut@cut:~$ sudo tail -30 /var/log/nextc-fqdn.log
{"reqId":"ACi1RTWRBkKKDwFM3HRv","level":3,"time":"2020-05-03T16:04:25+03:00","remoteAddr":"my_ip","user":"Jane","app":"no app in context","method":"PUT","url":"/remote.php/webdav/Photos/2012/2014-07-14%2020-22-11.JPG","message":{"Exception":"Sabre\\DAV\\Exception\\BadRequest","Message":"Expected filesize of 1980242 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 122880 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.","Code":0,"Trace":[{"file":"/mnt/data/www/fqdn/apps/dav/lib/Connector/Sabre/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/Server.php","line":1096,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["2014-07-14 20-22-11.JPG",null]},{"file":"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":525,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["Photos/2012/2014-07-14 20-22-11.JPG",null,null]},{"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https://fqdn/remote.php/webdav/Photos/2012/2014-07-14%2020-22-11.JPG","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/mnt/data/www/fqdn/3rdparty/sabre/event/lib/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpPut"],[{"absoluteUrl":"https://fqdn/remote.php/webdav/Photos/2012/2014-07-14%2020-22-11.JPG","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:PUT",[{"absoluteUrl":"https://fqdn/remote.php/webdav/Photos/2012/2014-07-14%2020-22-11.JPG","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https://fqdn/remote.php/webdav/Photos/2012/2014-07-14%2020-22-11.JPG","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/mnt/data/www/fqdn/apps/dav/appinfo/v1/webdav.php","line":82,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/mnt/data/www/fqdn/remote.php","line":165,"args":["/mnt/data/www/fqdn/apps/dav/appinfo/v1/webdav.php"],"function":"require_once"}],"File":"/mnt/data/www/fqdn/apps/dav/lib/Connector/Sabre/File.php","Line":229,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.96 YaBrowser/20.4.0.1461 Yowser/2.5 Safari/537.36","version":"18.0.4.2"}
The output of your config.php file in /path/to/nextcloud
(make sure you remove any identifiable information!):
sudo cat /mnt/data/www/fqdn/config/config.php
<?php
$CONFIG = array (
'instanceid' => '',
'passwordsalt' => '',
'secret' => '',
'trusted_domains' =>
array (
0 => '192.168.1.100',
1 => 'fqdn',
),
'datadirectory' => '/mnt/data/www/fqdn/data',
'dbtype' => 'mysql',
'version' => '18.0.4.2',
'overwrite.cli.url' => 'https://fqdn',
'dbname' => 'nxc-lv',
'dbhost' => 'localhost',
'dbport' => '',
'dbtableprefix' => 'oc_',
'dbuser' => '',
'dbpassword' => '',
'installed' => true,
'memcache.local' => '\\OC\\Memcache\\Redis',
'memcache.distributed' => '\\OC\\Memcache\\Redis',
'redis' =>
array (
'host' => '/var/run/redis/redis.sock',
'port' => 0,
),
'memcache.locking' => '\\OC\\Memcache\\Redis',
'mysql.utf8mb4' => true,
'maintenance' => false,
'default_language' => 'ru',
'default_locale' => 'ru_RU',
'allow_user_to_change_display_name' => true,
'auth.bruteforce.protection.enabled' => true,
'logtimezone' => 'Europe/Moscow',
'log_type' => 'owncloud',
'logfile' => '/var/log/nextc-fqdn.log',
'loglevel' => '2',
'log_rotate_size' => '104857600',
);
The output of your Apache/nginx/system log in /var/log/____
:
2020/05/03 16:31:14 [error] 16742#16742: *16404 connect() to unix:/var/run/php/php7.2-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 93.159.233.75, server: fqdn, request: "PUT /remote.php/dav/uploads/Jane/web-file-upload-e14b83a9e7da25ff32b530ddae0a6f97-1588512650618/0 HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "fqdn"
2020/05/03 16:31:14 [error] 16742#16742: *16404 connect() to unix:/var/run/php/php7.2-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 93.159.233.75, server: fqdn, request: "GET /index.php/apps/files/ajax/getstoragestats.php?dir=%2FPhotos HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "fqdn"
2020/05/03 16:31:14 [error] 16742#16742: *16404 connect() to unix:/var/run/php/php7.2-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 93.159.233.75, server: fqdn, request: "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "fqdn"
2020/05/03 16:32:41 [notice] 17024#17024: signal process started
2020/05/03 16:57:15 [error] 17025#17025: *20835 access forbidden by rule, client: my_ip, server: fqdn, request: "GET /data/.ocdata?t=1588514235160 HTTP/2.0", host: "fqdn"