Way to reduce CPU load?

Nextcloud version: 18.0.4

Operating system and version: Ubuntu 18.04
Nginx version: 1.14.0
PHP version: 7.2

The issue you are facing:
Many messages “Unknown error” (in russian locale “Произошла неизвестная ошибка”) appear while uploading large amount of small files (jpeg photos mostly) to a folder in my nextcloud. As a result few files are missing by the end of uploading.
In this example Im uploading 42GB in 6333 files. Uploading process is moving quite fast regarding to data’s size. But there is almost 100% of cpu load on server.
I have a small server - HP ML10G9 with G4400 3.3Ghz 4GB. It is in my home 1gbit LAN.
It works well processing small batches but almost dies under such tasks.

So my question is how to avoid this bottleneck? Is there any way to control the bandwidth of incoming traffic? I’ve tried limit php queries in nginx but it doesnt stacks with NXC and generate even more errors.

All checks of NXC are passed.
/mnt/data - is the mdadm array of 2 HDD.

Is this the first time you’ve seen this error?: N

Steps to replicate it:

  1. Server’s entry level cpu
  2. Upload 5000 files other the fast connection

The output of your Nextcloud log in Admin > Logging:

cut@cut:~$ sudo tail -30 /var/log/nextc-fqdn.log
{"reqId":"ACi1RTWRBkKKDwFM3HRv","level":3,"time":"2020-05-03T16:04:25+03:00","remoteAddr":"my_ip","user":"Jane","app":"no app in context","method":"PUT","url":"/remote.php/webdav/Photos/2012/2014-07-14%2020-22-11.JPG","message":{"Exception":"Sabre\\DAV\\Exception\\BadRequest","Message":"Expected filesize of 1980242 bytes but read (from Nextcloud client) and wrote (to Nextcloud storage) 122880 bytes. Could either be a network problem on the sending side or a problem writing to the storage on the server side.","Code":0,"Trace":[{"file":"/mnt/data/www/fqdn/apps/dav/lib/Connector/Sabre/Directory.php","line":156,"function":"put","class":"OCA\\DAV\\Connector\\Sabre\\File","type":"->","args":[null]},{"file":"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/Server.php","line":1096,"function":"createFile","class":"OCA\\DAV\\Connector\\Sabre\\Directory","type":"->","args":["2014-07-14 20-22-11.JPG",null]},{"file":"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":525,"function":"createFile","class":"Sabre\\DAV\\Server","type":"->","args":["Photos/2012/2014-07-14 20-22-11.JPG",null,null]},{"function":"httpPut","class":"Sabre\\DAV\\CorePlugin","type":"->","args":[{"absoluteUrl":"https://fqdn/remote.php/webdav/Photos/2012/2014-07-14%2020-22-11.JPG","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/mnt/data/www/fqdn/3rdparty/sabre/event/lib/EventEmitterTrait.php","line":105,"function":"call_user_func_array","args":[[{"__class__":"Sabre\\DAV\\CorePlugin"},"httpPut"],[{"absoluteUrl":"https://fqdn/remote.php/webdav/Photos/2012/2014-07-14%2020-22-11.JPG","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/Server.php","line":479,"function":"emit","class":"Sabre\\Event\\EventEmitter","type":"->","args":["method:PUT",[{"absoluteUrl":"https://fqdn/remote.php/webdav/Photos/2012/2014-07-14%2020-22-11.JPG","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]]},{"file":"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/Server.php","line":254,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->","args":[{"absoluteUrl":"https://fqdn/remote.php/webdav/Photos/2012/2014-07-14%2020-22-11.JPG","__class__":"Sabre\\HTTP\\Request"},{"__class__":"Sabre\\HTTP\\Response"}]},{"file":"/mnt/data/www/fqdn/apps/dav/appinfo/v1/webdav.php","line":82,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/mnt/data/www/fqdn/remote.php","line":165,"args":["/mnt/data/www/fqdn/apps/dav/appinfo/v1/webdav.php"],"function":"require_once"}],"File":"/mnt/data/www/fqdn/apps/dav/lib/Connector/Sabre/File.php","Line":229,"CustomMessage":"--"},"userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.96 YaBrowser/20.4.0.1461 Yowser/2.5 Safari/537.36","version":"18.0.4.2"}

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

sudo cat /mnt/data/www/fqdn/config/config.php
<?php
$CONFIG = array (
  'instanceid' => '',
  'passwordsalt' => '',
  'secret' => '',
  'trusted_domains' =>
  array (
    0 => '192.168.1.100',
    1 => 'fqdn',
  ),
  'datadirectory' => '/mnt/data/www/fqdn/data',
  'dbtype' => 'mysql',
  'version' => '18.0.4.2',
  'overwrite.cli.url' => 'https://fqdn',
  'dbname' => 'nxc-lv',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => '',
  'dbpassword' => '',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\Redis',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => '/var/run/redis/redis.sock',
    'port' => 0,
  ),
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'mysql.utf8mb4' => true,
  'maintenance' => false,
  'default_language' => 'ru',
  'default_locale' => 'ru_RU',
  'allow_user_to_change_display_name' => true,
  'auth.bruteforce.protection.enabled' => true,
  'logtimezone' => 'Europe/Moscow',
  'log_type' => 'owncloud',
  'logfile' => '/var/log/nextc-fqdn.log',
  'loglevel' => '2',
  'log_rotate_size' => '104857600',
);

The output of your Apache/nginx/system log in /var/log/____:

2020/05/03 16:31:14 [error] 16742#16742: *16404 connect() to unix:/var/run/php/php7.2-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 93.159.233.75, server: fqdn, request: "PUT /remote.php/dav/uploads/Jane/web-file-upload-e14b83a9e7da25ff32b530ddae0a6f97-1588512650618/0 HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "fqdn"
2020/05/03 16:31:14 [error] 16742#16742: *16404 connect() to unix:/var/run/php/php7.2-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 93.159.233.75, server: fqdn, request: "GET /index.php/apps/files/ajax/getstoragestats.php?dir=%2FPhotos HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "fqdn"
2020/05/03 16:31:14 [error] 16742#16742: *16404 connect() to unix:/var/run/php/php7.2-fpm.sock failed (11: Resource temporarily unavailable) while connecting to upstream, client: 93.159.233.75, server: fqdn, request: "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.2-fpm.sock:", host: "fqdn"
2020/05/03 16:32:41 [notice] 17024#17024: signal process started
2020/05/03 16:57:15 [error] 17025#17025: *20835 access forbidden by rule, client: my_ip, server: fqdn, request: "GET /data/.ocdata?t=1588514235160 HTTP/2.0", host: "fqdn"

Try to confirm if the same happens with non image files. It might be the thumbnailer that takes up all the CPU.

This isn’t much of a solution, but if this is a one-time upload that you just need to get past, you might try backing your NIC down to 100 Mbps temporarily.

Tried to upload big folder folder with PC game. The same result.
Uploading process is moving along with 1 mysqld (about 30% of cpu time) and 20-30 php-fpm subprocesses.

Brought down ethernet to 100 Mbps.
Uploading right now but the speed of upload is really low. Less than 100 Mb in 5 minutes…
image

The next screenshot show above average cpu load for now:
image

1 Gb in 1 hour isn’t enough for my purposes :smiley:
Pressing a key to abort upload caused many “unknown errors”.

NXC.log

{“reqId”:“xPyx3ExOkbcEVPkGy4u0”,“level”:4,“time”:“2020-05-04T13:06:51+03:00”,“remoteAddr”:“my_ip”,“user”:“Jane”,“app”:“webdav”,“method”:“PUT”,“url”:"/remote.php/dav/uploads/Jane/web-file-upload-e14b83a9e7da25ff32b530ddae0a6f97-1588583500765/0",“message”:{“Exception”:“Sabre\DAV\Exception”,“Message”:“Could not rename part file to final file”,“Code”:0,“Trace”:[{“file”:"/mnt/data/www/fqdn/apps/dav/lib/Connector/Sabre/Directory.php",“line”:156,“function”:“put”,“class”:“OCA\DAV\Connector\Sabre\File”,“type”:"->",“args”:[null]},{“file”:"/mnt/data/www/fqdn/apps/dav/lib/Upload/UploadFolder.php",“line”:47,“function”:“createFile”,“class”:“OCA\DAV\Connector\Sabre\Directory”,“type”:"->",“args”:[“0”,null]},{“file”:"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/Server.php",“line”:1096,“function”:“createFile”,“class”:“OCA\DAV\Upload\UploadFolder”,“type”:"->",“args”:[“0”,null]},{“file”:"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",“line”:525,“function”:“createFile”,“class”:“Sabre\DAV\Server”,“type”:"->",“args”:[“uploads/Jane/web-file-upload-e14b83a9e7da25ff32b530ddae0a6f97-1588583500765/0”,null,null]},{“function”:“httpPut”,“class”:“Sabre\DAV\CorePlugin”,“type”:"->",“args”:[{“absoluteUrl”:“https://fqdn/remote.php/dav/uploads/Jane/web-file-upload-e14b83a9e7da25ff32b530ddae0a6f97-1588583500765/0","class”:“Sabre\HTTP\Request”},{“class”:“Sabre\HTTP\Response”}]},{“file”:"/mnt/data/www/fqdn/3rdparty/sabre/event/lib/EventEmitterTrait.php",“line”:105,“function”:“call_user_func_array”,“args”:[[{“class”:“Sabre\DAV\CorePlugin”},“httpPut”],[{“absoluteUrl”:“https://fqdn/remote.php/dav/uploads/Jane/web-file-upload-e14b83a9e7da25ff32b530ddae0a6f97-1588583500765/0","class”:“Sabre\HTTP\Request”},{“class”:“Sabre\HTTP\Response”}]]},{“file”:"/mnt/data/www/nfqdn/3rdparty/sabre/dav/lib/DAV/Server.php",“line”:479,“function”:“emit”,“class”:“Sabre\Event\EventEmitter”,“type”:"->",“args”:[“method:PUT”,[{“absoluteUrl”:“https://fqdn/remote.php/dav/uploads/Jane/web-file-upload-e14b83a9e7da25ff32b530ddae0a6f97-1588583500765/0","class”:“Sabre\HTTP\Request”},{“class”:“Sabre\HTTP\Response”}]]},{“file”:"/mnt/data/www/fqdn/3rdparty/sabre/dav/lib/DAV/Server.php",“line”:254,“function”:“invokeMethod”,“class”:“Sabre\DAV\Server”,“type”:"->",“args”:[{“absoluteUrl”:“https://fqdn/remote.php/dav/uploads/Jane/web-file-upload-e14b83a9e7da25ff32b530ddae0a6f97-1588583500765/0","class”:“Sabre\HTTP\Request”},{“class”:“Sabre\HTTP\Response”}]},{“file”:"/mnt/data/www/fqdn/apps/dav/lib/Server.php",“line”:319,“function”:“exec”,“class”:“Sabre\DAV\Server”,“type”:"->",“args”:[]},{“file”:"/mnt/data/www/fqdn/apps/dav/appinfo/v2/remote.php",“line”:35,“function”:“exec”,“class”:“OCA\DAV\Server”,“type”:"->",“args”:[]},{“file”:"/mnt/data/www/fqdn/remote.php",“line”:165,“args”:["/mnt/data/www/fqdn/apps/dav/appinfo/v2/remote.php"],“function”:“require_once”}],“File”:"/mnt/data/www/fqdn/apps/dav/lib/Connector/Sabre/File.php",“Line”:280,“CustomMessage”:"–"},“userAgent”:“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.96 YaBrowser/20.4.0.1461 Yowser/2.5 Safari/537.36”,“version”:“18.0.4.2”}

php-fpm asks for servers. How much does it need for nxc?

[04-May-2020 13:05:53] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 4 idle, and 113 total children

You have something else going on then because 100 Mbps should transfer 1 GB in about a minute and a half.

Transfer same folder (30Gb PC game) by a samba works perfect and takes about 10 minutes over 1 Gbps. I mean my network and disk subsystem are in good condition.

Also getting few new messages in mysql slow.log after messages “unknown error”

slow.log

User@Host: myhost @ localhost []

Thread_id: 3305 Schema: myhost QC_hit: No

Query_time: 1.822770 Lock_time: 0.000076 Rows_sent: 0 Rows_examined: 0

Rows_affected: 1

SET timestamp=1588596032;
INSERT INTO oc_filecache (mimepart, mimetype, mtime, size, etag, storage_mtime, permissions, parent, checksum, path_hash, path, name, storage) VALUES(‘3’, ‘15’, ‘1588596030’, ‘32040’, ‘5d9323de40ab22a7f49808acb8c58db5’, ‘1588596030’, ‘27’, ‘93885’, ‘’, ‘1d0c826c6279a34ed0d08008fb23fbb6’, ‘files/WarThunder/cache/content/34/o6nliyj5ce63xhzibohkz2jtpek54v-7ji.ddsx’, ‘o6nliyj5ce63xhzibohkz2jtpek54v-7ji.ddsx’, ‘3’);

mariadb.cnf
[server]
skip-name-resolve
innodb_buffer_pool_size = 1024M
innodb_buffer_pool_instances = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 128M
innodb_max_dirty_pages_pct = 90
innodb_log_file_size = 128M
query_cache_type = 0
query_cache_limit = 2M
query_cache_min_res_unit = 2k
query_cache_size = 16M
tmp_table_size= 128M
max_heap_table_size= 128M
slow-query-log = 1
slow-query-log-file = /var/log/mysql/slow.log
long_query_time = 1

[client-server]

# Import all .cnf files from configuration directory
!includedir /etc/mysql/conf.d/
!includedir /etc/mysql/mariadb.conf.d/

[client]
default-character-set = utf8mb4

[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_general_ci
transaction_isolation = READ-COMMITTED
binlog_format = ROW
innodb_large_prefix=on
innodb_file_format=barracuda
innodb_file_per_table=1

But the upload speed was satisfactory before? Very odd. If nothing else changed, you should have gotten a steady transfer of about 11 MB/s.

So even at 1 Gbps, you were still only seeing about half wire speed on the transfer.

If you don’t mind me asking, why are you transferring an entire game installation? That’s an unusual thing to do. If it were me I’d just symlink the folder where the saves are rather than copy dozens of gigabytes of useless data.

** KarlF12**
Thank you a lot for participation! It is very important to have a support.

I switched to transfer another random but big folder to exclude possible influence of generating thumbnails when uploading thousands of jpegs. It is just for testing.

Was tested:
1 Gbps:
NXC - uploading speed is good. Have “unknown errors”.
Samba - uploading speed is very good. No errors.
100 Mbits:
NXC - uploading speed is very slow. Still errors. Much more less though.
Samba (just made this test) - uploading speed is quite slow. No errors.

100mbit samba

image

Maybe something wrong with my cheap TP-Link router on 100 mbit. But I would rather prefer working with 1 gbps because I need sometimes fast samba shares too.

Hmm. I wonder if maybe you had a duplex mismatch when you set it to 100 Mbps.

And I didn’t mean to leave it that way long term, just for the initial upload if the CPU utilization was causing a problem.

How are you uploading the files? Using the Nextcloud desktop client?

1 Like

Aint found current mode of eth ports on tp-link.
Was switching modes in Ubuntu (100/1000): ethtool -s eno1 speed 1000 duplex full autoneg off

Through web-interface. My bad I’ve never heard about desktop client! :smiley: Just downloaded it. Will try it shortly.

Yeah, sorry about that, I wouldn’t have suggested changing the link speed on a router like that.

One thing that can happen sometimes if you have one end of a link set to auto negotiate and the other set manual full duplex, the end set to autoneg will pick up the speed but drop to half duplex. Then that end starts registering constant collisions and the speed tanks.

KarlF12
Synchronization works perfectly with desktop client! Thanks a lot for your help!

Obviously there is an issue with 100 mbps but it is ok for me as long as 1 gpbs works good.

For such a large one-time upload, I’d think rsync would be the better tool in general, and then do a occ scan to sync the files to nextcloud’s database?

1 Like

TY but i’de prefer more universal solution. Desktop client is work perfectly for initial upload. There are still some issues if you are already have some files uploaded and want to add more files to the same folder with desktop app. But it is not a big problem.

If you don’t want to use the Nextcloud Desktop Client you can also use the app Flowupload. It lets you upload thousands of huge files without problems, because if a upload fails you can just resume it. The limit is pretty much your storage :slight_smile:
I like flowupload so much, that i contributed a lot of features via Pull Requests.

1 Like