Nextcloud version (eg, 19.0.3):
Operating system and version (QNAP QTS 4.4.3+Docker):
Apache or nginx version (Docker image 19.0.3):
CPU: Celeron J1900 @ 1.99GHz (4*cores)
RAM: 8GB
initially posted as comment to https://help.nextcloud.com/t/up-download-to-slow-only-2mb-s-after-installation-despite-good-hardware/96105/15 but break out to own topic due to recommendation of @KarlF12
it takes multiple days to perform initial sync to the clients - both on Win and Linux in local LAN with 1GBits/s. The folder contains 70k files with 40GB. I have local DNS server which points Nextcloud FQDN to local IP so there is no loop through Internet (split-brain DNS). I don’t see exhausted resource on server: CPU is running 30%-70% plenty of RAM is free, no error messages. if there is a big file I see transfer speeds up to 50-70 MB/s but the average is really bad as often it only syncs 1-2 files per seconds and consumes only few KB/s of network bandwidth. There are some directories with hundreds of small files (thumbnails of photo book software) - here I observe only 1-2 files are processed per second.
Upload/Download via Browser works with 4-30 MB/s.
download via Browser
Upload via Browser
I made a test and uploaded a video vile with 134MB with a browser - it took ~20 sec with very little pauses every 10MB (chunks) the speed shown in the progress bar was 3MB/s to 20MB/s. The file appeared on the clients local directory 30 sec later - sync took ~6 sec according to the client log, which is absolutely acceptable and shows disk and network perform fast. The problem I have is when lot of small files exist on a directory it takes ages…
2020-10-27 13:30:03:934 [ info nextcloud.gui.application ]: Sync state changed for folder "https://<my.nextcloud.fqdn>/remote.php/dav/files/<user>/Documents" : "Sync Running"
2020-10-27 13:30:03:935 [ info nextcloud.sync.propagator ]: Starting INSTRUCTION_NEW propagation of "Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" by OCC::PropagateDownloadFile(0x219c4d63090)
2020-10-27 13:30:03:936 [ info nextcloud.sync.accessmanager ]: 2 "" "https://<my.nextcloud.fqdn>/remote.php/dav/files/<user>/Documents/Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" has X-Request-ID "d30cff50-5b8f-4fb6-92f8-2a3fc1ce6c0a"
2020-10-27 13:30:03:937 [ info nextcloud.sync.networkjob ]: OCC::GETFileJob created for "https://<my.nextcloud.fqdn>" + "/Documents/Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" "OCC::PropagateDownloadFile"
2020-10-27 13:30:04:491 [ info nextcloud.sync.credentials.webflow ]: request finished
2020-10-27 13:30:04:491 [ info nextcloud.sync.networkjob.etag ]: Request Etag of QUrl("https://<my.nextcloud.fqdn>/remote.php/dav/files/<user>/Contacts") FINISHED WITH STATUS "OK"
2020-10-27 13:30:08:313 [ info nextcloud.sync.credentials.webflow ]: request finished
2020-10-27 13:30:08:326 [ info nextcloud.sync.checksums ]: Computing "SHA1" checksum of "W:/Nextcloud/Documents/.Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4.~fa9" in a thread
2020-10-27 13:30:08:660 [ info nextcloud.sync.database ]: Updating file record for path: "Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" inode: 53 modtime: 1507790579 type: 0 etag: "e8cc653832d587e1493e7a777b34c60f" fileId: "00475173ocshh0oigtc5" remotePerm: "WDNVR" fileSize: 140892641 checksum: "SHA1:d2021affa8db72d65f0220b44f07004a74eef3e9" e2eMangledName: "" isE2eEncrypted: false
2020-10-27 13:30:08:662 [ info nextcloud.sync.propagator ]: Completed propagation of "Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" by OCC::PropagateDownloadFile(0x219c4d63090) with status 4
2020-10-27 13:30:08:662 [ warning nextcloud.gui.activity ]: Item "Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" retrieved resulted in ""
2020-10-27 13:30:08:663 [ warning nextcloud.gui.activity ]: Item "Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4" retrieved successfully.
2020-10-27 13:30:08:663 [ info nextcloud.gui.activity ]: Successfully added to the activity list: ""
2020-10-27 13:30:08:664 [ info nextcloud.gui.folderwatcher ]: Detected changes in paths: QSet("W:/Nextcloud/Documents/Ask me anything, with Mark Russinovich on cloud, Sysinternals, security, and more-P1pk9VBdwUs.mp4")
2020-10-27 13:30:08:665 [ info nextcloud.sync.database ]: Closing DB "C:/Users/<user>/AppData/Roaming/Nextcloud/._sync_6d0d8033c51e.db"
2020-10-27 13:30:08:675 [ info nextcloud.sync.engine ]: CSync run took 6145 ms
from the log it look like the client i doing many many thing just to replicate one file - is it expected?
The file appeared on the local client few seconds later. I checked the client log during the sync and see it executed bunch SQL queries for ALL existing files in the directory where I put the upload is this expected?
2020-10-27 13:16:02:962 [ info nextcloud.sync.csync.updater ]: Database entry found for Readme.md, compare: 1597168289 <-> 1597168289, etag: 0aba3361429e9afd483f17bb8be7644a <-> 0aba3361429e9afd483f17bb8be7644a, inode: 0 <-> 1116, size: 136 <-> 136, perms: 9f <-> 9f, checksum: <-> , ignore: 0, e2e:
2020-10-27 13:16:02:962 [ info nextcloud.sync.csync.updater ]: file: Readme.md, instruction: INSTRUCTION_NONE <<=
2020-10-27 13:16:02:962 [ debug nextcloud.sync.csync.updater ] [ csync_walker ]: file: Welcome to Nextcloud Hub.docx [file_id=00000210ocshh0oigtc5 size=25150]
2020-10-27 13:16:02:963 [ debug nextcloud.sync.database.sql ] [ OCC::SqlQuery::bindValue ]: SQL bind 1 QVariant(qlonglong, 3622897607555501721)
2020-10-27 13:16:02:963 [ debug nextcloud.sync.database.sql ] [ OCC::SqlQuery::exec ]: SQL exec "SELECT path, inode, modtime, type, md5, fileid, remotePerm, filesize, ignoredChildrenRemote, contentchecksumtype.name || ':' || contentChecksum, e2eMangledName, isE2eEncrypted FROM metadata LEFT JOIN checksumtype as contentchecksumtype ON metadata.contentChecksumTypeId == contentchecksumtype.id WHERE phash=?1"
I already tuned my mysql with the following settings:
innodb_flush_log_at_trx_commit = 2
innodb_flush_log_at_timeout = 5
max_connections = 250
which made the system feel little more responsive but still not really good to sync thousands of files. having max_connections = 100 I had an error in the logs that DB stopped responding because of “too many connections” - why does the system with 2 users and 3 clients need more then 100 sql connections?
docker stats shows my Nextcloud container spiking from 0 to 130% cpu all the time (every 1-2 seconds)
CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
f3cbbca40b9d 130.33% 106.6MiB / 7.671GiB 1.36% 451GB / 111GB 183GB / 858MB 0
But syncing small files is terribly slow - I see the client only syncs 1-2 small files per second - even this are only KB. Web interface is pretty slow on initial load 5-10 sec spinning after entering a folder - but once it’s loaded one can view/download files really fast. I don’t see any errors in logs.
@KarlF12 assumed the disk is slow - from my feeling disk speed is not an issue as big files transfer fast. Copying the data from/to the system via NFS or SMB is fast and saturates the 1GBit/s network adapter. I run the speed test as recommended by @KarlF12 read test performed after clearing caches. The speed looks quite good for me…
#sync; dd if=/dev/zero of=./ddtestfile bs=1M count=1024;sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 8.834910 seconds, 115.9MB/s
# sync; dd if=./ddtestfile of=/dev/null bs=1M count=1024;sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.0GB) copied, 10.101529 seconds, 101.4MB/s
# hdparm -Tt /dev/md0
/dev/md0:
Timing cached reads: 4804 MB in 2.00 seconds = 2403.63 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
Timing buffered disk reads: 394 MB in 3.01 seconds = 131.09 MB/sec
HDIO_DRIVE_CMD(null) (wait for flush complete) failed: Inappropriate ioctl for device
The output of your Nextcloud log in Admin > Logging:
#can't upload the log due to post limits, the message are more or less the same majority of the logs are:
not enough messages to train a classifier
#and once an hour there is such message:
OCA\Mail\Exception\ServiceException: IMAP errorError connecting to mail server.
The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):
<?php
$CONFIG = array (
'htaccess.RewriteBase' => '/',
'memcache.local' => '\\OC\\Memcache\\APCu',
'apps_paths' =>
array (
0 =>
array (
'path' => '/var/www/html/apps',
'url' => '/apps',
'writable' => false,
),
1 =>
array (
'path' => '/var/www/html/custom_apps',
'url' => '/custom_apps',
'writable' => true,
),
),
'memcache.distributed' => '\\OC\\Memcache\\Redis',
'memcache.locking' => '\\OC\\Memcache\\Redis',
'redis' =>
array (
'host' => 'nextcloud-redis',
'password' => '***SECRET***',
'port' => 6379,
),
'instanceid' => 'ocshh0oigtc5',
'passwordsalt' => '***SECRET***',
'secret' => '***SECRET***',
'trusted_domains' =>
array (
0 => '<my.nextcloud.fqdn>',
),
'datadirectory' => '/var/www/html/data',
'dbtype' => 'mysql',
'version' => '19.0.3.1',
'overwrite.cli.url' => 'https://<my.nextcloud.fqdn>',
'dbname' => 'nextcloud',
'dbhost' => 'nextcloud-db',
'dbport' => '',
'dbtableprefix' => 'oc_',
'mysql.utf8mb4' => true,
'dbuser' => 'nextcloud',
'dbpassword' => '***SECRET***',
'installed' => true,
'overwritehost' => '<my.nextcloud.fqdn>',
'overwriteprotocol' => 'https',
'mail_smtpmode' => 'smtp',
'mail_sendmailmode' => 'smtp',
'mail_from_address' => 'nextcloud',
'mail_domain' => '***SECRET***',
'mail_smtphost' => '***SECRET***',
'mail_smtpauth' => 1,
'mail_smtpname' => '***SECRET***',
'mail_smtppassword' => '***SECRET***',
'mail_smtpport' => '465',
'mail_smtpauthtype' => 'PLAIN',
'mail_smtpsecure' => 'ssl',
'maintenance' => false,
'trusted_proxies' =>
array (
0 => '\'traefik',
1 => '172.16.0.0/12',
2 => '192.168.0.0/16',
3 => '10.0.0.0/8\'',
),
'theme' => '',
'loglevel' => 2,
);