Hey so I am actually using rsync now, I gave up on the app because it crashes for a long time before it resumes. It isn’t that the USB disk is slow, it’s fast, about 500mbps or so when it’s 100%, but the reads that also overload it aren’t transfering any data, they are just reads so the nextcloud app can have a list of file to transfer, it’s for that local database file that gets created I assume. I think this because the disk is 100% and the reads are in Kilobytes, no data is even being uploaded, app is just preparing to sync, it says file 0 out of, and it takes a very long time to reach 450000 and plus it crashes and starts from scratch.
The server is 2 physical cores. I’ve downloaded real files on it at 500mbps. Server isn’t here, I’m uploading from a Dell Optiples with USB 3.0 because Dropbox wouldn’t allow syncing to a SMB share.
Of course there are probably other factors too. For example when I count files on the server I do “find /folder | wc -l” and this takes one second but when I ran that command on the Desktop, and the Desktop is a Windows with Linux Subsystem, the disk got to a 100%, CPU core temp went to 54 degrees and the Fans sounded like a aeroplane taking off. Rsync just uses 1% of the disk locally. 12% of the CPU on the server, it’s around 30mbps but it did get to 102mbps once, it’s mostly 40 GB PDF files and then some 1mb photos etc.
So I guess the problem isn’t when the files are transfering as rsync does it well, it’s building the database and list of files. Disk gets overloaded the same with nextcloud app or with find command.
I can’t really tell what would happen if the disk wasn’t a USB disk, but I just ran the ‘find’ command on C:/Program Files and the regular HDD got to a 20%
So maybe that is a bottleneck but even though rsync should save me 6 days. At this rate it will complete in 3.7 days where as nextcloud app would take over 10 days.
The main reason I wanted to use the Nextcloud app was because some files had leading whitespaces so it let me know to rename them, and some characters aren’t allowed on nextcloud and then it would let me know.
I guess if rsync maxed out on 30mbps that’s it, it’s not really an nc desktop app issue. It could be the USB disk or just the fact that files are small and not much sequential writes happen like when you have movie files.
I’ll let you know how long it takes occ file:scan to scan 500000 files in 3 days or so