All, I am using a brand new AIO on a dedicated server on Ubuntu 22.04 LTS, which is locally hosted. I have installed the Deskop Client on my Windows Laptop and threw a test load of 15.000 files with a total size of 5.1GB into the Next Cloud folder in my File Explorer.
The sync started but seems slow: the client seems to sync in batches of 100 files and then takes a LOOOONG break.
I know that public services such as OneDrive rate-limit the sync, but when I set up Office 365 for non-profits I work with, I can easily sync 150-200 GB over night.
I have had my NC instance running for 24 hours now, and it has synced less than 2 GB so far.
At that rate, syncing the normal amount of data I usually work with would take half a year. So I assume this cant be normal?
I just have a vanilla AIO (latest version) on Ubuntu 22.04 LTS, and use that with the latest Windows Desktop App. All pretty much as they come āout of the boxā, as I assumed that I wonāt have to tweak technical parameters to make this product useable.
I could certainly go down the path you proposed, but I wonder: is the need to do this type of thing really common in the NC world?
The vanilla AIO installation is really a common starting point that provides a foundation install. They do make some assumptions regarding connectivity, DNS, networking, etc. that make the installation easy, but may require some tweaking to get the most out of it. The good news is that itās enterprise software, and it can be tweaked.
The software runs well in a relatively small footprint but that may need to be expanded a little bit, the defaults for small files handling may need to be optimized for your use case (some have used rsync for the initial bulk file load) and certain features may add additional overhead/delay.
Are you cross loading the files via LAN or via Internet connection?
How is the NextCloud install connected to the network/internet?
How is the server configured for file storage?
The changes to the system files are relatively easy to make in the containers:
Access the container command line from the linux host:
docker exec -it nextcloud-aio-nextcloud bash
Install nano to edit files locally:
apk add nano (remove via apk delete nano when finished)
make a few changes (nano filename, ctl-W to save)
May be worth while testing the changes, see what impact they have in addition to checking out the nextcloud logs to see if there are any issues - https://[your nc server]:8080 has a nice log access GUI, or go command line.
I tend to check the logs first, see if there are any issues, check the basic configuration, then make changes and test.
I sync from my Desktop (pretty vanilla Windows 10 Pro) to my server (Dell T320 with 6-core XEON, 48GB RAM, 4x 1TB SAS HDD in RAID-5 with Dell HBA, equipped with Ubuntu 22.04 LTS and no hypervisor) using a Gigabit LAN connection. Both server and windows workstation are wired, no wifi involved.
No load on the server, no load on the windows workstation.
I am still sycing the first 5.1GB, by the way, and I am over 50 hours in.
I would definitely expect AIO to offer suitable performance out of the box given my environment.
Having to tweak configuration and dive deep into the technology of a product such as NC to get suitable performance and use out of it is something I have seen before, and it was usually attributed to āearly stageā of the product lifecycle. Perhaps NC is still too new? Perhaps I just need to wait a few years and then revisit it, because if this level of technology involvement is required to even get basic use cases working, I have to assume it is a maturity issue, and NC is just ānot there yetā.
Still, I believe the simplicity of things like AIO is key to mass adoption. Other products that do a great job with that are pihole and Splunk, I think. Super easy to get a basic system up and running in minutes, but plenty of blueprints to scale it out.
Slow syncing could be due to limited server resources, misconfigured Nextcloud settings, network issues, or the desktop clientās handling of many files. Check your serverās CPU, RAM, and disk space, optimize Nextcloud for large files, ensure a fast network connection, and consider excluding unnecessary files. Also, check for database load and optimize it.
Depending on the router, its hardware can be the bottle neck. When using a domain for nextcloud and not tweaking DNS, the whole traffic does not go directly via LAN to the server but has to pass the WAN interface in the router. That may slow it down. You could test from āoutsideā by using a mobile connection and see whether the upload increases.
limited server resources: doubt it. Itās a Dell T320 with 6-Core CPU, 48 GB RAM, 4x 1TB SAS Drive array in RAID-5 using a hardware HBA, and the onboard GB NIC, using Ubuntu 22.04 LTS (fairly vanilla install) and no hypervisor. Other than a tiny minecraft instance, that server does nothing. TOP shows it mostly idle.
misconfigured Nextcloud settings: I use the AIO settings as they come in the latest release. Have not changed any settings, so if the NC settings are the issue, the root cause would be in the AIO images. But then other users would expecience the same issue, I would think.
Check your serverās CPU, RAM, and disk space: CPU load average under 20% across all cores, 33 of 48 GB RAM available, 2.3TB of 3 TB disk space available.
ensure a fast network connection: wired GB ethernet end to end
consider excluding unnecessary files: do you mean choosing to NOT sync unnecessary files? See, the ~15.000 files (5.1 GB total) that I still try to sync (still not done today) was just a test balloon. In normal use cases, I would sync 100x more files in the low TB range. But if NC canāt handle 1% of a typical workload in a decent amount of time, it may just not be suitable for me. But I think this canāt be the case because if this was a common issue, nobody would use NC, I assume.
I use the Ubiquiti USG-Pro-4 as router, so my backplane is limited to 1 GBit/s, and since my server and my windows PC are in different Subnets, traffic does indeed have to traverse the router.
However, when I interact with the server through other means, I have never experienced the network being the issue.
In fact, just for fun, I copied the ~15.000 files (5.1 GB total) from the same windows PC to the same server using WinSCP and despite the overhead involved, I get averate transfer speeds of 60-70 MBytes/s (not MBit/s), and the WinSCP upload of the entire set of files was done in less than 2 minutes.
I do use an internal DNS record to ensure the traffic between my Windows PC and the server does not have to go through the WAN interface. But even if it had to do that, I have a symmetric GB/s connection.
thank you for that. Not fully, but lets go through the list:
Use SSDs: I donāt have SSDs, only SAS Enterprise drives in a hardware RAID-5, but for my current test case, that should not be a limiting factor.
Run Docker and AIO not in a VM: Yes, I run Docker and AIO on a plain Ubuntu 22.04 LTS Server install, not inside a hypervisor
Deactivate all optional containers: Well, the only set of containers running inside that server is the AIO, and I neither added nor removed any. Its just plain AIO as it comes out of the box.
For faster previews, it is recommended to set up the preview generator app: I have not done that, and dont think I should have to, to be honest. right now, sync is my issue, not the previews (even though I can already tell that the previews are lagging far behind the actual sync).
Do not use Cloudflare: I donāt.
Have 1 vCore per active user: well, right now, I have 1 active user and 12vCores
Have 1GB RAM per active user: I have 48 GB RAM, of which 33 GB are shown as āavailableā in TOP
Have a fiber-optic connection to the outside with the highest available upload rate: WAN is symmetric GB fiber optic business internet line with static IP, and the LAN is a mixed 10GB/s and 1GB/s network using the Unifi products from Ubiquiti.
Have a good router that is capable of handling everything that you throw at it: I would say that I do have that. It is the older Ubiquiti USG-Pro-4 router. Using the same router and WinSCP between the exact same devices, the upload of the test data set takes less than 2 minutes. Only the sync with the Nextcloud Windows Client is still ongoing even after 3 days. So I would argue itās not the router.
Connect your server via Gbit LAN-cable and not Wi-fi to the router: Yep, itās all wired. No WiFi involved whatsoever.
Set up a local DNS-server: Yep, I have done that. And I can confirm that the DNS called from within my network resolves to the local server address, while the DNS called from outside my network resolves to my public static IP. But the test scenario I am currently running is 100% internal.
To test it, I made the modifications outlined in your link and dumped another 12GB files (~5000 files) into the Nextcloud folder and had the Desktop App sync it. Took about 10 minutes.
So basically, it was no issue with my environment, my network, or my configuration, but a bug that has been reported 1 year ago, and which still has not been fixed.
Nice! This issue got me also earlier when I put some git repositories in Nextcloud. The .git folders contain many little files and Nextcloud sync was choking too.
Funny thing: I dumped some MORE files into the sync folder and now itās slow again. The settings are still at what I set them based on the bug report.
But it seems the NextCloud App is artificially slowing the sync by tackling 100 files at a time, and then waiting forever for take the next 100.
So there must be yet another issue or config item with the Windows App.
Still far from the user experience of those Apps provided by Google or Microsoft, I am afraid. Too bad.