Is it just mis-configuration or is it normal that bulk file upload fails with numerous errors and folder structure change removes files?

Support intro

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can. :heart:

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • 30.0.2
  • Operating system and version (e.g., Ubuntu 24.04):
    • Ubuntu 22.04
  • Web server and version (e.g, Apache 2.4.25):
    • Nginx 1.27.2
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • Nginx 1.27.2
  • PHP version (e.g, 8.3):
    • 8.3.12
  • Is this the first time you’ve seen this error? (Yes / No):
    • No
  • When did this problem seem to first start?
    • When bulk uploading files
  • Installation method (e.g. AIO, NCP, Bare Metal/Archive, etc.)
    • Bare Metal
  • Are you using Cloudflare, mod_security, or similar? (Yes / No)
    • Cloudflare

Summary of the issue you are facing:

I am trying to migrate all my files from Microsoft Sharepoint to newly installed NextCloud instance. I hoped that just connecting the sharepoint folders as an external storage would help me to access all files from NextCloud, but apparently it is not. So gotta do this manually.

There’s 30 some GB image/video/PDF in the sharepoint. I initially tried to drag and dropped the local synced folder to Nextcloud’s web → files → group folder.

I got below error filling the entire screen

  • Error during upload: The server was unable to complete your request. If this happens again, please send the technical details below to the server administrator. More details can be found in the server log.

So I scaled down one by one. First started with a branch of the sharepoint folder, and eventually just 5+ files from a single folder.

When I try with nested folders, it only uploads folders, after spending hours. I guess all file uploads are failed. When I try only with files (not folders), it does upload files, but takes ages.

Then, when I move that folder to another location, say abc/bbc/cbc to abc/xyz/cbc, then all files in cbc folder are gone. It just disappears. I’ve also tried with external storages with S3, and even on the S3, the files were gone.

Log entries

The same error over and over, so I just picked a fraction of the full error log.

{"reqId":"f4c2OtZ46P1FghHngL9v","level":3,"time":"2024-12-03T00:39:50+00:00","remoteAddr":"192.168.88.1","user":"microsoft-xxxx","app":"no app in context","method":"PUT","url":"/remote.php/dav/files/microsoft-xxx/GIAI%2>
{"reqId":"f4c2OtZ46P1FghHngL9v","level":3,"time":"2024-12-03T00:39:50+00:00","remoteAddr":"192.168.88.1","user":"microsoft-xxxx","app":"no app in context","method":"PUT","url":"/remote.php/dav/files/microsoft-xxxx/GIAI%2>
{"reqId":"f4c2OtZ46P1FghHngL9v","level":3,"time":"2024-12-03T00:39:50+00:00","remoteAddr":"192.168.88.1","user":"microsoft-xxxx","app":"webdav","method":"PUT","url":"/remote.php/dav/files/microsoft-xxxx/GIAI%20Korea/The%>
{"reqId":"v4GnuK4HI23vHUt0OfSA","level":3,"time":"2024-12-03T00:39:54+00:00","remoteAddr":"192.168.88.1","user":"microsoft-xxxxg","app":"no app in context","method":"PUT","url":"/remote.php/dav/files/microsoft-xxxx/GIAI%2>
{"reqId":"v4GnuK4HI23vHUt0OfSA","level":3,"time":"2024-12-03T00:39:54+00:00","remoteAddr":"192.168.88.1","user":"microsoft-xxxx","app":"no app in context","method":"PUT","url":"/remote.php/dav/files/microsoft-xxxx/GIAI%2>
{"reqId":"v4GnuK4HI23vHUt0OfSA","level":3,"time":"2024-12-03T00:39:54+00:00","remoteAddr":"192.168.88.1","user":"microsoft-xxxx","app":"webdav","method":"PUT","url":"/remote.php/dav/files/microsoft-xxxx/GIAI%20Korea/The%>
{"reqId":"ZSfTyKXYbDJd21hj8sxi",

I just replaced my ID after microsoft to ‘xxxx’.

I’ve double checked in my Nginx configuration that I do have /.well-known/ location block set in both proxy and backend servers accordingly with NGINX configuration — Nextcloud latest Administration Manual latest documentation.

Also PHP, Redis configurations are also in line with the model configuration. In the Administration → Overview, I have no warning other than PostgreSQL version is too high (17.1).

Is this only for community version? Is the corporate version any different? Thought this is the most known open source replacement to Dropbox, but if it is not due to my mis-configuration, I should re-consider relying on Nextcloud for corporate use.

Bulk upload, bulk remove, folder move, and even folder copy emit the same error. It is not limited to local storage or external storage via S3. Haven’t tested out other external storage options, but it is too much error.

Would be great to know what causes this…

While it is overly too slow to upload files, I tried with more than one window for parallel uploading. I tried with 3 windows on a single browser, each pointing different folders.

About 50 files in each upload, and all got mixed to a single folder.

I don’t know what I did wrong in configuration, but I really hope that this is just because of mis-configuration.

I did another test on a fresh installed instance.

I can now confirm that it is due largely to S3.
Whether S3 is primary or external, it doesn’t really matter that it slows down the file processing. And the situation is a lot more aggrevated if it is an external storage.

For 250 MB 2000+ images w/ 4 layers nested folders, it took less than 10 mins for local storage, 24 mins for S3 as a primary storage, and 2 hours does not seem enough for S3 as an external storage.

Upload error occurs mostly for external storage. When S3 is the primary storage, I see the error only for a few times. For S3 as external storage, it covers the entire screen and keeps generated.

Image slideshow was also affected. When it is local, it was same as local image folders. At least I did not feel any noticeable delay. When S3 as a primary, I did have seen lag, but compromisable speed once it is cached. When S3 as an external storage, it is not really a slide show any more.

I’ve also checked if group folders can create an issue, but it seems not. The speed of bulk upload is almost nearly identical between personal and group storages. Group size doesn’t matter neither.

In short, at least for bulk upload, it is not recommended to use S3 as an external storage.

Just in case, my S3 is MinIO w/ 4 of 2TB NVMe on 4x4x4x4 PCIe 4.0 w/ 16 lanes. Two servers are running as an exact copy for load balancing. My NextCloud instances are also running on the same servers behind the proxy server. So, even if it uses S3, it is practically the same as reading from local drive, which is why I have never felt this much delay with other programs.

Let me summarize the speed differences as

  • Local: Nearly as fast as NVMe
  • S3 Primary: Somewhere between SSD and HDD
  • S3 External: As bad as floppy disk
1 Like