Problem uploading large files to Nextcloud with S3 as primary storage

ℹ️ Support
, ,
1

Hi everyone,

I’m facing a serious issue with my Nextcloud server and I’m looking for help or ideas:

  • Nextcloud version: 31.0.6.2
  • Server specs: 24GB RAM, 4 CPU cores
  • Primary storage: S3-compatible storage (Wasabi or others)
  • Environment:
    • PHP: 8.1
    • Web server: Apache 2.4
    • Database: MySQL 10.11.13
  • PHP configuration:
    • upload_max_filesize, post_max_size, and memory_limit tested with 20GB, 50GB, even 1TB.
    • max_execution_time and max_input_time set to very high values.

What happens:

  • Uploading small files (<1GB) works perfectly.
  • Uploading large files (>5GB) always fails with errors like “Unable to read from stream”, and the upload never completes, regardless of PHP limits.
  • The S3 bucket is correctly configured and works fine with other clients (e.g., AWS CLI, s3cmd), but not with Nextcloud.

I have already tried adjusting Nextcloud’s objectstore options:

  • uploadPartSize set to 500MB
  • concurrency tried with values like 1, 2, and 5
  • increased timeouts

Nothing solves it. Nextcloud starts the upload, but then it fails with stream read errors.

My question:
:backhand_index_pointing_right: Is there any configuration in Nextcloud, PHP, Apache, or the database that could still be limiting large file uploads?
:backhand_index_pointing_right: Has anyone successfully used Nextcloud with S3 (Wasabi or others) as the primary storage and been able to upload files larger than 5GB or 10GB without errors?

Any advice or shared experiences would be greatly appreciated! :folded_hands:

2

How are you uploading? Via the Web UI? Or one of the other clients? Via some 3rd-party client?

Also, are you uploading as a logged in user or via a share?

Please post the entire raw log entry.

Also your Nextcloud config.

Basically details and as described in the support template. :slight_smile:

upload_max_filesize, post_max_size, and memory_limit tested with 20GB, 50GB, even 1TB.

Most of these aren’t relevant (well unless they’re super tiny) unless you’re doing anonymous uploads or using third-party clients lacking chunking support.

Is there any configuration in Nextcloud, PHP, Apache, or the database that could still be limiting large file uploads?

The biggest that comes to mind is the requirement for having a sane memcache.distributed setup (Redis) for chunked upload streaming.

Has anyone successfully used Nextcloud with S3 (Wasabi or others) as the primary storage and been able to upload files larger than 5GB or 10GB without errors?

Yes.

3

I used the web interface, and the upload sends data to S3, but at the end the file does not appear in Nextcloud. However, I tested the same file with the Nextcloud Desktop Client, and it worked normally — the file uploaded and appeared correctly.

6

look at this Fail to upload to Cloudflare R2 for files larger than 50MB and/or chunked files topic - maybe it’s worth to double check if you config in fact applies (using occ to list config is always good idea).

Maybe it’s worth trying out all available config options especially following sound promising:

timeout defaults to 15
uploadPartSize defaults to 524288000
putSizeLimit defaults to 104857600
useMultipartCopy defaults to true
copySizeLimit defaults to 5242880000

Please review similar bug reports

and definitely increase log level and share log file while uploading