Help Finding Upload Bottleneck

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • 31.0.4
  • Operating system and version (e.g., Ubuntu 24.04):
    • Debian 12 inside Proxmox LXC
  • Web server and version (e.g, Apache 2.4.25):
    • nginx 1.22.1
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • nginx proxy manager
  • PHP version (e.g, 8.3):
    • 8.2.28
  • Is this the first time you’ve seen this error? (Yes / No):
    • No
  • When did this problem seem to first start?
    • From the moment the server was setup
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • Bare Metal on LXC container in Proxmox
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • No

Summary of the issue you are facing:

[Uploads Bottlenecked to 6MB/s]

Log entries

N/A (All it has is warnings related to uploads I started and cancelled)

Configuration

Nextcloud

The output of occ config:list system or similar is best, but, if not possible, the contents of your config.php file from /path/to/nextcloud is fine (make sure to remove any identifiable information!):

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "cloud.encodedbird.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "31.0.4.1",
        "overwritehost": "cloud.encodedbird.com",
        "overwriteprotocol": "https",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "overwritecondaddr": "^10\\.0\\.0\\.2$",
        "installed": true,
        "overwrite.cli.url": "https:\/\/cloud.encodedbird.com",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 0
        },
        "default_phone_region": "US",
        "skeletondirectory": "",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_smtpauth": true,
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "maintenance_window_start": 1,
        "app_install_overwrite": []
    }
}

/etc/nginx/sites-available/nextcloud.conf

php-fpm.conf

All comments except including pool.d/www.conf

php.ini

pool.d/www.conf

Nginx Proxy Manager custom settings

client_body_buffer_size 2m;
proxy_read_timeout 86400s;
client_max_body_size 0;
proxy_buffering off;
proxy_request_buffering off;

Apps

Enabled:
  - activity: 4.0.0
  - app_api: 5.0.2
  - appointments: 2.4.3
  - bruteforcesettings: 4.0.0
  - calendar: 5.2.2
  - circles: 31.0.0
  - cloud_federation_api: 1.14.0
  - comments: 1.21.0
  - contacts: 7.0.6
  - contactsinteraction: 1.12.0
  - dashboard: 7.11.0
  - dav: 1.33.0
  - deck: 1.15.1
  - federatedfilesharing: 1.21.0
  - federation: 1.21.0
  - files: 2.3.1
  - files_downloadlimit: 4.0.0
  - files_pdfviewer: 4.0.0
  - files_reminders: 1.4.0
  - files_sharing: 1.23.1
  - files_trashbin: 1.21.0
  - files_versions: 1.24.0
  - firstrunwizard: 4.0.0
  - forms: 5.1.0
  - logreader: 4.0.0
  - lookup_server_connector: 1.19.0
  - nextcloud_announcements: 3.0.0
  - notes: 4.12.0
  - notifications: 4.0.0
  - oauth2: 1.19.1
  - password_policy: 3.0.0
  - photos: 4.0.0-dev.1
  - privacy: 3.0.0
  - profile: 1.0.0
  - provisioning_api: 1.21.0
  - recommendations: 4.0.0
  - related_resources: 2.0.0
  - serverinfo: 3.0.0
  - settings: 1.14.0
  - sharebymail: 1.21.0
  - support: 3.0.0
  - survey_client: 3.0.0
  - systemtags: 1.21.1
  - tasks: 0.16.1
  - text: 5.0.0
  - theming: 2.6.1
  - twofactor_backupcodes: 1.20.0
  - updatenotification: 1.21.0
  - user_status: 1.11.0
  - viewer: 4.0.0
  - weather_status: 1.11.0
  - webhook_listeners: 1.2.0
  - workflowengine: 2.13.0
Disabled:
  - admin_audit: 1.21.0
  - encryption: 2.19.0
  - files_external: 1.23.0
  - suspicious_login: 9.0.1
  - twofactor_nextcloud_notification: 5.0.0
  - twofactor_totp: 13.0.0-dev.0
  - user_ldap: 1.22.0

Description

I am trying to get my nextcloud setup tuned with nginx on debian.
I confirmed that my bandwidth is not a factor and you can see speedtest results at Speedtest by Ookla - The Global Broadband Speed Test.

I have it tuned so that no warnings appear in the administration page. When doing downloads, I have it working at a clean 100MB/s.
When doing uploads via browser, it is bottlenecked to 6-8MB/s.
When doing uploads via WebDAV, it performs around the same.

I do not believe this is a duplicate of other tickets because most others I read demonstrated little understanding of the web server configuration and didn’t provide complete config information.

I can’t figure out where the bottleneck is

Things I tried already

  • Increasing php.ini memory_limit to 1G or 2G. This had no effect.
  • Enabling/Disabling http/2 in Nginx Proxy Manager. I find enabling it improves webdav performance, but otherwise it has no impact on max upload speed.
  • Testing Chunked vs Unchunked using a public upload link in incognito mode.
  • Tuning linux kernel TCP settings in hypervisor. This did help to achieve the high download speeds.

Download it is the same speed?

During upload, with top and iotop, can you see which process uses all the resources?

If you haven’t touched the database configuration yet, cache optimization at this point can help a lot (not sure for uploads…)

You are right I mentioned nothing about disk usage. CPU and ram usage are typically very low, as if only a single PHP child is handling the upload (something like 1-3%).
Disk usage per iotop never shows more than 6% on a single disk. Its a RAIDZ2 ZFS array of 4 of them.

As I mentioned, download gets a good 100MB/s (not megabits, megabytes). So its not the same speed. I can’t imagine how you would meaningfully cache an unknown upload.

I have done little on the DB end. I used the default mariadb per the general recommendations, and I have changed none of its settings outside of creating the database and setting credentials. This was a fresh nextcloud install so it should have been able to set indices and keys the way it liked. The DB is not used for anything else. Could the DB bottleneck uploads?

you mentioned there is not much load on the system so maybe very valuable - but still worth mentioning official docs (maybe most important advice - use http2):

please look at topics tagged performance as similar discussions come up from time to time.

the other tweak important for files upload is chunking - files exceeding specific size are split into chunks, uploaded in parts and connected again on the server side - obviously chunking add overhead and on fast and reliable connection it could be beneficial to increase chunk sizes. AFAIK there was a change shortly but maybe you want go further in you installation. Another option I remember is bulk upload

Normally, this can be critical with many small files. Besides of that, db caching would help for other usage to reduce the load on the db.
wwe’s tips with the chunk sizes is a good pointer.

Just to be sure, if you write directly on your disk, you get more speed? Not that you have problems with your disks (if used in an array, that it doesn’t sync in the background, or one disk is very low speed already). Or test it with the whole network, upload large file via sftp.

I implement http2 at the proxy manager level, but like I said it doesn’t really have an impact on uploads much.
I should mention my test isn’t a bunch of small files, but rather one big file >10GB. For many small files I was never really worried about measuring it.

My disk absolutely achieves greater speeds, around 54MB/s when I was testing.

I was told that public file drop links do not chunk at all when uploads are performed by unauthenticated users. I didn’t know about the bulk upload API, but I will test that on Thursday and report back how it performs.