Big Uploads Fail without any Error Message

swiftbird07 · May 14, 2024, 4:45pm

Hello, I have an issue where I can’t upload files bigger than ~1GB.

Nextcloud version: Nextcloud Hub 8 (29.0.0)
Operating system and version : Docker: Linux 5.15.0-105-generic x86_64
Apache or nginx version: Apache/2.4.57 (Debian)
PHP version: 8.2.13

The issue you are facing:

Can’t upload any big files via web UI.

Is this the first time you’ve seen this error? : Yes

Steps to replicate it:

Use the provided docker compose file to create the Nextcloud instance
Upload a file bigger than 1GB

Docker compose file:

...
  nextcloud_app:
    container_name: nextcloud_app
    image: nextcloud
    restart: always
    ports:
      - 8080:80
    links:
      - nextcloud_db
    extra_hosts:
      - "REDACTED:10.20.1.36"
    volumes:
      - nextcloud_html:/var/www/html
      - /mnt/nas/REDACTED/NEXTCLOUD_DATA:/var/www/html/data
      - nextcloud_apache:/etc/apache2
    environment:
      - MYSQL_PASSWORD=REDACTED
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_HOST=nextcloud_db
      - NEXTCLOUD_DATA_DIR=/var/www/html/data
      - NEXTCLOUD_TRUSTED_DOMAINS=nextcloud.nas.REDACTED
      #- REDIS_HOST=nextcloud_redis
      #- REDIS_HOST_PASSWORD=REDACTED
      #- REDIS_HOST_PORT=6379
      - PHP_MEMORY_LIMIT=10G
      - PHP_UPLOAD_LIMIT=10G
      - APACHE_BODY_LIMIT=0
      - APACHE_DISABLE_REWRITE_IP=1
...

As you can see I already did what was recommended regarding setting the upload limit, as well as setting the Apache setting LimitRequestBody to 0 in the .htaccess file.

Basically the upload always fails for big files, but how that fail looks like is different:

Either after a time the upload bar is getting red and the browser / logs report 409 for the following PUTs.

Here are the logs for that scenario:

nextcloud_app    | 192.168.48.1 - - [14/May/2024:16:16:24 +0000] "GET /apps/logreader/api/poll?lastReqId=hreYyVJyjrEKwKwZ6ljN HTTP/1.1" 200 793 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0"
nextcloud_app    | 192.168.48.1 - - [14/May/2024:16:16:34 +0000] "GET /apps/logreader/api/poll?lastReqId=hreYyVJyjrEKwKwZ6ljN HTTP/1.1" 200 793 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0"
nextcloud_app    | 192.168.48.1 - - [14/May/2024:16:16:44 +0000] "GET /apps/logreader/api/poll?lastReqId=hreYyVJyjrEKwKwZ6ljN HTTP/1.1" 200 793 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0"
nextcloud_app    | 192.168.48.1 - - [14/May/2024:16:16:48 +0000] "PUT /ocs/v2.php/apps/user_status/api/v1/heartbeat?format=json HTTP/1.1" 200 965 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0"
nextcloud_app    | 192.168.48.1 - - [14/May/2024:16:16:51 +0000] "DELETE /remote.php/dav/uploads/db9c028c79eebafcc3e96ac173a54c6feb2b4ef96d795dad6eb4721c9bf36159/web-file-upload-e7215fef304d2239 HTTP/1.1" 204 550 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0"
nextcloud_app    | 192.168.48.1 - - [14/May/2024:16:16:51 +0000] "PUT /remote.php/dav/uploads/db9c028c79eebafcc3e96ac173a54c6feb2b4ef96d795dad6eb4721c9bf36159/web-file-upload-e7215fef304d2239/51 HTTP/1.1" 409 723 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0"
nextcloud_app    | 192.168.48.1 - - [14/May/2024:16:16:51 +0000] "PUT /remote.php/dav/uploads/db9c028c79eebafcc3e96ac173a54c6feb2b4ef96d795dad6eb4721c9bf36159/web-file-upload-e7215fef304d2239/52 HTTP/1.1" 409 723 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0"

Inspecting the response of the PUT that fail in the browser I see the following response:

<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\Conflict</s:exception>
  <s:message/>
</d:error>

Or - and this is the most frustrating scenario: The upload is seemingly successful and then it just isn’t uploaded. No error or anything, the file will just not show up after the upload finishes.

The only error message I was able to recover in this scenario is the one from the nextcloud log stating that the deletion of the (I guess temporary-) upload file failed because there are still files in it:

{"reqId":"hreYyVJyjrEKwKwZ6ljN","level":3,"time":"2024-05-14T15:57:33+00:00","remoteAddr":"192.168.48.1","user":"db9c028c79eebafcc3e96ac173a54c6feb2b4ef96d795dad6eb4721c9bf36159","app":"PHP","method":"DELETE","url":"/remote.php/dav/uploads/db9c028c79eebafcc3e96ac173a54c6feb2b4ef96d795dad6eb4721c9bf36159/web-file-upload-fec56b7f214a44e5","message":"rmdir(/var/www/html/data/db9c028c79eebafcc3e96ac173a54c6feb2b4ef96d795dad6eb4721c9bf36159/uploads/web-file-upload-fec56b7f214a44e5): Directory not empty at /var/www/html/lib/private/Files/Storage/Local.php#157","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) Gecko/20100101 Firefox/125.0","version":"28.0.1.1","data":{"app":"PHP"}}

Looking at that upload directory I could indeed confirm that there are still hundreds of 10MB files in it. So I guess that the issue is something about Nextcloud not being able to move these files from the upload directory to the target directory for some reason, tho I could confirm that the permissions seem to be correct (and small uploads work anyway so that is unlikely the problem):

root@824ea620cf85:/var/www/html# ls -la data/db9c028c79eebafcc3e96ac173a54c6feb2b4ef96d795dad6eb4721c9bf36159/files/Temp\ Share/
total 682072
drwxr-xr-x 3 www-data www-data      4096 May 14 14:35  .
drwxr-xr-x 9 www-data www-data      4096 May  4 20:41  ..
-rw-r--r-- 1 www-data www-data  13900568 May 14 13:34 '2024-04-09 v2_watermarked-2-1-1.pdf'

I would appreciate any help.

jtr · May 14, 2024, 5:45pm

Please upload to a supported maintenance release - in your case 28.0.5.

There were browser upload bugs addressed with large files already awhile ago.

The rest of the changes regarding PHP and Apache parameters aren’t relevant since you’re using the official Web client (which uses chunking).

swiftbird07 · May 14, 2024, 7:30pm

Ok I have updated to 29.0 the issue still persists.

jtr · May 14, 2024, 7:40pm

Please post your actual running Nextcloud config (occ config:list system).

So I guess that the issue is something about Nextcloud not being able to move these files from the upload directory to the target directory for some reason

What’s the underlying filesystem type? I.e. What actually is /mnt/nas/REDACTED/NEXTCLOUD_DATA? Based on the file path it appears it may not be a local file, but some sort of network based filesystem.

swiftbird07 · May 14, 2024, 7:54pm

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "upgrade.disable-web": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.nas.REDACTED",
            "nextcloud.REDACTED"
        ],
        "overwrite.cli.url": "http:\/\/nextcloud.nas.REDACTED",
        "overwriteprotocol": "https",
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "29.0.0.19",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "allow_local_remote_servers": true,
        "app_install_overwrite": [
            "memories"
        ],
        "memories.exiftool": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/exiftool-amd64-glibc",
        "memories.vod.path": "\/var\/www\/html\/custom_apps\/memories\/bin-ext\/go-vod-amd64",
        "memories.vod.ffmpeg": "\/usr\/bin\/ffmpeg",
        "memories.vod.ffprobe": "\/usr\/bin\/ffprobe",
        "enabledPreviewProviders": [
            "OC\\Preview\\Image",
            "OC\\Preview\\HEIC",
            "OC\\Preview\\TIFF",
            "OC\\Preview\\Movie"
        ],
        "loglevel": 2,
        "maintenance": false
    }
}

/mnt/nas/… is a normal ext4 HDD VM mount no network storage or anything like that:

Filesystem     Type  1K-blocks       Used  Available Use% Mounted on
/dev/sdb1      ext4 6435258508 3287552508 2829604808  54% /mnt/nas

Mornsgrans · May 14, 2024, 8:36pm

What are the permissions of /mnt/nas?

swiftbird07 · May 14, 2024, 8:55pm

/mnt/nas itself is root www-data, tho whats mounted inside the container (/mnt/nas/REDACTED/NEXTCLOUD_DATA) is www-data www-data:

root@hdocker ~# ls -la /mnt/nas/REDACTED/NEXTCLOUD_DATA
total 82860
drwxrwx— 8 www-data www-data 4096 May 14 17:49 .
drwxrwx— 9 root www-data 4096 May 14 17:24 …

swiftbird07 · May 16, 2024, 2:05pm

Anyone? It’s a bummer that I can’t use Nextcloud as an actual cloud

mrplexcloud · May 16, 2024, 4:01pm

i had same issues i was able to solve by updating these settings in Truenas scale app section for nextcloud

Works perfectly now

jtr · May 16, 2024, 4:11pm

Any idea why the timestamps are so different on what is presumably the same upload transaction here?

According to this there was a DELETE request at 15:57:33 and the other says 16:16:51.

jtr · May 16, 2024, 4:14pm

Are you using a reverse proxy? Where is HTTPS being terminated? Just trying to understand your topology better.

swiftbird07 · May 16, 2024, 5:25pm

I don’t use Nexcloud on a TrueNAS system, so that does not apply to me, tho thanks for trying to help me.

swiftbird07 · May 16, 2024, 5:25pm

These two describe two different scenarios. These happen independent from each other (for one upload it could be the red bar for the other one the “silent error”, I don’t know why it that happens). That’s why these are different timestamps.

swiftbird07 · May 16, 2024, 5:26pm

Yes I have an Nginx Proxy Manager instance that proxies the connections to Nextcloud so that I don’t have to deal with certs on the Nextcloud instance itself. I have already set the needed configs so the file upload is not limited as recommended by other users:

proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_max_temp_file_size 16384m;
client_max_body_size 0;

so that shouldn’t be the problem.

jtr · May 16, 2024, 6:04pm

Oh I was looking at the wrong random string. I thought I saw them matching, but your username is random so I transposed them. Oops.

Perhaps you can try step back for troubleshooting purposes and simplify things by trying two scenarios simpler:

bring up a generic stock Nextcloud app container that doesn’t use any special volumes; point NPM at it and test uploads
if still seeing problems, test uploads directly to the app container w/o NPM on front

What other changes did you make to your Apache config? I see you mounted a config directory for it. APACHE_BODY_LIMIT=0 already covers the LimitRequestBody (which also shouldn’t matter since chunking is being used here).

swiftbird07 · May 17, 2024, 11:43am

Ok I just tried it with a brand new Nextcloud container and it doesn’t work here either tho the log message is different now:

Server.php","line":321,"function":"start","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/lib/Server.php","line":374,"function":"exec","class":"Sabre\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/apps/dav/appinfo/v2/remote.php","line":35,"function":"exec","class":"OCA\\DAV\\Server","type":"->","args":[]},{"file":"/var/www/html/remote.php","line":172,"args":["/var/www/html/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/html/apps/dav/lib/Connector/Sabre/File.php","Line":505,"message":"Could not open file","exception":{},"CustomMessage":"Could not open file"}}

After that I tried connecting directly to that instance and it worked without a problem! What do you think can cause that?

Trying to do the same to my production nextcloud does not work tho, as it will not let me login (after I enter the admin credentials it just doesn’t do anything), so I can’t test it there.

What other changes did you make to your Apache config? I see you mounted a config directory for it. APACHE_BODY_LIMIT=0 already covers the LimitRequestBody (which also shouldn’t matter since chunking is being used here).

I just added LimitRequestBody=0 to the .htaccess file as I didn’t know at that time that APACHE_BODY_LIMIT=0 was a thing (removed it since so it’s back to the original).

swiftbird07 · May 17, 2024, 11:59am

I was going through the nginx error logs and found this:

2024/05/17 11:11:51 [error] 1237#1237: *26529 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.178.151, server: nextcloudtest.nas.REDACTED, request: "MOVE /remote.php/dav/uploads/admin/web-file-upload-ceb709ad3d5b1c9c/.file HTTP/1.1", upstream: "http://10.27.1.3:8081/remote.php/dav/uploads/admin/web-file-upload-ceb709ad3d5b1c9c/.file", host: "nextcloudtest.nas.REDACTED"

could that be it? A timeout trying to move the file to its destination?

swiftbird07 · May 17, 2024, 12:13pm

To answer my own question: Yes that was the problem.

I had to set the proxy read timeout for nginx to a higher value according to this post 504 Gateway timeout for large file uploads - #7 by GR4 and now everything works!

Thank you @jtr for spending all the time helping me troubleshoot this!

system · May 25, 2024, 12:13pm

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.