Uploads w. many files fail on Official VM - no error msg

[ncdocker]

Hi,

tldr

Uploads to NC 15 (latest) instance of Official VM image that have many files (see examples below) consistently fail. Usually no error from UI, nor in any logfile I can find (high verbosity in all logs). VM installed basically with defaults no extras, powerful server not doing anything else. I can successfully upload these test files to a commercial NC server (version unknown) on owndrive.com

No issues uploading single large files or other uploads that don’t have especially large number of individual files, all uploads tested over https.

*Details

So there’s no error usually, usually just the upload progress meter is stuck midway…once in a while I will get an “An unknown error has occurred” shown in the UI.

I can watch the apache access and redis logs things seem to be humming along requests are being processed then suddenly everything stops, it almost looks like redis stops a second before apache does…

There were a few things the redis log was warning about I tried changing such as increasing
net.core.somaxconn
net.ipv4.tcp_max_syn_backlog

and doing
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Tried increasing redis keep alives, and a few other redis parameters with no change.

php-fpm I upped these to quite high, no effect.
pm.max_children
pm.start_servers
pm.min_spare_servers
pm.max_spare_servers

In NCs config.php I tried changing
memcache.local

from redis to use APCu, no effect.

Also upped the ‘memcache.locking’ => ‘\OC\Memcache\Redis’, timeout from .5 to ‘timeout’ => 1.5 no effect.

I’ve stopped at this point because I’m just shooting in the dark now.

*Other info

Ubuntu 18.04 3ghz 8 core server 16 gigs mem, Raid 10 - doing nothing
VM imported to virtualbox with defaults, running headless, private network self signed https

Uploads, using either of extracted below to test:
https://openresty.org/download/openresty-1.13.6.2.tar.gz
https://download.nextcloud.com/server/releases/nextcloud-15.0.0.tar.bz2

Testing with both latest firefox and chrome, no extensions installed.

Logs checked:
NC Logging in web admin
PHP-FPM
Apache access & error
PHP (had to enable)
Redis (increased verbosity)
Postgres (enabled verbosity increased)

Thanks for any help!

[enoch85]

There’s a limit in the GUI of uploading 1000 MB, could that be the issue?

Use the client for syncing stuff, just an advise. The client chunks it into smaller pieces and there’s no limit whatsoever.

Hope it helps!

[ncdocker]

Thanks for the reply

@enoch85

There’s a limit in the GUI of uploading 1000 MB, could that be the issue?

No total size of upload is only 32 megs and uploading large files, or fewer files works great. As near as I can tell it has to do with uploading a large number of items (2762) in the case of https://openresty.org/download/openresty-1.13.6.2.tar.gz

@enoch85

Use the client for syncing stuff, just an advise. The client chunks it into smaller pieces and there’s no limit whatsoever.

Tried the client on Win 7 over lan, and Ubuntu 18.04 over wifi with same results (server is on same lan). Results being there are still errors, yet client seems to keep trying until eventually it succeeds… So I guess that’s better than a complete fail. See image for an example of the errors.

Incidentally we also reproduced this on the docker nextcloudpi instance, and I believe one developer reproduced it using one of the docker-compose nginx based examples.

So it looks like it’s basically a nextcloud bug of some type, not an issue with the VM image, I’ll go ahead an open an issue on GitHub for it.

At least in the meantime hopefully we will have success with the client despite the errors it spits.

thank you again

[enoch85]

Well, in the VM the default upload limit is 1000 MB in the WebGUI, so if you only have 32 MB then you either changed some value yourself, or it’s something else. Because not even PHP default is 32 MB.

You can try Googling how to increase the max upload file size in PHP and see if that helps.

[ncdocker]

Thanks for the reply, there’s no problem uploading large files, we’ve uploaded up to 800 meg files, it’s fast.

I’m saying there’s apparently a NC bug when uploading many files/folders at once, such as when one extracts the open resty example I’ve linked, and attempts to upload the extracted folder.

Not a bug with the VM though, I’ll report it on the NC github, it’s very easy to reproduce.

cheers

[enoch85]

Ah ok, great! Thanks for the clarification, and thanks for posting an issue report in Github!