I think this is a root cause: when something being saved on nextcloud external storage and this is not local folder, it could be downloaded to localhost first and then given to the user.
I do not have S3, but tried with WebDav and for me download started immediately: when I hit download, it starts like this External Webdav → Nextcloud Server → Client. You can see External Webdav dowload start as a green popup on external router. But you can’t see transmitted, because it was transmitted to the user via the Server in LAN and not external router.
Blue Arrow starting download on client causing immediately download from the webdav.
Red Arrow. I stop download in client.
Orange Arrow. You can see that basically download to the Nextcloud Server was not stopped with client, but seems it goes to cache my big file from the external storage.
Blue Arrow 2. I start download on a client again and you can see this does not cause download from the external storage. And Sync speed was higher then my external (internet) connection.
Orange Arrow 2. Seems Nextcloud did not cache whole file and as soon as client requested new chunks, it stars download new peaces.
Red Arrow 2. I stop download on a client, but you can see that Nextcloud Server did not stop to download it from the external storage.
At the end I believe you have bottleneck ether in a cache, or in a S3 Bucket, or in between of them (e.g. internet connection).
Can we try to trace this issue?
Try to hit download few times and check logs of Apache2 and e.g. htop what is hitting your CPU when you start downloading.
…
The access.log is consistent with the one I reported in the main body of this thread at the top.
That is, propfind and get requests for the requested file for download.
Nothing appears at request time in the error.log.
My use of the external S3 storage in this issue was purely for test purposes, to see if behavior was consistent with what I’m experiencing when trying to download a file from local storage.
Are you suggesting that I entirely remove external storages from NC to see if problem is linked to these external storages ?
Hi @gas85
Thanks for the suggestion.
And sorry for the delay in my answer, couldn’t run tests earlier.
So I’ve tried to trace the process and got this thread of answers from the process :
0x00007fdff0868654 in ?? () from target:/usr/lib/apache2/modules/mod_http2.so
(gdb) bt #0 0x00007fdff0868654 in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #1 0x00007fdff0884fa8 in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #2 0x00007fdff08853dd in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #3 0x0000559fc6482221 in ap_content_length_filter () #4 0x00007fdfef72b09b in ?? () from target:/usr/lib/apache2/modules/mod_proxy_fcgi.so #5 0x00007fdfef72c449 in ?? () from target:/usr/lib/apache2/modules/mod_proxy_fcgi.so #6 0x00007fdfef93abec in proxy_run_scheme_handler () from target:/usr/lib/apache2/modules/mod_proxy.so #7 0x00007fdfef93b9f9 in ?? () from target:/usr/lib/apache2/modules/mod_proxy.so #8 0x0000559fc6499910 in ap_run_handler () #9 0x0000559fc6499e8d in ap_invoke_handler () #10 0x0000559fc64b1d8b in ap_process_async_request () #11 0x0000559fc64b1f6e in ap_process_request () #12 0x00007fdff08849a3 in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #13 0x0000559fc64a3280 in ap_run_process_connection () #14 0x00007fdff0885b8b in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #15 0x00007fdff08893a6 in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #16 0x00007fdff35686db in start_thread (arg=0x7fdfdc5b4700) at pthread_create.c:463 #17 0x00007fdff329188f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95
Triggering a download and then immediately cancelling it (just closing web browser of NC), still triggers the cpu spike of the same process for a few minutes (just no i/o activity because no download). With a read activity of the data drive. As if file was copied/loaded somewhere (temp location? Redis ? some cache ?).
Graph :
OK.
Sooooo…
Having seen a lot of mod_http2 in the process thread… I tried to disable http2
(a2dismod http2, and then a service restart for apache)
And BOOM : Downloads start immediatlely !!
(Also transfer rates were incomparably higher…)
So, what could be the problem here ? … getting closer though…
I have a feeling http2 is not the cause, just a symptom.
Could this issue be linked with apache MPM modules configuration ?
Thank you for your answer !
Am reviewing my entire apache configuration. Will report here if anything.
Thank you for your very interesting documentation.
Have a good day…
Did you get a chance to find out the root cause?
I have a NCP instance in a VPS and my upload is 10~15Mbps… when I disabled HTTP2 in Apache2 it jumps to 70~100Mbps.
Hi @Edson_Rodrigues
Unfortunately I never got to the last word on this one.
Having suffered a critical crash sometime after, I had to reinstall from scratch (I only backup the data and closely document all my install steps). (Which also gave me the opportunity to beef up the hardware a little).
But now, newer versions of NC, php, and etc all components, impossible to compare.
It turns out that after installation, the problem never reappeared…
So, in my current config, I’m running apache event mode and http2_module enabled.
Not a single issue like the one mentionned before… Sorry I can’t be of more help.
