Well, that explains why the “waiting”/downloading times are close - it’s the same process done twice…
Could be.
Or maybe this is done with all files, just not noticeable on smaller ones…
I have never used S3 (type) storage but this workflow is not unheard of.
Consider 7zip.
If you open a (big) zipped file/folder and drag the file from the 7zip window to a non-C: location, it will first copy it into a temp folder on the system drive and after that move it to destination…
But I’d expect this to be configurable…
EDIT:
Do you have the S3 storage encrypted?
Then the first “waiting” step could be decryption…
This article (at the end) talks about /tmp folders when uploading
Thank you Henry.
This is indeed a very interesting documentation !
It doesn’t however fit my use case, my S3 external storage, is not -and will never be- my primary storage (also it is not encrypted). Primary storage will remain internal HDD. And since my problem is existing for this storage, solution will not come from the S3 external storage configuration.
I’m really very grateful for the time you have spent looking at my issue and configuration.
Problem unfortunately is still there, and my research of a solution ongoing !..
I think “primary storage” here means the location of the data, not the OS Nextcloud runs on.
I have multiple instances of NC and always run the OS+NC on a mirrored SSD (50-100GB) and the data folder on HDD. I believe this is a normal setup…
There are few old but ongoing “S3 is slow on Nextcloud” discussions on GitHub…
Yes I agreee with you.
My setup is :
sda/ is Ubuntuserver sysdrive, hosted NC instance in /var/www/nextcloud
sdb/ is NC data drive (4To)
sdc/ is a backup drive mirror of sdb/ copied everynight from sdb with rclone sync through a cron job.
S3 is extra external storage for archives.
I have no particular problems with S3.
My problem is a problem because I want to be able to send share links for specific files (from main data drive sdb/) to specific people, and these people of course don’t understand why their download takes so long to start…
So right now I CAN’T share files because of this issue …
Wow ! Long shot indeed. Thank you again Henry for looking into this.
I’m not running an arm architecture (not RK3399 cpu, not Rpi).
My machine is a Dell x86 machine with 82579LM chip for network.
But I’ll have a deeper (and very curious) look into this … Thanks
I think this is a root cause: when something being saved on nextcloud external storage and this is not local folder, it could be downloaded to localhost first and then given to the user.
I do not have S3, but tried with WebDav and for me download started immediately: when I hit download, it starts like this External Webdav → Nextcloud Server → Client. You can see External Webdav dowload start as a green popup on external router. But you can’t see transmitted, because it was transmitted to the user via the Server in LAN and not external router.
Blue Arrow starting download on client causing immediately download from the webdav.
Red Arrow. I stop download in client.
Orange Arrow. You can see that basically download to the Nextcloud Server was not stopped with client, but seems it goes to cache my big file from the external storage.
Blue Arrow 2. I start download on a client again and you can see this does not cause download from the external storage. And Sync speed was higher then my external (internet) connection.
Orange Arrow 2. Seems Nextcloud did not cache whole file and as soon as client requested new chunks, it stars download new peaces.
Red Arrow 2. I stop download on a client, but you can see that Nextcloud Server did not stop to download it from the external storage.
At the end I believe you have bottleneck ether in a cache, or in a S3 Bucket, or in between of them (e.g. internet connection).
Can we try to trace this issue?
Try to hit download few times and check logs of Apache2 and e.g. htop what is hitting your CPU when you start downloading.
…
The access.log is consistent with the one I reported in the main body of this thread at the top.
That is, propfind and get requests for the requested file for download.
Nothing appears at request time in the error.log.
My use of the external S3 storage in this issue was purely for test purposes, to see if behavior was consistent with what I’m experiencing when trying to download a file from local storage.
Are you suggesting that I entirely remove external storages from NC to see if problem is linked to these external storages ?
Hi @gas85
Thanks for the suggestion.
And sorry for the delay in my answer, couldn’t run tests earlier.
So I’ve tried to trace the process and got this thread of answers from the process :
0x00007fdff0868654 in ?? () from target:/usr/lib/apache2/modules/mod_http2.so
(gdb) bt #0 0x00007fdff0868654 in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #1 0x00007fdff0884fa8 in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #2 0x00007fdff08853dd in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #3 0x0000559fc6482221 in ap_content_length_filter () #4 0x00007fdfef72b09b in ?? () from target:/usr/lib/apache2/modules/mod_proxy_fcgi.so #5 0x00007fdfef72c449 in ?? () from target:/usr/lib/apache2/modules/mod_proxy_fcgi.so #6 0x00007fdfef93abec in proxy_run_scheme_handler () from target:/usr/lib/apache2/modules/mod_proxy.so #7 0x00007fdfef93b9f9 in ?? () from target:/usr/lib/apache2/modules/mod_proxy.so #8 0x0000559fc6499910 in ap_run_handler () #9 0x0000559fc6499e8d in ap_invoke_handler () #10 0x0000559fc64b1d8b in ap_process_async_request () #11 0x0000559fc64b1f6e in ap_process_request () #12 0x00007fdff08849a3 in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #13 0x0000559fc64a3280 in ap_run_process_connection () #14 0x00007fdff0885b8b in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #15 0x00007fdff08893a6 in ?? () from target:/usr/lib/apache2/modules/mod_http2.so #16 0x00007fdff35686db in start_thread (arg=0x7fdfdc5b4700) at pthread_create.c:463 #17 0x00007fdff329188f in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:95
Triggering a download and then immediately cancelling it (just closing web browser of NC), still triggers the cpu spike of the same process for a few minutes (just no i/o activity because no download). With a read activity of the data drive. As if file was copied/loaded somewhere (temp location? Redis ? some cache ?).
Graph :
OK.
Sooooo…
Having seen a lot of mod_http2 in the process thread… I tried to disable http2
(a2dismod http2, and then a service restart for apache)
And BOOM : Downloads start immediatlely !!
(Also transfer rates were incomparably higher…)
So, what could be the problem here ? … getting closer though…
I have a feeling http2 is not the cause, just a symptom.
Could this issue be linked with apache MPM modules configuration ?
Thank you for your answer !
Am reviewing my entire apache configuration. Will report here if anything.
Thank you for your very interesting documentation.
Have a good day…
Did you get a chance to find out the root cause?
I have a NCP instance in a VPS and my upload is 10~15Mbps… when I disabled HTTP2 in Apache2 it jumps to 70~100Mbps.
