Downloads from Nextcloud stalling (S3 Primary Storage)

Nextcloud version: 14.0.3
Operating system and version: Ubuntu 18.04
Apache version: 2.4.29
PHP version: 7.2

I had Nextcloud 13 installed on an AWS server running Ubuntu LAMP and using S3 for primary object storage. It worked fine.

I was restructuring my cloud instances and decided to use a LightSail instance to host my Nextcloud server. I first set it up as a LEMP config (still with Ubuntu OS). Everything looked good at first so I upgraded from Nextcloud 13 to 14. After that I found that Nextcloud downloads and uploads were stalling midway.

I troubleshot the issue for a week or so and then decided to try a fresh Ubuntu 18.04 instance with Apache2 (configuration listed above). The issue was not resolved. I’m having a lot of trouble figuring out what the issue is because I don’t see any pertinent entries in the Apache, php, or Nextcloud error logs.

I have tried downloading from S3 directly and it works fine. I have downloaded from apache directly and it works fine. The issue seems to be when downloading via Nextcloud.

Any hints on what to check or try?

Downloads halt after 20-40MB for a large file (this one is ~2GB but it happens with any file larger than 20-40MB). They do not error out immediately… the transfer just stops progressing. Small files like PDFs download just fine so I know that Nextcloud is actually connecting to my S3 bucket.

I set up aws cli on the Ubuntu LightSail instance that hosts Nextcloud and I’m able to sync large files from S3 to the Ubuntu instance successfully.

I don’t see any errors being generated when the download stalls in the nextcloud.log, php_errors.log or apache2 error.log.

From what I’ve read, when I am trying to transfer to or from S3 with Nextcloud it will actually use the Ubuntu AWS instance that hosts Nextcloud as a buffer (either upstream or downstream) to maintain encryption. Is this done with php? Is there a way to make the process more verbose so I can see if there’s a failure that’s not being reported?

Currently downloads from Nextcloud will start at ~4MB/s and then drop to ~1KB/s. once ~30MB has been reached. Previously with Nginx I would get timeouts but now with my current Apache2 configuration it appears that the transfer is plodding along at a super slow pace (I have a fiber connection to my office so ~4MB/s, continuous, would be expected during normal operation).

Is there a way to diagnose what is restricting the transfer speed?

The download that was running really really slow just died 40+ minutes after it started. The only thing I see in the logs that may be pertinent is the Apache2 access.log:

__MY_CLIENT_IP_REMOVED__ - - [16/Oct/2018:16:58:32 +0000] "GET /remote.php/webdav/LARGE_FILE_NAME.zip HTTP/1.1" 200 32879827 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"
::1 - - [16/Oct/2018:18:17:43 +0000] "OPTIONS * HTTP/1.0" 200 126 "-" "Apache/2.4.29 (Ubuntu) OpenSSL/1.1.0g (internal dummy connection)"
__MY_CLIENT_IP_REMOVED__ - - [16/Oct/2018:18:17:54 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 1078 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"
__MY_CLIENT_IP_REMOVED__ - - [16/Oct/2018:18:18:01 +0000] "GET /csrftoken HTTP/1.1" 200 1333 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:61.0) Gecko/20100101 Firefox/61.0"

^ That was added to the log around when the download failed. I’m not sure if there’s enough info there to indicate anything in particular.

I did more testing and found that with my current configuration that uploads seem to work most (if not all) of the time.

The issue is downloads from Nextcloud. I’m at a loss for what could be causing this.

I tried to trace what’s going on in the php scripts and it looks like Sabre DAV is used when downloading a file from Nextcloud when using S3 primary storage. Is there a good way to track what’s going on with the Sabre DAV server once it executes?

Did you find a solution yet? I have exactly the same problem with the same configuration.

No, unfortunately, I have not. I tried troubleshooting on my own for a few days but without some outside input I’m at a loss. Perhaps someone familiar with this part of the application can chime in with some suggestions on things for me to check.

I mounted local directories via external storage and moved my files from S3 to the EBS volume. Downloads are working that way. The problem is definitely with the S3 implementation of Nextcloud.