Nextcloud works except LARGE federated shares. cURL 500 errors in log

ℹ️ Support
, , , ,
1

Hi all,

I’ve set up two nextcloud machines which are on low bandwidth and geographically distant.

both machines live in a dmz in their respective buildings, and have direct ip4 connectivity via a port forwarding of port 443. I dont forward port 80. DNS functions correctly on both machines.

both machines will set up to federate with each other, with their resective “federation icon” being yellow in each case. (why not green?).

I can share files to user@https://othermachine.com/ from my own installation, and for small files, it seems to work.

Larger files DO NOT WORK AT ALL. attempting to download or play a shared movie file from the other machine via the web interface or android/ios app results in “CURL timeouts” for incomplete 1.3GiB transfers appearing in the log, and if I check the nginx log files, I see that there are lines like the following each time

192.168.1.170 - - [16/Feb/2024:16:25:44 +0000] “PROPFIND /remote.php/dav/files/kacey/bni.mp4 HTTP/2.0” 207 651 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0”
192.168.1.170 - - [16/Feb/2024:16:26:16 +0000] “GET /remote.php/dav/files/kacey/bni.mp4 HTTP/2.0” 500 12541 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0”
192.168.1.170 - - [16/Feb/2024:16:26:16 +0000] “GET /remote.php/dav/files/kacey/bni.mp4 HTTP/2.0” 500 12541 “-” “Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0”

As you can see, the PROPFIND seems to go OK, but the GET requests (which take a long time to occur, like 30s or so) both fail with 500 codes.

I don’t know if these are related or not (I think they are), but its incredibly frustrating.

I’ve spent HOURS searching for the solution, most of which seems to be people ignorantly increasing buffer sizes to insane looking values, and out of desperation, tried them all.

I cannot find a functional description of nextcloud’s file synchronisation anywhere, either, so if anyone could point me at that, it’d be very useful.

Help!

-Ian

1 Like
2

First, you didn’t provide enough information to help you out directly.

  • What version of Nextcloud Server?
  • What installation method are you using?
  • Are you using the documented NGINX config for Nextcloud from here[1] or something else?
  • Are you using a reverse proxy?

Second, you need to look at your Nextcloud Server log (not just the web/proxy/nginx logs). Those will have the details of the causes for the 500 error. By default that log is in data/nextcloud.log. Also viewable within the Web UI under Administration settings->Logging.

Third, I’d suggest ignoring federation (for troubleshooting purposes) and confirming you can download the files directly from each respective server via the respective Web UI.

By default, chunking will be used by any of the official clients (including the Web UI) so file size should not matter for the most part unless you’ve overridden chunking support.

There are a number of other possibilities, but it depends a great deal on installation method and topology.

I cannot find a functional description of nextcloud’s file synchronisation anywhere, either, so if anyone could point me at that, it’d be very useful.

This isn’t synchronization (that would be limited to the Desktop client for the most part), but I understand what you’re asking. All the documentation is spread across the Admin, User, and Dev Manuals posted here[2].

[1] NGINX configuration — Nextcloud latest Administration Manual latest documentation
[2] https://docs.nextcloud.com/

3

Hi,

To answer your questions:

I’m using NC 28 (whatever latest was about 3 days ago)
copied it into webroot, set up the nginx config as per the NC wiki, visited the site, configured DB access, etc.

I’m not using a reverse proxy - nginx is simply installed on my server’s main OS, in /var/www/nextcloud/

Both NC servers are behind OpenWRT routers with port forwarding enabled, NAT loopback (for faster routing when I’m at home), and I have valid let’sencrypt certs automatically obtained and renewed by dns-rfc2136 on both machines, which the web interface confirms to be valid. (I run my own DNS)

I’m an ex linux kernel dev, so although this stuff isnt familiar, I believe i’ve made a good stab at it.

I did look at the NC log, which is where I’m seeing the same CURL timeouts (CURLE_TIMEDOUT, #28) as about 80% of the rest of the people in the google results.

This problem seems to be posted over and over with never any solid answer posted.

I haven’t touched anything to do with chunking, other than to reduce the chunksize to something smaller (1MB, I think), which made no difference whatsoever.

Hopefully this is enough information, please ask if you need more.

Thanks

-Ian

4

Oh, it’s worth mentioning that the server is on a DMZ, so the firewall has a rule (I checked) to allow DNAT from the LAN to the DMZ where the NC box lives.

Here’s the config file with a couple of annotations.

config.php:

$CONFIG = array (
‘instanceid’ => ‘nope’,
‘passwordsalt’ => ‘nope’,
‘secret’ => ‘nope’,
‘trusted_domains’ =>
array (
0 => 'domain.uk', # <----- domain of my NC box
1 => ‘1.2.3.4’, # <------ wan IP of my router
2 => ‘192.168.200.100’,# <---- internal IP of NC box
),

IIRC, this fixed cert issues when on the LAN, rather than WAN

‘trusted_proxies’ =>
array (
0 => '1.2.3.4, <---- WAN IP on my router
),

‘overwrite.cli.url’ => 'https://domain.uk',
‘overwriteprotocol’ => ‘https’,
‘overwritehost’ => 'domain.uk',
‘overwritecondaddr’ => ‘^1\.2\.3\.4$’,

‘datadirectory’ => ‘/var/www/nextcloud/data’,
‘dbtype’ => ‘pgsql’,
‘version’ => ‘28.0.2.5’,
‘overwrite.cli.url’ => ‘https://domain.uk’,
‘dbname’ => ‘nextcloud’,
‘dbhost’ => ‘localhost’,
‘dbport’ => ‘’,
‘dbtableprefix’ => ‘oc_’,
‘dbuser’ => ‘dbuser’,
‘dbpassword’ => ‘password’,
‘installed’ => true,
‘app_install_overwrite’ =>
array (
0 => ‘maps’,
),
‘maintenance’ => false,
‘maintenance_window_start’ => 1,
‘theme’ => ‘’,
‘loglevel’ => 2,
‘default_locale’ => ‘en_GB’,
‘default_phone_region’ => ‘GB’,
‘filelocking.enabled’ => true,
‘filelocking.ttl’ => 3600,
‘memcache.locking’ => ‘\OC\Memcache\Redis’,
‘memcache.distributed’ => ‘\OC\Memcache\Redis’,
‘memcache.local’ => ‘\OC\Memcache\Redis’,
‘redis’ =>
array (
‘host’ => ‘/run/redis/redis-server.sock’,
‘port’ => 0,
),
‘mail_domain’ => ‘domain.uk’,
‘mail_from_address’ => ‘root’,
‘mail_smtpmode’ => ‘smtp’,
‘mail_sendmailmode’ => ‘smtp’,
‘mail_smtphost’ => ‘mail.domain.uk’,
‘mail_smtpport’ => ‘25’,

‘allow_local_remote_servers’ => true,# <— additonal unknown why.
);

5

the above config file had to be massaged (with ``) or I couldnt post it (I was wondering earlier why all the posts on here have awful quotemarks!)

6

I did look at the NC log, which is where I’m seeing the same CURL timeouts (CURLE_TIMEDOUT, #28) as about 80% of the rest of the people in the google results.

Can you please post that log entry? We need the stack trace to know what is going on.

7

Also, can you please try downloading directly from the other servers own Web UI as I suggested? cURL is not involved in that.

8

Hi again,

Thanks for your replies,

I can download any file (from the web ui), large or small, to my laptop from either nextcloud server, from either location (sorry, I missed the request to test this earlier).

The cURL entry, which only occurs when attempting to download or play a federated file / video (with backtrace) is as follows:

(the bytes transferred by the time it times out varies from error to error, obviously).

{
  "reqId": "mKwL4YGBVE2sZl8U8nhQ",
  "level": 3,
  "time": "2024-02-16T15:11:05+00:00",
  "remoteAddr": "192.168.1.170",
  "user": "kacey",
  "app": "webdav",
  "method": "GET",
  "url": "/remote.php/dav/files/kacey/batteries%20not%20included%20(1988)%20%5B470p%5D.mp4",
  "message": "cURL error 28: Operation timed out after 30000 milliseconds with 20867696 out of 1309419430 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://otherdomain.uk/public.php/webdav/",
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/113.0",
  "version": "28.0.2.5",
  "exception": {
    "Exception": "Sabre\\DAV\\Exception",
    "Message": "cURL error 28: Operation timed out after 30000 milliseconds with 20867696 out of 1309419430 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://otherdomain.uk/public.php/webdav/",
    "Code": 0,
    "Trace": [
      {
        "file": "/mnt/vg0-data/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php",
        "line": 501,
        "function": "convertToSabreException",
        "class": "OCA\\DAV\\Connector\\Sabre\\File",
        "type": "->"
      },
      {
        "file": "/mnt/vg0-data/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",
        "line": 85,
        "function": "get",
        "class": "OCA\\DAV\\Connector\\Sabre\\File",
        "type": "->"
      },
      {
        "file": "/mnt/vg0-data/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
        "line": 89,
        "function": "httpGet",
        "class": "Sabre\\DAV\\CorePlugin",
        "type": "->"
      },
      {
        "file": "/mnt/vg0-data/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 472,
        "function": "emit",
        "class": "Sabre\\DAV\\Server",
        "type": "->"
      },
      {
        "file": "/mnt/vg0-data/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 253,
        "function": "invokeMethod",
        "class": "Sabre\\DAV\\Server",
        "type": "->"
      },
      {
        "file": "/mnt/vg0-data/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
        "line": 321,
        "function": "start",
        "class": "Sabre\\DAV\\Server",
        "type": "->"
      },
      {
        "file": "/mnt/vg0-data/www/nextcloud/apps/dav/lib/Server.php",
        "line": 370,
        "function": "exec",
        "class": "Sabre\\DAV\\Server",
        "type": "->"
      },
      {
        "file": "/mnt/vg0-data/www/nextcloud/apps/dav/appinfo/v2/remote.php",
        "line": 35,
        "function": "exec",
        "class": "OCA\\DAV\\Server",
        "type": "->"
      },
      {
        "file": "/mnt/vg0-data/www/nextcloud/remote.php",
        "line": 172,
        "args": [
          "/mnt/vg0-data/www/nextcloud/apps/dav/appinfo/v2/remote.php"
        ],
        "function": "require_once"
      }
    ],
    "File": "/mnt/vg0-data/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php",
    "Line": 765,
    "Previous": {
      "Exception": "GuzzleHttp\\Exception\\ConnectException",
      "Message": "cURL error 28: Operation timed out after 30000 milliseconds with 20867696 out of 1309419430 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://otherdomain.uk/public.php/webdav/",
      "Code": 0,
      "Trace": [
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php",
          "line": 158,
          "function": "createRejection",
          "class": "GuzzleHttp\\Handler\\CurlFactory",
          "type": "::",
          "args": [
            "*** sensitive parameters replaced ***"
          ]
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php",
          "line": 110,
          "function": "finishError",
          "class": "GuzzleHttp\\Handler\\CurlFactory",
          "type": "::"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlHandler.php",
          "line": 47,
          "function": "finish",
          "class": "GuzzleHttp\\Handler\\CurlFactory",
          "type": "::"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php",
          "line": 137,
          "function": "__invoke",
          "class": "GuzzleHttp\\Handler\\CurlHandler",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/lib/private/Http/Client/DnsPinMiddleware.php",
          "line": 121,
          "function": "GuzzleHttp\\{closure}",
          "class": "GuzzleHttp\\Middleware",
          "type": "::",
          "args": [
            "*** sensitive parameters replaced ***"
          ]
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/PrepareBodyMiddleware.php",
          "line": 35,
          "function": "OC\\Http\\Client\\{closure}",
          "class": "OC\\Http\\Client\\DnsPinMiddleware",
          "type": "->",
          "args": [
            "*** sensitive parameters replaced ***"
          ]
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php",
          "line": 31,
          "function": "__invoke",
          "class": "GuzzleHttp\\PrepareBodyMiddleware",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/RedirectMiddleware.php",
          "line": 71,
          "function": "GuzzleHttp\\{closure}",
          "class": "GuzzleHttp\\Middleware",
          "type": "::",
          "args": [
            "*** sensitive parameters replaced ***"
          ]
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Middleware.php",
          "line": 63,
          "function": "__invoke",
          "class": "GuzzleHttp\\RedirectMiddleware",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/HandlerStack.php",
          "line": 75,
          "function": "GuzzleHttp\\{closure}",
          "class": "GuzzleHttp\\Middleware",
          "type": "::",
          "args": [
            "*** sensitive parameters replaced ***"
          ]
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php",
          "line": 331,
          "function": "__invoke",
          "class": "GuzzleHttp\\HandlerStack",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php",
          "line": 168,
          "function": "transfer",
          "class": "GuzzleHttp\\Client",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Client.php",
          "line": 187,
          "function": "requestAsync",
          "class": "GuzzleHttp\\Client",
          "type": "->",
          "args": [
            "*** sensitive parameters replaced ***"
          ]
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/lib/private/Http/Client/Client.php",
          "line": 230,
          "function": "request",
          "class": "GuzzleHttp\\Client",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/lib/private/Files/Storage/DAV.php",
          "line": 366,
          "function": "get",
          "class": "OC\\Http\\Client\\Client",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php",
          "line": 298,
          "function": "fopen",
          "class": "OC\\Files\\Storage\\DAV",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php",
          "line": 314,
          "function": "fopen",
          "class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php",
          "line": 298,
          "function": "fopen",
          "class": "OC\\Files\\Storage\\Wrapper\\Availability",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/lib/private/Files/View.php",
          "line": 1159,
          "function": "fopen",
          "class": "OC\\Files\\Storage\\Wrapper\\Wrapper",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/lib/private/Files/View.php",
          "line": 987,
          "function": "basicOperation",
          "class": "OC\\Files\\View",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/apps/dav/lib/Connector/Sabre/File.php",
          "line": 499,
          "function": "fopen",
          "class": "OC\\Files\\View",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php",
          "line": 85,
          "function": "get",
          "class": "OCA\\DAV\\Connector\\Sabre\\File",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php",
          "line": 89,
          "function": "httpGet",
          "class": "Sabre\\DAV\\CorePlugin",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
          "line": 472,
          "function": "emit",
          "class": "Sabre\\DAV\\Server",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
          "line": 253,
          "function": "invokeMethod",
          "class": "Sabre\\DAV\\Server",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php",
          "line": 321,
          "function": "start",
          "class": "Sabre\\DAV\\Server",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/apps/dav/lib/Server.php",
          "line": 370,
          "function": "exec",
          "class": "Sabre\\DAV\\Server",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/apps/dav/appinfo/v2/remote.php",
          "line": 35,
          "function": "exec",
          "class": "OCA\\DAV\\Server",
          "type": "->"
        },
        {
          "file": "/mnt/vg0-data/www/nextcloud/remote.php",
          "line": 172,
          "args": [
            "/mnt/vg0-data/www/nextcloud/apps/dav/appinfo/v2/remote.php"
          ],
          "function": "require_once"
        }
      ],
      "File": "/mnt/vg0-data/www/nextcloud/3rdparty/guzzlehttp/guzzle/src/Handler/CurlFactory.php",
      "Line": 210
    },
    "message": "cURL error 28: Operation timed out after 30000 milliseconds with 20867696 out of 1309419430 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://otherdomain.uk/public.php/webdav/",
    "exception": [],
    "CustomMessage": "cURL error 28: Operation timed out after 30000 milliseconds with 20867696 out of 1309419430 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://otherdomain.uk/public.php/webdav/"
  },
  "id": "65cff1081654d"
}
9

hmmm. anonymised log output would be a nice feature to be on by default, if it existed :wink:

10

oh and I renamed the test file to remove the spaces and special characters at some point (no change though, still failed)

11

I’ve been searching and prodding this all day and still it refuses to work.

Any idea?

12

Sorry, I’m new to these forums - I dont appear to have replied directly, so I’m replying here in case.

13

Anything? or is this tumble-weed time again?

14

This is a curl issue. Default timeout is 30 seconds. And if it takes longer it will break.

1.3gb on slow and distant connections will take longer time.

There is a config option for it

“If you encounter timeouts for downloading or uploading large files, you can use the option davstorage.request_timeout in your config.php to increase the timeout. The default value is 30 seconds.”

The servers will become green when files has been shared both directions and accepted and a successful adressbok exchange has been made (run by Cron or manually with occ federation:sync-addressbooks)

15

I’ll check that once the other server is back online. Ta.

16

shouldnt that 1.3gib be transferred in “chunks” or something though?

I can easily see why 30s isnt enough time to xfer 1.3GiB over a 20Mbit uplink, but surely the file is only pulled up “on demand”, esp. if just streaming the video from the web ui?

If I play a movie from a federated share, I don’t want to wait for it to be copied entirely, I just want (and expect) to stream it.

What’s the point in a share otherwise? I might as well just copy the entire movie archive over onto the other machine, if it’s required to copy the entire file prior to playback anyway!

17

Pure WebDAV is never chunked. And I don’t think that the protocol
between severs are chunked. ( I am a bit unsure on this but from the docs and how I have used it before this fits)

Nextcloud federation is built so you are in control of what you share. That means that a shared file over federation is never copied over to the server you share to.

The person that receive a share can’t just cut the Internet and do what he want on the file. You can always unshare it and the other server doesn’t have to respond to the unshare. It is simply just gone.

So the file is provided through a server to you.

This logic could perhaps be improved a bit for better performance. (Even if it is chunked proxying it through a server would be a hit of performance)

18

Thats… exactly what I want - I dont want the file copied at all, only to be able to access and view / seek videos from the federated servers shares (in particular) in real time. I really could care less if any of the file ever hits the destination servers disk (other than a bit of caching!).

But it’s not what happens. If I try to play a movie, it waits 30s, errors, and I get the cURL timeout. If it’s not trying to copy the entire movie, then why is it timing out? surely as long as the file arrives faster than it’s being played back, it should just work?

19

I rarely work with large federated video files. I mostly do documents. And so does 99% of my clients that does federated shares.

I should try some since my hobby is photo and video and see what I find with federation.

There are some bugs about WebDAV in general (it is a slow protocol) and there are some about federation. I agree that this could be improved. If a large file works then small files should work better.

Design so far from what I have seen has been around smaller files.

20

Update: I tried again. Full reinstall on Debian Trixie, postgres, nginx.

I don’t know why exactly, but the certificate stuff above was all bollocks - none of that stuff seems to be necessary.

Configuring php-fpm to have a max of 4 children seems to have kept memory usage same, as has tweaking postgres a little, but nothing out of the ordinary.

I’ve kept notes on the build of this server, and its fairly stable so far. There are some issues with the kernel on rockpro64 (I havent had time to investigate, so I’ve replaced the kernel with something a bit newer.

The other older, non-rebuilt machine is on bookworm, with a kernel from bookworm-backports, and seems stable.

Both run NC30.0.2 now. For some reason federation works with essentially the default config. Both machines (eventually, it took a while) showed up green in the sharing config page.

I still have to disable bulk transfers in the config, or the Linux sync client will hang unpredictably (but repeatably when it does). No idea why still.

Performance is still subpar, but some of that seems to have been down to my (fairly old) router.

Recognize seems to really foul the machine up though. Unsure why - it passes memtest and stress-ng tests (even simultaneously). I’m a bit reluctant to re-enable it until I have a strategy for backing up the database.