Very odd issues with docker-compose setup (mariadb + traefik)

odinsride · September 24, 2019, 2:40pm

Nextcloud version (eg, 12.0.2): 16.0.4, via docker-compose
Operating system and version (eg, Ubuntu 17.04): Debian Buster 10.1
Apache or nginx version (eg, Apache 2.4.25): Not sure, whatever is in the Docker version
PHP version (eg, 7.1): Not sure, whatever is in the Docker version
Client OS/Browser: macOS Mojave / Firefox 69.0.1

The issue you are facing: When accessing nextcloud web from my home network, I get an error when trying to download particular files (I have seen the issue with zip files as well as mp4 and png files):

Downloads/vh4tAhFe.zip.part could not be saved, because the source file could not be read.
Try again later, or contact the server administrator.

The odd part is when I moved to a different network (same computer and browser), I was able to download the same file without any issues. Additionally, if I use another browser (Safari), the file appears to download without that error, but when I check the download progress, the download speed keeps dwindling down over a period of a few minutes until it reaches 0. It gets stuck at low download percentages like 2-5%.

I am thinking there is an issue with my local network setup, I just don’t know what it is. I am using docker-compose to stand up nextcloud, mariadb, and using traefik for reverse proxy.

I am accessing my Nextcloud instance from (https://cloud.mydomain.com). Finally, when I use the Nextcloud iOS app from within my home network, I am able to download the same file without issue as well. I’m really lost here, can anyone help?

Is this the first time you’ve seen this error? (Y/N): N, brand new Nextcloud install

Steps to replicate it:

Navigate to https://cloud.mydomain.com
Click to download a specific file
Error message is presented in firefox download dialog

The output of your Nextcloud log in Admin > Logging:

(I see 2 main errors in my logs, but these are from hours ago and I don’t think related to the download error. In fact, getting the download error doesn’t produce any additional log entry)

[webdav] Fatal: Sabre\DAV\Exception\BadRequest: expected filesize 3099799 got 2555904 at <<closure>>

0. /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php line 156
   OCA\DAV\Connector\Sabre\File->put(null)
1. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 1096
   OCA\DAV\Connector\Sabre\Directory->createFile("2019-03-22_2872.dng", null)
2. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 525
   Sabre\DAV\Server->createFile("Photos/2019/03/2019-03-22_2872.dng", null, null)
3. <<closure>>
   Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
4. /var/www/html/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
   undefinedundefinedcall_user_func_array([Sabre\DAV\CorePlugin {},"httpPut"], [Sabre\HTTP\Requ ... }])
5. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 479
   Sabre\Event\EventEmitter->emit("method:PUT", [Sabre\HTTP\Requ ... }])
6. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 254
   Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
7. /var/www/html/apps/dav/appinfo/v1/webdav.php line 80
   Sabre\DAV\Server->exec()
8. /var/www/html/remote.php line 163
   undefinedundefinedrequire_once("/var/www/html/a ... p")

PUT /remote.php/webdav/Photos/2019/03/2019-03-22_2872.dng
from ${MY_IP_ADDR} by sitheris at 2019-09-24T11:00:51+00:100:

[no app in context] Error: Sabre\DAV\Exception\BadRequest: expected filesize 3099799 got 2555904 at <<closure>>

0. /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php line 156
   OCA\DAV\Connector\Sabre\File->put(null)
1. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 1096
   OCA\DAV\Connector\Sabre\Directory->createFile("2019-03-22_2872.dng", null)
2. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 525
   Sabre\DAV\Server->createFile("Photos/2019/03/2019-03-22_2872.dng", null, null)
3. <<closure>>
   Sabre\DAV\CorePlugin->httpPut(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
4. /var/www/html/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
   undefinedundefinedcall_user_func_array([Sabre\DAV\CorePlugin {},"httpPut"], [Sabre\HTTP\Requ ... }])
5. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 479
   Sabre\Event\EventEmitter->emit("method:PUT", [Sabre\HTTP\Requ ... }])
6. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 254
   Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
7. /var/www/html/apps/dav/appinfo/v1/webdav.php line 80
   Sabre\DAV\Server->exec()
8. /var/www/html/remote.php line 163
   undefinedundefinedrequire_once("/var/www/html/a ... p")

PUT /remote.php/webdav/Photos/2019/03/2019-03-22_2872.dng
from ${MY_IP_ADDR} by sitheris at 2019-09-24T11:00:51+00:00

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'instanceid' => '<<redacted>>',
  'passwordsalt' => '<<redacted>>',
  'secret' => '<<redacted>>',
  'trusted_domains' =>
  array (
    0 => 'cloud.mydomain.com',
    1 => '192.168.1.21:8181',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '16.0.4.1',
  'overwrite.cli.url' => 'https://cloud.mydomain.com',
  'overwriteprotocol' => 'https',
  'dbname' => 'nextcloud-db',
  'dbhost' => 'nextcloud-db:3306',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => '<<redacted>>',
  'dbpassword' => '<<redacted>>',
  'installed' => true,
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
);

The output of your Apache/nginx/system log in /var/log/____:

Could not find this in docker container

My docker-compose.yml:

version: "3.6"

services:
  traefik:
    hostname: traefik
    image: traefik:v1.7.16
    container_name: traefik
    restart: always
    domainname: ${DOMAINNAME}
    networks:
      - internal
      - traefik_proxy
    ports:
      - "80:80"
      - "443:443"
      - "8080:8080"
    environment:
      - CF_API_EMAIL=${CF_API_EMAIL}
      - CF_API_KEY=${CF_API_KEY}
    labels:
      - "traefik.enable=true"
      - "traefik.backend=traefik"
      - "traefik.frontend.rule=Host:traefik.${DOMAINNAME}"
      - "traefik.port=8080"
      - "traefik.docker.network=traefik_proxy"
      - "traefik.frontend.headers.SSLRedirect=true"
      - "traefik.frontend.headers.STSSeconds=315360000"
      - "traefik.frontend.headers.browserXSSFilter=true"
      - "traefik.frontend.headers.contentTypeNosniff=true"
      - "traefik.frontend.headers.forceSTSHeader=true"
      - "traefik.frontend.headers.SSLHost=oluhome.com"
      - "traefik.frontend.headers.STSIncludeSubdomains=true"
      - "traefik.frontend.headers.STSPreload=true"
      - "traefik.frontend.headers.frameDeny=true"
      - "traefik.frontend.auth.basic.users=${HTTP_USERNAME}:${HTTP_PASSWORD}"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${USERDIR}/docker/traefik:/etc/traefik
      - ${USERDIR}/docker/shared:/shared

  nextcloud-db:
    image: mariadb
    container_name: nextcloud-db
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    volumes:
      - ${USERDIR}/docker/mariadb/db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_DATABASE=nextcloud-db
      - MYSQL_USER=nextcloud
      - MYSQL_PASSWORD=nextcloud
      - MYSQL_HOST=nextcloud-db:3306
      - TZ=${TZ}
      - PUID=${PUID}
      - PGID=${PGID}
    restart: always
    networks:
      - internal

  nextcloud:
    image: nextcloud
    container_name: nextcloud
    ports:
      - "8181:80"
    depends_on:
      - nextcloud-db
    volumes:
      - ${USERDIR}/docker/nextcloud/main:/var/www/html
      - ${USERDIR}/docker/nextcloud/apps:/var/www/html/apps
      - ${USERDIR}/docker/nextcloud/config:/var/www/html/config
      - ${USERDIR}/docker/nextcloud/custom_apps:/var/www/html/custom_apps
      - /neptune-s1/nextcloud/data:/var/www/html/data
    restart: always
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
    networks:
      - traefik_proxy
      - internal
    labels:
      - "traefik.enable=true"
      - "traefik.backend=nextcloud"
      - "traefik.frontend.rule=Host:cloud.${DOMAINNAME}"
      - "traefik.port=80"
      - "traefik.protocol=http"
      - "traefik.docker.network=traefik_proxy"
      - "traefik.frontend.headers.SSLRedirect=true"
      - "traefik.frontend.headers.STSSeconds=315360000"
      - "traefik.frontend.headers.browserXSSFilter=true"
      - "traefik.frontend.headers.contentTypeNosniff=true"
      - "traefik.frontend.headers.forceSTSHeader=true"
      - "traefik.frontend.headers.SSLHost=cloud.${DOMAINNAME}"
      - "traefik.frontend.headers.SSLForceHost=true"
      - "traefik.frontend.headers.STSIncludeSubdomains=true"
      - "traefik.frontend.headers.STSPreload=true"
      - "traefik.frontend.headers.frameDeny=true"

networks:
  traefik_proxy:
    external:
      name: traefik_proxy
  internal:
    driver: bridge
    external: false

KarlF12 · September 24, 2019, 10:18pm

When you connect from your LAN, are you still connecting through traefik?

odinsride · September 24, 2019, 10:42pm

Pretty sure I am, as I access it through https://cloud.mydomain.com

KarlF12 · September 25, 2019, 12:09am

So that FQDN will resolve to one of:

The LAN IP of the proxy
The corresponding WAN NAT IP
The system running Docker

Ideally this would be the proxy LAN IP as managed by split DNS.

odinsride · September 25, 2019, 4:18pm

Hmm, when I ping it from various devices on my local network (both Wifi and Ethernet, and including the problematic device), they all resolve cloud.mydomain.com to my WAN IP. How would I change it to resolve to the LAN IP when I’m connected to my local network?

KarlF12 · September 25, 2019, 5:15pm

You would need to run a local DNS server and set up a record for it. If you don’t have a device handy that has a built in DNS server, you can install bind on your server. I run mine on pfSense.

odinsride · September 27, 2019, 1:25pm

I don’t have a local DNS server. Is this something I can set up in my router? Also, why do some devices work fine with Nextcloud on my network when others don’t? It makes no sense.

KarlF12 · September 27, 2019, 2:16pm

That depends entirely on your router. But as I said, you can always install bind on Linux for local DNS.

I’m really not sure. I’m just mentioning this because some routers don’t work well when going out through NAT at coming back in in a port forward on the same interface. Some routers don’t allow that at all. So as a best practice you should use the LAN IP while on the local network. That may or may not have any bearing on your problem.

You can test by adding the LAN IP to the device’s hosts file and see if that fixes it.

odinsride · October 3, 2019, 11:39am

So I spent yesterday digging through this again. I rebuilt my entire Nextcloud setup from scratch, this time using the images from linuxserver repo (linuxserver/nextcloud, linuxserver/mariadb), and dropped traefik in favor of the letsencrypt container. I got everything working again only to find the exact same issues I had in the original post, which baffled me.

I then spent several hours rebuilding the entire setup (including blowing away the config folders each time), messing with the letsencrypt proxy settings for nextcloud as well as the NC config.php file, to no avail. I additionally checked into making sure permissions were all set correctly for all NC volumes defined in my docker-compose file. (Set everything to be owned by my user:docker)…still no luck.

Finally, I stumbled upon another forum post here which mentioned issues with my specific router (Netgear R7800). They said flashing the router to OpenWRT alleviated the issues. So I did this with my router, and what do you know, my NC instance is working perfectly now! So there is something weird in the stock firmware for the Netgear routers apparently. Just glad I finally resolved this and can start enjoying Nextcloud

I’m posting all this info in the event someone googles the same issue and happens to stumble upon this thread. Hi future viewers

Anyway, thank you for your help @KarlF12!

KarlF12 · October 3, 2019, 11:41pm

Yes, home routers can be weird or incomplete sometimes or not be trustworthy for security, so that’s why I run a virtualized firewall in my home setup. If you ever get the inclination to try it, check out pfSense. I have yet to come across anything it would not do or had problems with.

odinsride · October 4, 2019, 1:48pm

Thanks, I will make note of that and look into it when I have some time. I believe using pfSense requires multiple NICs from what I’ve read on it before, is that correct? My server is just a beefy laptop and only has one NIC so not sure if I could go that route until I get some new hardware

KarlF12 · October 4, 2019, 7:40pm

Well that kind of depends on the situation. It’s a firewall so yes for normal operation it would need at least a LAN NIC and WAN NIC.

But, if we’re talking about a virtual machine, then we’re also talking about virtual NICs which can be on the same physical adapter trunking to a switch on different VLANs. If you were using a VLAN capable switch and hypervisor then you could set it up using only a single physical NIC on the server hardware. And on physical hardware I think pfSense does support VLAN subinterfaces so something could be done there too with a good switch.

zefrabila · November 24, 2019, 8:02pm

Hey @odinsride,

if you already flashed OpenWrt on your router you can leverage the inbuilt DNS server of the router to append your domain name (e.g. mydomain.com) to your clients in the LAN.

So if you configure your router to give your server (laptop) a static ip via its mac address you can also set the given hostname (only inside the router tables) to “cloud” so it would be reachable inside your lan by calling cloud.mydomain.com which would then be resolved to your internal ip instead of your wan ip.