Internal Server Error (500) when attempting to access Nextcloud behind reverse proxy (Traefik)

havenoclu · October 8, 2023, 6:17pm

Support intro

Sorry to hear you’re facing problems

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can

Nextcloud version: 27.0.2.1 (Official Docker image, currently latest)
Operating system and version: Podman on Fedora 37
Apache version: 2.4.57
PHP version: 8.2.10

The issue you are facing:
I am receiving 500 errors when I try to access my Nextcloud instance behind Traefik. If I take Traefik out of the equation, it works fine.

A couple notes:

Traefik works with another container as expected, so I believe Traefik is doing what it should be…so it’s prob my label configuration on the Nextcloud side or something
I cannot seem to get the level of debug I would like to see to help troubleshoot although I have not done anything in the tcpdump arena as of yet (which may or may not help?)

Steps to replicate it:

Stand up Nextcloud via docker image (Docker) (details below)
Bring up Traefik container
Attempt to access Nextcloud webUI
Observe Internal Server Error

The output of your Nextcloud log in Admin > Logging:

Is this the same as /var/www/html/data/nextcloud.log? If so, there are no relevant logs contained in the logfile. I have tried adding the following to the config.php but things do not seem to be more verbose:

  'debug' => 'true',

and tried:

  'loglevel_frontend' => '0',
  'loglevel' => '0',

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '127.0.0.1',
    'password' => 'password',
    'port' => 6379,
  ),
  'passwordsalt' => 'passwordsalt',
  'secret' => 'secret',
  'trusted_domains' => 
  array (
    0 => 'localhost',
    1 => 'cloud.example.com',
    2 => '10.89.0.0/24',
    3 => '10.88.0.0/24',
  ),
  'trusted_proxies' =>
  array (
    0 => '192.168.2.0/24',
    1 => '10.89.0.0/24',
    2 => '10.88.0.0/24',
  ),
  'forwarded-for-headers' =>
  array (
    0 => 'X-Forwarded-For',
    1 => 'HTTP_X_FORWARDED_FOR',
  ),
  'debug' => 'true',
  'loglevel_frontend' => '0',
  'loglevel' => '0',
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '27.0.2.1',
  'overwrite.cli.url' => 'http://localhost',
  'dbname' => 'nextcloud',
  'dbhost' => '127.0.0.1',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'dbpassword',
  'installed' => true,
  'instanceid' => 'oc7lpgj2huhc',
  'mail_from_address' => 'mail',
  'mail_smtpmode' => 'smtp',
  'mail_sendmailmode' => 'smtp',
  'mail_domain' => 'example.com',
  'mail_smtphost' => 'mail.example.com',
  'mail_smtpport' => '25',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'smtp@example.com',
  'mail_smtppassword' => 'smtppassword',
);

The output of your Apache/nginx/system log in /var/log/____:
Note: I enabled LogLevel debug in Apache but still didn’t get much info. Here is a snippet from when I try to access the webUI via podman logs <nextcloud_container>:

[Sun Oct 08 18:06:36.596332 2023] [remoteip:debug] [pid 46] mod_remoteip.c(679): [client 10.89.0.55:38758] AH01569: RemoteIP: Header X-Real-IP value of 192.168.2.192 appears to be a private IP or nonsensical.  Ignored
[Sun Oct 08 18:06:36.597239 2023] [authz_core:debug] [pid 46] mod_authz_core.c(815): [client 10.89.0.55:38758] AH01626: authorization result of Require all granted: granted
[Sun Oct 08 18:06:36.597266 2023] [authz_core:debug] [pid 46] mod_authz_core.c(815): [client 10.89.0.55:38758] AH01626: authorization result of <RequireAny>: granted
[Sun Oct 08 18:06:36.597431 2023] [authz_core:debug] [pid 46] mod_authz_core.c(815): [client 10.89.0.55:38758] AH01626: authorization result of Require all granted: granted
[Sun Oct 08 18:06:36.597441 2023] [authz_core:debug] [pid 46] mod_authz_core.c(815): [client 10.89.0.55:38758] AH01626: authorization result of <RequireAny>: granted
10.89.0.55 192.168.2.192 - - [08/Oct/2023:18:06:36 +0000] "GET /index.php/apps/files/ HTTP/1.1" 500 702 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
[Sun Oct 08 18:06:36.709940 2023] [remoteip:debug] [pid 45] mod_remoteip.c(679): [client 10.89.0.55:38762] AH01569: RemoteIP: Header X-Real-IP value of 192.168.2.192 appears to be a private IP or nonsensical.  Ignored
[Sun Oct 08 18:06:36.710487 2023] [authz_core:debug] [pid 45] mod_authz_core.c(815): [client 10.89.0.55:38762] AH01626: authorization result of Require all granted: granted
[Sun Oct 08 18:06:36.710505 2023] [authz_core:debug] [pid 45] mod_authz_core.c(815): [client 10.89.0.55:38762] AH01626: authorization result of <RequireAny>: granted
[Sun Oct 08 18:06:36.710602 2023] [core:info] [pid 45] [client 10.89.0.55:38762] AH00128: File does not exist: /var/www/html/favicon.ico
[Sun Oct 08 18:06:36.710630 2023] [remoteip:debug] [pid 45] mod_remoteip.c(679): [client 10.89.0.55:38762] AH01569: RemoteIP: Header X-Real-IP value of 192.168.2.192 appears to be a private IP or nonsensical.  Ignored
[Sun Oct 08 18:06:36.710711 2023] [authz_core:debug] [pid 45] mod_authz_core.c(815): [client 10.89.0.55:38762] AH01626: authorization result of Require all granted: granted
[Sun Oct 08 18:06:36.710721 2023] [authz_core:debug] [pid 45] mod_authz_core.c(815): [client 10.89.0.55:38762] AH01626: authorization result of <RequireAny>: granted
[Sun Oct 08 18:06:36.710898 2023] [authz_core:debug] [pid 45] mod_authz_core.c(815): [client 10.89.0.55:38762] AH01626: authorization result of Require all granted: granted
[Sun Oct 08 18:06:36.710907 2023] [authz_core:debug] [pid 45] mod_authz_core.c(815): [client 10.89.0.55:38762] AH01626: authorization result of <RequireAny>: granted
10.89.0.55 192.168.2.192 - - [08/Oct/2023:18:06:36 +0000] "GET /favicon.ico HTTP/1.1" 500 702 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"

Here is the podman run command I am using to start the container:

/usr/bin/podman run -d --pod nextcloud --name nextcloud-service \
	-v /containers/nextcloud/var/www/html:/var/www/html \
	-v /containers/nextcloud/apache2:/etc/apache2 \
	-e MYSQL_USER=nextcloud \
	-e MYSQL_PASSWORD=mysqlpassword \
	-e MYSQL_DATABASE=nextcloud \
	-e MYSQL_HOST=127.0.0.1 \
	-e NEXTCLOUD_ADMIN_USER=admin \
	-e NEXTCLOUD_ADMIN_PASSWORD=adminpassword \
	-e NEXTCLOUD_TRUSTED_DOMAINS=cloud.example.com \
	-e TRUSTED_PROXIES="192.168.2.0/24 10.8.0.0/16" \
	-e REDIS_HOST=127.0.0.1 \
	-e REDIS_HOST_PORT=6379 \
	-e REDIS_HOST_PASSWORD=redispassword \
	--net web \
	-l traefik.http.routers.nextcloud.entrypoints="web" \
        -l traefik.http.routers.nextcloud.rule=Host\(\`cloud.example.com\`\) \
        -l traefik.http.middlewares.https-redirect.redirectscheme.scheme="https" \
        -l traefik.http.routers.nextcloud.middlewares="nc-header,https-redirect" \
        -l traefik.http.routers.nextcloud-secure.entrypoints="websecure" \
        -l traefik.http.routers.nextcloud-secure.rule=Host\(\`cloud.example.com\`\) \
        -l traefik.http.middlewares.nc-rep.redirectregex.regex='https://(.*)/.well-known/(card|cal)dav' \
        -l traefik.http.middlewares.nc-rep.redirectregex.replacement='https://$$1/remote.php/dav/' \
        -l traefik.http.middlewares.nc-rep.redirectregex.permanent="true" \
        -l traefik.http.middlewares.nc-header.headers.customFrameOptionsValue="SAMEORIGIN" \
        -l traefik.http.middlewares.nc-header.headers.customResponseHeaders.Strict-Transport-Security="15552000" \
        -l traefik.http.routers.nextcloud-secure.middlewares="nc-rep,nc-header" \
        -l traefik.http.routers.nextcloud-secure.tls="true" \
        -l traefik.http.routers.nextcloud-secure.tls.certresolver="le" \
        -l traefik.http.routers.nextcloud-secure.service="nextcloud" \
        -l traefik.http.services.nextcloud.loadbalancer.server.port="80" \
        -l traefik.http.services.nextcloud.loadbalancer.passHostHeader="true" \
	docker.io/library/nextcloud:latest

wwe · October 8, 2023, 8:27pm

it looks you are doing lot of stuff like custom middlewares in your traefik config. I would recommend you start with simple config and add complexity once you get the simple setup working. maybe you find my example useful Docker - nextcloud oder linuxserver/nextcloud - #6 by wwe

havenoclu · October 9, 2023, 1:41pm

Turns out, it was the podman network I was specifying as everything started working when I removed the --net web reference (which I suppose makes the container use the default podman network). Thanks for getting back to me @wwe - I modified my traefik labels to essentially mirror yours and things still didn’t work, so pulling the network was a bit of a last resort. I’m a little perplexed on why that was the cause, but will investigate. I’m wondering if it had something to do with the nextcloud container being in a pod (the pod contained nextcloud, redis, and mariadb). We shall see… Time to turn off all this debug now

In any case, I’m happy things are working as expected!