Getting 503 Service Unavailable after upgrade to 27.1.5.1

Nextcloud version: 27.1.5.1 (dockerized version)
Operating system and version: Ubuntu 22.04.3 LTS
Apache or nginx version: Using lighttpd 1.4.64 as a reverse proxy
PHP version: 8.2.11 (cli)

I am unable to connect to my self hosted instance of Nextcloud getting 503 Service Unavailable (standard https connection on port 443, which is used by all the clients) after upgrade to 27.1.5.1, upgrade itself seems to have gone alright. I am able to connect to the AIO on port 8080 using the ip address and it says “Your containers are up-to-date.” All containers are running.

I am using reverse proxy and it was working properly before the upgrade and nothing changed on that front, so it should not be the cause. I’ve tested with simple server client application that the port translation from 443 to custom port (I am not using the default proposed by documentation when using reverse proxy) works.

Errors from nextcloud-aio-nextcloud container:

Activating Collabora config...
Failed to activate any config changes
Server error: `GET https://nextcloud.customdomain.cz/hosting/discovery` resulted in a `503 Service Unavailable` response:
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "htt (truncated...)

#0 /var/www/html/3rdparty/guzzlehttp/guzzle/src/Middleware.php(69): GuzzleHttp\Exception\RequestException::create(Object(GuzzleHttp\Psr7\Request), Object(GuzzleHttp\Psr7\Response), NULL, Array, NULL)
#1 /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php(204): GuzzleHttp\Middleware::GuzzleHttp\{closure}(Object(GuzzleHttp\Psr7\Response))
#2 /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php(153): GuzzleHttp\Promise\Promise::callHandler(1, Object(GuzzleHttp\Psr7\Response), NULL)
#3 /var/www/html/3rdparty/guzzlehttp/promises/src/TaskQueue.php(48): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}()
#4 /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php(248): GuzzleHttp\Promise\TaskQueue->run(true)
#5 /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php(224): GuzzleHttp\Promise\Promise->invokeWaitFn()
#6 /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php(269): GuzzleHttp\Promise\Promise->waitIfPending()
#7 /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php(226): GuzzleHttp\Promise\Promise->invokeWaitList()
#8 /var/www/html/3rdparty/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#9 /var/www/html/3rdparty/guzzlehttp/guzzle/src/Client.php(187): GuzzleHttp\Promise\Promise->wait()
#10 /var/www/html/lib/private/Http/Client/Client.php(230): GuzzleHttp\Client->request('get', 'https://nextclo...', Array)
#11 /var/www/html/custom_apps/richdocuments/lib/WOPI/DiscoveryManager.php(89): OC\Http\Client\Client->get('https://nextclo...', Array)
#12 /var/www/html/custom_apps/richdocuments/lib/WOPI/DiscoveryManager.php(61): OCA\Richdocuments\WOPI\DiscoveryManager->fetchFromRemote()
#13 /var/www/html/custom_apps/richdocuments/lib/WOPI/Parser.php(41): OCA\Richdocuments\WOPI\DiscoveryManager->get()
#14 /var/www/html/custom_apps/richdocuments/lib/Command/ActivateConfig.php(67): OCA\Richdocuments\WOPI\Parser->getUrlSrc('Capabilities')
#15 /var/www/html/3rdparty/symfony/console/Command/Command.php(298): OCA\Richdocuments\Command\ActivateConfig->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /var/www/html/3rdparty/symfony/console/Application.php(1040): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /var/www/html/3rdparty/symfony/console/Application.php(301): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Richdocuments\Command\ActivateConfig), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /var/www/html/3rdparty/symfony/console/Application.php(171): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#19 /var/www/html/lib/private/Console/Application.php(211): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#20 /var/www/html/console.php(100): OC\Console\Application->run()
#21 /var/www/html/occ(11): require_once('/var/www/html/c...')
#22 {main}

Error from nextcloud-aio-mastercontainer container:

Deleting duplicate sessions
NOTICE: PHP message: 404 Not Found
Type: Slim\Exception\HttpNotFoundException
Code: 404
Message: Not found.
File: /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/RoutingMiddleware.php
Line: 76
Trace: #0 /var/www/docker-aio/php/vendor/slim/slim/Slim/Routing/RouteRunner.php(56): Slim\Middleware\RoutingMiddleware->performRouting(Object(GuzzleHttp\Psr7\ServerRequest))
#1 /var/www/docker-aio/php/vendor/slim/csrf/src/Guard.php(476): Slim\Routing\RouteRunner->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#2 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(168): Slim\Csrf\Guard->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Slim\Routing\RouteRunner))
#3 /var/www/docker-aio/php/vendor/slim/twig-view/src/TwigMiddleware.php(115): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#4 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Views\TwigMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#5 /var/www/docker-aio/php/src/Middleware/AuthMiddleware.php(38): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#6 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(269): AIO\Middleware\AuthMiddleware->__invoke(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#7 /var/www/docker-aio/php/vendor/slim/slim/Slim/Middleware/ErrorMiddleware.php(76): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#8 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(121): Slim\Middleware\ErrorMiddleware->process(Object(GuzzleHttp\Psr7\ServerRequest), Object(Psr\Http\Server\RequestHandlerInterface@anonymous))
#9 /var/www/docker-aio/php/vendor/slim/slim/Slim/MiddlewareDispatcher.php(65): Psr\Http\Server\RequestHandlerInterface@anonymous->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#10 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(199): Slim\MiddlewareDispatcher->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#11 /var/www/docker-aio/php/vendor/slim/slim/Slim/App.php(183): Slim\App->handle(Object(GuzzleHttp\Psr7\ServerRequest))
#12 /var/www/docker-aio/php/public/index.php(185): Slim\App->run()
#13 {main}
Tips: To display error details in HTTP response set "displayErrorDetails" to true in the ErrorHandler constructor.

The output of your Nextcloud log in Admin > Logging:
Not sure where to access this, if it’s from the UI on port 443 then it’s not reachable.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'one-click-instance' => true,
  'one-click-instance.user-limit' => 100,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'appsallowlist' => 
  array (
  ),
  'check_data_directory_permissions' => false,
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => 'nextcloud-aio-redis',
    'password' => '****',
    'port' => 6379,
  ),
  'overwritehost' => 'nextcloud.customdomain.cz',
  'overwriteprotocol' => 'https',
  'passwordsalt' => '****',
  'secret' => '****',
  'trusted_domains' => 
  array (
    0 => 'localhost',
    1 => 'nextcloud.customdomain.cz',
  ),
  'datadirectory' => '/mnt/ncdata',
  'dbtype' => 'pgsql',
  'version' => '27.1.5.1',
  'overwrite.cli.url' => 'https://nextcloud.customdomain.cz/',
  'dbname' => 'nextcloud_database',
  'dbhost' => 'nextcloud-aio-database',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'oc_nextcloud',
  'dbpassword' => '****',
  'installed' => true,
  'instanceid' => '****',
  'maintenance' => false,
  'loglevel' => '2',
  'log_type' => 'file',
  'logfile' => '/var/www/html/data/nextcloud.log',
  'log_rotate_size' => '10485760',
  'log.condition' => 
  array (
    'apps' => 
    array (
      0 => 'admin_audit',
    ),
  ),
  'preview_max_x' => '2048',
  'preview_max_y' => '2048',
  'jpeg_quality' => '60',
  'enabledPreviewProviders' => 
  array (
    1 => 'OC\\Preview\\Image',
    2 => 'OC\\Preview\\MarkDown',
    3 => 'OC\\Preview\\MP3',
    4 => 'OC\\Preview\\TXT',
    5 => 'OC\\Preview\\OpenDocument',
    6 => 'OC\\Preview\\Movie',
    7 => 'OC\\Preview\\Krita',
    0 => 'OC\\Preview\\Imaginary',
  ),
  'enable_previews' => true,
  'upgrade.disable-web' => true,
  'mail_smtpmode' => 'smtp',
  'trashbin_retention_obligation' => 'auto, 30',
  'versions_retention_obligation' => 'auto, 30',
  'activity_expire_days' => '30',
  'simpleSignUpLink.shown' => false,
  'share_folder' => '/Shared',
  'one-click-instance.link' => 'https://nextcloud.com/all-in-one/',
  'upgrade.cli-upgrade-link' => 'https://github.com/nextcloud/all-in-one/discussions/2726',
  'updatedirectory' => '/nc-updater',
  'allow_local_remote_servers' => true,
  'davstorage.request_timeout' => 3600,
  'htaccess.RewriteBase' => '/',
  'dbpersistent' => false,
  'files_external_allow_create_new_local' => false,
  'trusted_proxies' => 
  array (
    0 => '127.0.0.1',
    1 => '::1',
  ),
  'preview_imaginary_url' => 'http://nextcloud-aio-imaginary:9000',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'info',
  'mail_domain' => 'customdomain.cz',
  'mail_smtphost' => 'smtp.customdomain.cz',
  'mail_smtpport' => '587',
  'mail_smtpauth' => 1,
  'mail_smtpname' => '****',
  'mail_smtppassword' => '****',
  "mail_smtpstreamoptions" => array(
    'ssl' => array(
        'allow_self_signed' => true,
        'verify_peer' => false,
        'verify_peer_name' => false
    )
  ),
);

The output of lighttpd log serving as reverse proxy:

lighttpd[1]: (gw_backend.c.285) establishing connection failed: socket: tcp:custom_ip:custom_port: Connection refused
lighttpd[1]: (gw_backend.c.995) all handlers for /? on  are down.
lighttpd[1]: 192.168.53.220 nextcloud.customdomain.cz - [13/Jan/2024:20:15:48 +0000] "GET / HTTP/2.0" 503 361 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"
lighttpd[1]: 192.168.53.220 nextcloud.customdomain.cz - [13/Jan/2024:20:15:49 +0000] "GET /index.php/apps/files/preview-service-worker.js HTTP/2.0" 503 361 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"
lighttpd[1]: 192.168.53.220 nextcloud.customdomain.cz - [13/Jan/2024:20:15:49 +0000] "GET / HTTP/2.0" 503 361 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"
lighttpd[1]: 192.168.53.220 nextcloud.customdomain.cz - [13/Jan/2024:20:15:50 +0000] "GET /index.php/apps/files/preview-service-worker.js HTTP/2.0" 503 361 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"
lighttpd[1]: (gw_backend.c.362) gw-server re-enabled: tcp:custom_ip:custom_port custom_ip custom_port 
lighttpd[1]: (gw_backend.c.285) establishing connection failed: socket: tcp:custom_ip:custom_port: Connection refused
lighttpd[1]: (gw_backend.c.995) all handlers for /? on  are down.
lighttpd[1]: 192.168.53.220 nextcloud.customdomain.cz - [13/Jan/2024:20:15:51 +0000] "GET / HTTP/2.0" 503 361 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"
lighttpd[1]: 192.168.53.220 nextcloud.customdomain.cz - [13/Jan/2024:20:15:52 +0000] "GET /index.php/apps/files/preview-service-worker.js HTTP/2.0" 503 361 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/115.0"
lighttpd[1]: (gw_backend.c.362) gw-server re-enabled: tcp:custom_ip:custom_port custom_ip custom_port 
lighttpd[1]: (gw_backend.c.285) establishing connection failed: socket: tcp:custom_ip:custom_port: Connection refused
lighttpd[1]: (gw_backend.c.995) all handlers for /apps/richdocuments/settings/fonts.json? on  are down.
lighttpd[1]: 172.20.0.1 nextcloud.customdomain.cz - [13/Jan/2024:20:15:56 +0000] "GET /apps/richdocuments/settings/fonts.json HTTP/1.1" 503 361 "-" "COOLWSD HTTP Agent 23.05.6.4"
lighttpd[1]: 192.168.53.220 nextcloud.customdomain.cz - [13/Jan/2024:20:15:58 +0000] "PRI * HTTP/2.0" 100 2548 "-" "-"
lighttpd[1]: (gw_backend.c.362) gw-server re-enabled: tcp:custom_ip:custom_port custom_ip custom_port

Any help is will be appreciated, glad to answer any questions.

Hi, can you post your reverse proxy config here and the output of sudo docker inspect nextcloud-aio-apache ?

Reverse proxy (lighttpd):

$HTTP["host"] =~ "(^|\.)nextcloud.customdomain.cz$" {
        $SERVER["socket"] == ":443" {
        proxy.server = (
            "" => (
                (
                    "host" => "nextcloud.customdomain.cz",
                    "port" => customport,
                )
            ),
        )
        }
}

sudo docker inspect nextcloud-aio-apache

[
    {
        "Id": "61f3cfafc364d59ddef7056b70ffb39997c92567b811b7d4e2f4ff2f8d723b4d",
        "Created": "2024-01-13T19:32:58.554324987Z",
        "Path": "/start.sh",
        "Args": [
            "/usr/bin/supervisord",
            "-c",
            "/supervisord.conf"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 575927,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2024-01-14T12:59:57.585272939Z",
            "FinishedAt": "2024-01-13T19:58:13.083713556Z",
            "Health": {
                "Status": "healthy",
                "FailingStreak": 0,
                "Log": [
                    {
                        "Start": "2024-01-14T14:00:27.585535176+01:00",
                        "End": "2024-01-14T14:00:27.640588235+01:00",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2024-01-14T14:00:57.64322568+01:00",
                        "End": "2024-01-14T14:00:57.716687633+01:00",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2024-01-14T14:01:27.719770996+01:00",
                        "End": "2024-01-14T14:01:27.796463781+01:00",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2024-01-14T14:01:57.79933976+01:00",
                        "End": "2024-01-14T14:01:57.874149081+01:00",
                        "ExitCode": 0,
                        "Output": ""
                    },
                    {
                        "Start": "2024-01-14T14:02:27.876887856+01:00",
                        "End": "2024-01-14T14:02:27.947737698+01:00",
                        "ExitCode": 0,
                        "Output": ""
                    }
                ]
            }
        },
        "Image": "sha256:e5b984fb183ba178d1c8e7d23d47c03a28e39135e174acd28ff7b1d36306fc65",
        "ResolvConfPath": "/var/snap/docker/common/var-lib-docker/containers/61f3cfafc364d59ddef7056b70ffb39997c92567b811b7d4e2f4ff2f8d723b4d/resolv.conf",
        "HostnamePath": "/var/snap/docker/common/var-lib-docker/containers/61f3cfafc364d59ddef7056b70ffb39997c92567b811b7d4e2f4ff2f8d723b4d/hostname",
        "HostsPath": "/var/snap/docker/common/var-lib-docker/containers/61f3cfafc364d59ddef7056b70ffb39997c92567b811b7d4e2f4ff2f8d723b4d/hosts",
        "LogPath": "/var/snap/docker/common/var-lib-docker/containers/61f3cfafc364d59ddef7056b70ffb39997c92567b811b7d4e2f4ff2f8d723b4d/61f3cfafc364d59ddef7056b70ffb39997c92567b811b7d4e2f4ff2f8d723b4d-json.log",
        "Name": "/nextcloud-aio-apache",
        "RestartCount": 0,
        "Driver": "btrfs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "nextcloud_aio_nextcloud:/var/www/html:ro",
                "nextcloud_aio_apache:/mnt/data:rw"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "nextcloud-aio",
            "PortBindings": {
                "customport/tcp": [
                    {
                        "HostIp": "127.0.0.1",
                        "HostPort": "customport"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "unless-stopped",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": [
                "NET_RAW"
            ],
            "CgroupnsMode": "private",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": true,
            "SecurityOpt": null,
            "Tmpfs": {
                "/home/www-data": "",
                "/tmp": "",
                "/usr/local/apache2/logs": "",
                "/var/log/supervisord": "",
                "/var/run/supervisord": ""
            },
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": null,
            "PidsLimit": null,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ],
            "Init": true
        },
        "GraphDriver": {
            "Data": null,
            "Name": "btrfs"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "nextcloud_aio_nextcloud",
                "Source": "/var/snap/docker/common/var-lib-docker/volumes/nextcloud_aio_nextcloud/_data",
                "Destination": "/var/www/html",
                "Driver": "local",
                "Mode": "ro",
                "RW": false,
                "Propagation": ""
            },
            {
                "Type": "volume",
                "Name": "nextcloud_aio_apache",
                "Source": "/var/snap/docker/common/var-lib-docker/volumes/nextcloud_aio_apache/_data",
                "Destination": "/mnt/data",
                "Driver": "local",
                "Mode": "rw",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "61f3cfafc364",
            "Domainname": "",
            "User": "www-data",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "customport/tcp": {},
                "80/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "NC_DOMAIN=nextcloud.customdomain.cz",
                "NEXTCLOUD_HOST=nextcloud-aio-nextcloud",
                "COLLABORA_HOST=nextcloud-aio-collabora",
                "TALK_HOST=nextcloud-aio-talk",
                "APACHE_PORT=customport",
                "ONLYOFFICE_HOST=nextcloud-aio-onlyoffice",
                "TZ=Etc/UTC",
                "APACHE_MAX_SIZE=10737418240",
                "APACHE_MAX_TIME=3600",
                "NOTIFY_PUSH_HOST=nextcloud-aio-notify-push",
                "PATH=/usr/local/apache2/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "HTTPD_PREFIX=/usr/local/apache2",
                "HTTPD_VERSION=2.4.58",
                "HTTPD_SHA256=fa16d72a078210a54c47dd5bef2f8b9b8a01d94909a51453956b3ec6442ea4c5",
                "HTTPD_PATCHES="
            ],
            "Cmd": [
                "/usr/bin/supervisord",
                "-c",
                "/supervisord.conf"
            ],
            "Healthcheck": {
                "Test": [
                    "CMD-SHELL",
                    "/healthcheck.sh"
                ]
            },
            "Image": "nextcloud/aio-apache:latest",
            "Volumes": {
                "/mnt/data": {}
            },
            "WorkingDir": "/usr/local/apache2",
            "Entrypoint": [
                "/start.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.centurylinklabs.watchtower.enable": "false"
            },
            "StopSignal": "SIGWINCH"
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "826f713e6e9d005cb0ea5bd616021826d1023734e298d033f2096009be691b62",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "customport/tcp": [
                    {
                        "HostIp": "127.0.0.1",
                        "HostPort": "customport"
                    }
                ],
                "80/tcp": null
            },
            "SandboxKey": "/run/snap.docker/netns/826f713e6e9d",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "nextcloud-aio": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "61f3cfafc364"
                    ],
                    "NetworkID": "fb52dccabb9d1cc9d09f090ca8944c4ba60fcd6b4ca1245f1fc5d780a616e2d5",
                    "EndpointID": "985ad7a0b339fe8ebfd7c501625982dd74b9c7ce56d40974eae7ff6c5950f1b5",
                    "Gateway": "172.23.0.1",
                    "IPAddress": "172.23.0.9",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:17:00:09",
                    "DriverOpts": null
                }
            }
        }
    }
]

So am I right that you run lighttpd as service on the host or is it running in a container?
Also, can you use "host" => "127.0.0.1", if you should be running it on the host instead of "host" => "nextcloud.customdomain.cz", ?

Lighttpd is running in a docker container

Ah I see. Then please follow all-in-one/reverse-proxy.md at main · nextcloud/all-in-one · GitHub

[EDIT]: I’ve marked this as solution since it solved the original error. For the follow up error the fix is a few comments lower.

So the problem was in APACHE_IP_BINDING=127.0.0.1 in my docker compose file. It has been set this for at least a few weeks, but the update probably propagated this change, or maybe I tried recreating the master container after the Nextcloud became unresponsive after the update.
I’ve set APACHE_IP_BINDING=0.0.0.0 and recreated the master container. Then I’ve stopped and started all the other containers using a button in the web UI and the change was propagated to the Apache container and it is now reachable. :white_check_mark:

However, I am still experiencing problems. Now I am getting:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

The synchronization of files works, from mobile app it looks like everything’s ok.
Seems like my PHP container is fighting over port 9000 with nextcloud-aio-nextcloud container. I will investigate and report more here.

Thanks for the help!

Ok, it was not the ports. No idea what’s wrong. Apache logs don’t show any errors.

nextcloud-aio-nextcloud logs don’t show anything either:

 2024-01-14 21:16:47.96823+00
(1 row)

+ '[' -f /dev-dri-group-was-added ']'
++ find /dev -maxdepth 1 -mindepth 1 -name dri
+ '[' -n '' ']'
+ set +x
Configuring Redis as session handler...
Applying one-click-instance settings...
System config value one-click-instance set to boolean true
System config value one-click-instance.user-limit set to integer 100
System config value one-click-instance.link set to string https://nextcloud.com/all-in-one/
support already enabled
Adjusting log files...
System config value upgrade.cli-upgrade-link set to string https://github.com/nextcloud/all-in-one/discussions/2726
System config value logfile set to string /var/www/html/data/nextcloud.log
Config value logfile for app admin_audit set to /var/www/html/data/audit.log
System config value updatedirectory set to string /nc-updater
Applying network settings...
System config value allow_local_remote_servers set to boolean true
System config value davstorage.request_timeout set to integer 3600
System config value trusted_domains => 1 set to string nextcloud.customdomain.cz
System config value overwrite.cli.url set to string https://nextcloud.customdomain.cz/
System config value htaccess.RewriteBase set to string /
.htaccess has been updated
System config value dbpersistent set to boolean false
System config value files_external_allow_create_new_local set to boolean false
System config value trusted_proxies => 0 set to string 127.0.0.1
System config value trusted_proxies => 1 set to string ::1
Config value base_endpoint for app notify_push set to https://nextcloud.customdomain.cz/push
Config value wopi_url for app richdocuments set to https://nextcloud.customdomain.cz/
No ipv6-address found for nextcloud.customdomain.cz.
Config value wopi_allowlist for app richdocuments set to mypublicip,127.0.0.1/8,192.168.0.0/16,172.16.0.0/12,10.0.0.0/8,fd00::/8,::1
System config value enabledPreviewProviders => 0 set to string OC\Preview\Imaginary
System config value preview_imaginary_url set to string http://nextcloud-aio-imaginary:9000
+ '[' true = true ']'
+ '[' customport = 443 ']'
+ '[' 0.0.0.0 = 127.0.0.1 ']'
+ '[' 0.0.0.0 = ::1 ']'
+ set +x
[14-Jan-2024 21:16:59] NOTICE: fpm is running, pid 280
[14-Jan-2024 21:16:59] NOTICE: ready to handle connections
Activating Collabora config...
Activated any config changes

Does anyone have an idea where to look?

Hm… The logs seem fine. what exactly does not work?

When I go to https://nextcloud.customdomain.cz/login I get

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

Which is not very specific

You will need to look into the nextcloud logs then which are usually stored in /var/lib/docker/volumes/nextcloud_aio_nextcloud/_data/data/nextcloud.log

Thanks for pointing out yet another source of logs.

After going through the logs I have exactly this issue: [Bug]: Upgrade to 27.1.2.1 Not working: `file_get_contents(/var/www/html/data/appdata_oczcy358mcjy/js/core/merged-template-prepend.js.deps): Failed to open stream: No such file or directory at /var/www/html/lib/private/Files/Storage/Local.php#305` · Issue #40886 · nextcloud/server · GitHub

And there is a fix mentioned, which worked for me as far as I can tell:

docker exec --user www-data nextcloud-aio-nextcloud php occ maintenance:repair
docker exec --user www-data nextcloud-aio-nextcloud php occ files:scan-app-data

1 Like

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.