Nextcloud suddenly failing with "Internal Server Error"

Hi,
my Nextcloud was running for a few Months now without any issues. But today something is broken. When i try to load the web interface i only get:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

Since i use docker i checked all logs with docker and to me it seems that only one log is showing relevant information:

docker logs nextcloud-db-1 
2024-03-24 10:47:47+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.16+maria~ubu2004 started.
2024-03-24 10:47:47+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
/usr/local/bin/docker-entrypoint.sh: line 644: /usr/local/bin/gosu: Input/output error
2024-03-24 10:47:48+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.6.16+maria~ubu2004 started.
2024-03-24 10:47:48+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
/usr/local/bin/docker-entrypoint.sh: line 644: /usr/local/bin/gosu: Input/output error
[...]

when i google this i couldn’t find anything usefull so far. I tried to restart the docker container which is built by these docker-compose.yml and nginx.conf files:

worker_processes auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    # Prevent nginx HTTP Server Detection
    server_tokens   off;

    keepalive_timeout  65;

    #gzip  on;

    upstream php-handler {
        server app:9000;
    }

    server {
        listen 80;

        # HSTS settings
        # WARNING: Only add the preload option once you read about
        # the consequences in https://hstspreload.org/. This option
        # will add the domain to a hardcoded list that is shipped
        # in all major browsers and getting removed from this list
        # could take several months.
        add_header Strict-Transport-Security "max-age=15768000; includeSubDomains;" always;

        # set max upload size
        client_max_body_size 512M;
        fastcgi_buffers 64 4K;

        # Enable gzip but do not remove ETag headers
        gzip on;
        gzip_vary on;
        gzip_comp_level 4;
        gzip_min_length 256;
        gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
        gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

        # Pagespeed is not supported by Nextcloud, so if your server is built
        # with the `ngx_pagespeed` module, uncomment this line to disable it.
        #pagespeed off;

        # HTTP response headers borrowed from Nextcloud `.htaccess`
        add_header Referrer-Policy                      "no-referrer"       always;
        add_header X-Content-Type-Options               "nosniff"           always;
        add_header X-Frame-Options                      "SAMEORIGIN"        always;
        add_header X-Permitted-Cross-Domain-Policies    "none"              always;
        add_header X-Robots-Tag                         "noindex, nofollow" always;
        add_header X-XSS-Protection                     "1; mode=block"     always;

        # Remove X-Powered-By, which is an information leak
        fastcgi_hide_header X-Powered-By;

        # Path to the root of your installation
        root /var/www/html;

        # Specify how to handle directories -- specifying `/index.php$request_uri`
        # here as the fallback means that Nginx always exhibits the desired behaviour
        # when a client requests a path that corresponds to a directory that exists
        # on the server. In particular, if that directory contains an index.php file,
        # that file is correctly served; if it doesn't, then the request is passed to
        # the front-end controller. This consistent behaviour means that we don't need
        # to specify custom rules for certain paths (e.g. images and other assets,
        # `/updater`, `/ocm-provider`, `/ocs-provider`), and thus
        # `try_files $uri $uri/ /index.php$request_uri`
        # always provides the desired behaviour.
        index index.php index.html /index.php$request_uri;

        # Rule borrowed from `.htaccess` to handle Microsoft DAV clients
        location = / {
            if ( $http_user_agent ~ ^DavClnt ) {
                return 302 https://$host/remote.php/webdav/$is_args$args;
            }
        }

        location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }

        # Make a regex exception for `/.well-known` so that clients can still
        # access it despite the existence of the regex rule
        # `location ~ /(\.|autotest|...)` which would otherwise handle requests
        # for `/.well-known`.
        location ^~ /.well-known {
            # The rules in this block are an adaptation of the rules
            # in `.htaccess` that concern `/.well-known`.

            location = /.well-known/carddav { return 301 https://$host/remote.php/dav; }
            location = /.well-known/caldav  { return 301 https://$host/remote.php/dav; }

            location /.well-known/acme-challenge    { try_files $uri $uri/ =404; }
            location /.well-known/pki-validation    { try_files $uri $uri/ =404; }

            # Let Nextcloud's API for `/.well-known` URIs handle all other
            # requests by passing them to the front-end controller.
            return 301 https://$host/index.php$request_uri;
        }

        # Rules borrowed from `.htaccess` to hide certain paths from clients
        location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)(?:$|/)  { return 404; }
        location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console)                { return 404; }

        # Ensure this block, which passes PHP files to the PHP process, is above the blocks
        # which handle static assets (as seen below). If this block is not declared first,
        # then Nginx will encounter an infinite rewriting loop when it prepends `/index.php`
        # to the URI, resulting in a HTTP 500 error response.
        location ~ \.php(?:$|/) {
            # Required for legacy support
            rewrite ^/(?!index|remote|public|cron|core\/ajax\/update|status|ocs\/v[12]|updater\/.+|oc[ms]-provider\/.+|.+\/richdocumentscode\/proxy) /index.php$request_uri;

            fastcgi_split_path_info ^(.+?\.php)(/.*)$;
            set $path_info $fastcgi_path_info;

            try_files $fastcgi_script_name =404;

            include fastcgi_params;
            fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
            fastcgi_param PATH_INFO $path_info;
            #fastcgi_param HTTPS on;

            fastcgi_param modHeadersAvailable true;         # Avoid sending the security headers twice
            fastcgi_param front_controller_active true;     # Enable pretty urls
            fastcgi_pass php-handler;

            fastcgi_intercept_errors on;
            fastcgi_request_buffering off;
        }

        location ~ \.(?:css|js|svg|gif)$ {
            try_files $uri /index.php$request_uri;
            expires 6M;         # Cache-Control policy borrowed from `.htaccess`
            access_log off;     # Optional: Don't log access to assets
        }

        location ~ \.woff2?$ {
            try_files $uri /index.php$request_uri;
            expires 7d;         # Cache-Control policy borrowed from `.htaccess`
            access_log off;     # Optional: Don't log access to assets
        }

        # Rule borrowed from `.htaccess`
        location /remote {
            return 301 /remote.php$request_uri;
        }

        location / {
            try_files $uri $uri/ /index.php$request_uri;
        }
    }
}
version: '2'

services:
  db:
    image: mariadb:10.6
    restart: always
    command: --transaction-isolation=READ-COMMITTED --log-bin=binlog --binlog-format=ROW
    volumes:
      - db:/var/lib/mysql
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud

  cron:
    image: nextcloud:fpm
    restart: always
    volumes:
      - nextcloud:/var/www/html:z
    entrypoint: /cron.sh
    depends_on:
      - db
      - redis

  redis:
    image: redis:alpine
    restart: always

  app:
    image: nextcloud:fpm
    restart: always
    links:
      - db
      - redis
    volumes:
      - nextcloud:/var/www/html
      - share:/var/www/html/share
    environment:
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
      - MYSQL_HOST=db
      - REDIS_HOST=redis
      - TRUSTED_PROXIES=web
      - APACHE_DISABLE_REWRITE_IP = true
      - NEXTCLOUD_TRUSTED_DOMAINS=my.domain.de   
      - OVERWRITEHOST=my.domain.de
      - OVERWRITEPROTOCOL=https
      - OVERWRITECLIURL=https://my.domain.de
      - bulkupload.enabled=false

  web:
    image: nginx
    restart: always
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.nextcloud.rule=Host(`my.domain.de`)"
      - "traefik.http.services.nextcloud.loadbalancer.server.port=80"
      - "traefik.http.routers.nextcloud.tls=true"
      - "traefik.http.routers.nextcloud.tls.certresolver=letsencrypt"
      - "traefik.http.routers.nextcloud.entrypoints=websecure"
      - "traefik.nextcloud.redirect.permanent='true'"
      - "traefik.nextcloud.redirect.regex='https://(.*)/.well-known/(?:card|cal)dav'"
      - "traefik.nextcloud.redirect.replacement='https://$$1/remote.php/dav'"
    links:
      - app
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    volumes_from:
      - app

networks:
  default:
    external: true
    name: external

volumes:
  nextcloud:
    driver_opts:
      type: "zfs"
      device: "karla/nextcloud/nextcloud"
  db:
    driver_opts:
      type: "zfs"
      device: "karla/nextcloud/db"
  share:
    driver_opts:
      type: "zfs"
      device: "karla/share"

I am running Ubuntu 22.04LTS which is currently up to date. Any Ideas?
Thanks

so i switched from

image: mariadb:10.6

to

image: mariadb:latest

stoped the old container with docker stop ID and ran docker compose up -d afterwards which seems to have fixed it. Still not sure what actually happened or why i suddenly stopped working. :worried:

1 Like

Not sure offhand, but looks like your 10.6 image was slightly outdated (10.6.17 is the latest in the 10.6 LTS tree).

A docker compose pull would have brought that up-to-date, but not sure if it would have addressed whatever you encountered. Whatever it was, it looks like it was specific to your db container and not Nextcloud itself.

As an aside: Running latest of MariaDB bumps you up to the latest short-term release version and also technically means you’re running a version that Nextcloud doesn’t consider tested/supported with it. You’re likely fine but just something to be aware of for the future.

There are ways to downgrade the db if concerned / you’d prefer to get back to a supported environment. Safest approach is generally a db dump (export) then restore (import) into an appropriate container version. Basically the same process as used for a standard backup/restore.

P.S. Specifics of the internal server error from Nextcloud will be in your nextcloud.log, but chances are it’s just going to say it couldn’t connect to the db server since it appears your en container would have been offline.