Internal Server Error - Nothing apparently wrong in the logs

ℹ️ Support
, ,
1

Hi All,

My Nextcloud instance has suddenly stopped working. I have not made any major (or minor, for that matter) changes to the configuration or instances. I am running on a Lenovo Thin client with Nextcloud and MariaDB in docker containers spun up with docker-compose.

When accessing the URL, I see this:

image
image1246×165 4.9 KB

The server log does not seem to give anything of substance.

And the DB logs also are not showing anything major from my reading.

2024-03-28 14:29:50+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.2.2+maria~ubu2204 started.
2024-03-28 14:29:50+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-03-28 14:29:50+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:11.2.2+maria~ubu2204 started.
2024-03-28 14:29:51+00:00 [Note] [Entrypoint]: MariaDB upgrade not required
2024-03-28 14:29:51 0 [Note] Starting MariaDB 11.2.2-MariaDB-1:11.2.2+maria~ubu2204 source revision 929532a9426d085111c24c63de9c23cc54382259 as process 1
2024-03-28 14:29:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-03-28 14:29:51 0 [Note] InnoDB: Number of transaction pools: 1
2024-03-28 14:29:51 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2024-03-28 14:29:51 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
2024-03-28 14:29:51 0 [Warning] mariadbd: io_uring_queue_init() failed with ENOMEM: try larger memory locked limit, ulimit -l, or https://mariadb.com/kb/en/systemd/#configuring-limitmemlock under systemd (262144 bytes required)
2024-03-28 14:29:51 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF
2024-03-28 14:29:51 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB
2024-03-28 14:29:51 0 [Note] InnoDB: Completed initialization of buffer pool
2024-03-28 14:29:51 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes)
2024-03-28 14:29:51 0 [Note] InnoDB: End of log at LSN=47763
2024-03-28 14:29:51 0 [Note] InnoDB: Opened 3 undo tablespaces
2024-03-28 14:29:51 0 [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active.
2024-03-28 14:29:51 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2024-03-28 14:29:51 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2024-03-28 14:29:51 0 [Note] InnoDB: log sequence number 47763; transaction id 14
2024-03-28 14:29:51 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2024-03-28 14:29:51 0 [Note] Plugin 'FEEDBACK' is disabled.
2024-03-28 14:29:51 0 [Note] Plugin 'wsrep-provider' is disabled.
2024-03-28 14:29:51 0 [Note] InnoDB: Buffer pool(s) load completed at 240328 14:29:51
2024-03-28 14:29:51 0 [Note] Server socket created on IP: '0.0.0.0'.
2024-03-28 14:29:51 0 [Note] Server socket created on IP: '::'.
2024-03-28 14:29:51 0 [Note] mariadbd: Event Scheduler: Loaded 0 events
2024-03-28 14:29:51 0 [Note] mariadbd: ready for connections.
Version: '11.2.2-MariaDB-1:11.2.2+maria~ubu2204'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution

Here is the compose file:

version: '2'

services:

  nextcloud:
    container_name: nextcloud
    restart: unless-stopped
    image: kilrah/nextcloud-ffmpeg
    ports:
      - 80:80
    environment:
      PHP_MEMORY_LIMIT: 5G
      PHP_UPLOAD_LIMIT: 5G
    volumes:
      - /yacht/AppData/Data:/var/www/html/apps
      - /yacht/AppData/Config:/var/www/html/config
      - /mnt/Ncloud/Data:/var/www/html/data
    depends_on:
      - db

  db:
    container_name: maria-db
    restart: unless-stopped
    image: mariadb
    command: --innodb-read-only-compressed=OFF #Maria-DB fix
    environment:
      MYSQL_ROOT_PASSWORD: REDACT
      MYSQL_DATABASE: REDACT
      MYSQL_USER: REDACT
      MYSQL_PASSWORD: REDACT
    volumes:
      - /yacht/AppData/Data:/var/lib/mysql
  
  adminer:
    container_name: adminer
    restart: always
    image: adminer
    ports:
      - 7777:8080

This instance uses a private network bridge between itself and the DB. I have verified that it is possible to ping the DB container from the Nextcloud one.

Or for longer, use three backticks above and below the code snippet:

Nextcloud version: 28.0.3.2 - kilrah/nextcloud-ffmpeg
Operating system and version: Ubuntu 20.04.6 LTS
Apache or nginx version: NOT SURE - NEED HELP FINDING
PHP version: 8.2.16

Is this the first time you’ve seen this error?: Y

Steps to replicate it:

  1. Start Nextcloud container as normal.
  2. Attempt to access webUI.
  3. Note error.

The output of your config.php file:

<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'upgrade.disable-web' => true,
  'instanceid' => '************',
  'passwordsalt' => '*********************',
  'secret' => '*********************',
  'trusted_domains' =>
  array (
    0 => '10.0.5.6',
    1 => '***************',
    2 => '10.0.50.5',
  ),
  'overwriteprotocol' => 'https',
  'overwrite.cli.url' => '*******************',
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '28.0.1.1',
  'dbname' => 'ncdb',
  'dbhost' => 'maria-db',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'nextcloud',
  'dbpassword' => 'nextcloud',
  'installed' => true,
  'mail_smtpmode' => 'smtp',
  'mail_smtpauth' => 1,
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'nextcloud',
  'mail_domain' => '************',
  'mail_smtphost' => '***********',
  'mail_smtpport' => '465',
  'mail_smtpsecure' => 'ssl',
  'mail_smtpname' => '****************',
  'mail_smtppassword' => '*************',
  'enable_previews' => true,
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\TXT',
    1 => 'OC\\Preview\\MarkDown',
    3 => 'OC\\Preview\\Image',
    4 => 'OC\\Preview\\Photoshop',
    5 => 'OC\\Preview\\TIFF',
    6 => 'OC\\Preview\\SVG',
    7 => 'OC\\Preview\\Font',
    8 => 'OC\\Preview\\MP3',
    9 => 'OC\\Preview\\Movie',
    10 => 'OC\\Preview\\MKV',
    11 => 'OC\\Preview\\MP4',
    12 => 'OC\\Preview\\AVI',
  ),
  'preview_max_memory' => 1024,
  'preview_max_x' => NULL,
  'preview_max_y' => NULL,
  'preview_max_filesize_image' => 400,
  'app_install_overwrite' =>
  array (
    0 => 'memories',
  ),
  'memories.exiftool' => '/var/www/html/custom_apps/memories/bin-ext/exiftool-amd64-glibc',
  'memories.vod.path' => '/var/www/html/custom_apps/memories/bin-ext/go-vod-amd64',
  'memories.vod.ffmpeg' => '/usr/bin/ffmpeg',
  'memories.vod.ffprobe' => '/usr/bin/ffprobe',
  'maintenance' => false,
);

The output of your Apache/nginx/system log in /var/log/apache2:

EMPTY

Output errors in nextcloud.log in /var/www/

Initializing nextcloud 28.0.3.2 ...
New nextcloud instance
Initializing finished
=> Searching for scripts (*.sh) to run, located in the folder: /docker-entrypoint-hooks.d/before-starting
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.20.0.4. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.20.0.4. Set the 'ServerName' directive globally to suppress this message
[Thu Mar 28 14:29:58.570621 2024] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.57 (Debian) PHP/8.2.16 configured -- resuming normal operations
[Thu Mar 28 14:29:58.570687 2024] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
74.114.REDACT - rjDipcord [28/Mar/2024:14:30:05 +0000] "GET /index.php/204 HTTP/1.1" 500 702 "-" "Mozilla/5.0 (Windows) mirall/3.12.1stable-Win64 (build 20240306) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
74.114.REDACT - - [28/Mar/2024:14:30:23 +0000] "GET / HTTP/1.1" 500 702 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0"
74.114.REDACT - - [28/Mar/2024:14:30:23 +0000] "GET /favicon.ico HTTP/1.1" 404 644 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0"
74.114.REDACT - - [28/Mar/2024:14:30:24 +0000] "GET /index.php/apps/files/preview-service-worker.js HTTP/1.1" 500 702 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0"
74.114.REDACT - rjDipcord [28/Mar/2024:14:31:07 +0000] "GET /index.php/204 HTTP/1.1" 500 702 "-" "Mozilla/5.0 (Windows) mirall/3.12.1stable-Win64 (build 20240306) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
74.114.REDACT - rjDipcord [28/Mar/2024:14:32:09 +0000] "GET /index.php/204 HTTP/1.1" 500 702 "-" "Mozilla/5.0 (Windows) mirall/3.12.1stable-Win64 (build 20240306) (Nextcloud, windows-10.0.19045 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
2

Based on your config, your log file will be found here (nextcloud.log). It will have the details of the 500 error.

3

Thanks for helping me find that. Here are the last few lines from that log. Can you help me decipher it? It seems to be DB related.

Paste

Edit: Oh god I think I see what’s happening. It can’t find the nextcloud database. I hope it’s not gone.

Edit2: Yep… the db called “ncdb” is not in the table. How could this have happened? Is there anything I can do in adminer to find the reason it occurred?

4

Yep… the db called “ncdb” is not in the table. How could this have happened? Is there anything I can do in adminer to find the reason it occurred?

Looks like you have a dedicated docker container or host. I’d check the logs for that container. Might also be worth confirming that container has a persistent data volume of some sort associated with it. Feel free to post your Compose file if you are so inclined.

Do you maintain backups of your data and database?

5

Compose file is above.

And normally I have backups running, but this is a machine I’ve been just playing around with for a bit. So I had yet to implement the backup strategy. Literally writing that script as we speak.

I guess that’s what I get. I should know better.

At least the data directory is backed up since it’s on an entirely different machine with parity disks.

6 
  nextcloud:
    [...]
    volumes:
      - /yacht/AppData/Data:/var/www/html/apps
      - /yacht/AppData/Config:/var/www/html/config
      - /mnt/Ncloud/Data:/var/www/html/data
[...]

  db:
    [...]
    volumes:
      - /yacht/AppData/Data:/var/lib/mysql

So two things. First one is probably what happened to your database:

  • You’re using /yacht/AppData/Data twice:
    • For Nextcloud:
      /yacht/AppData/Data:/var/www/html/apps
    • For MariaDB:
      /yacht/AppData/Data:/var/lib/mysql

That’s not gonna work! Also, since the /var/www/html/apps folder happens to be the one used for shipped apps in Nextcloud Server, the contents are wiped at every update (which with your Compose config also contained your MariaDB database data files…).

The other issue, which is likely unrelated to your current situation, is that you’re not using a named volume for Nextcloud’s /var/www/html [1][2]

[1] GitHub - nextcloud/docker: ⛴ Docker image of Nextcloud
[2] GitHub - nextcloud/docker: ⛴ Docker image of Nextcloud

7

Well thank you for pointing that out!

Its a good thing I hadn’t done a whole lot with this server yet. I guess i’ll be updating my docker-compose and starting from scratch again. But also, ensuring my backups are done correctly.

1 Like
8

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.