Database not initiating after NC29 update

svethitov · June 19, 2024, 12:54pm

Support intro

Sorry to hear you’re facing problems

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can

Hi community,
I am having troubles with the database container after the update to AIO v9.0.1.

Nextcloud version (eg, 29.0.5): NA
Operating system and version (eg, Ubuntu 29.04): Debian GNU/Linux 12 (bookworm)
Apache or nginx version (eg, Apache 2.4.25): NA
PHP version (eg, 8.3): NA

The issue you are facing:
After the update to v9.0.1 the database container is not getting to a healthy state. From the logs it appears that the database never gets initiated. I have already tried restoring from a previous back up. I even tried resetting the entire instance by deleting all of the containers and volumes and then restoring from backup.

...
+ psql -d postgresql://oc_nextcloud:<pg_pass>@127.0.0.1:11000/nextcloud_database -c 'select now()'
psql: error: connection to server at "127.0.0.1", port 11000 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
+ echo 'Waiting for the database to start.'
Waiting for the database to start.
+ sleep 5
...

root@docker-host-01:~# docker exec -it nextcloud-aio-database ps aux
PID   USER     TIME  COMMAND
    1 postgres  0:00 /sbin/docker-init -- /start.sh
    7 postgres  0:04 {start.sh} /bin/bash /start.sh
   10 postgres  0:01 tee -i /mnt/data/database-import.log
12613 postgres  0:00 /bin/bash
12629 postgres  0:00 sleep 5
12630 postgres  0:00 ps aux

Is this the first time you’ve seen this error? (Y/N): Yes

I wonder if it has anything to do with the update to PostgreSQL but I have no clue where to start looking.

szaimen · June 19, 2024, 12:56pm

Hi, after restoring a backup and starting the containers, can you post the database container logs here?

svethitov · June 19, 2024, 1:39pm

Hi @szaimen

These are the first 50 lines. After that it goes on and on with the 'Waiting for the database to start`.

root@docker-host-01:~# head -n 50 db_log.txt
Restoring from database dump.
+ rm -rf /var/lib/postgresql/data/PG_VERSION /var/lib/postgresql/data/base /var/lib/postgresql/data/global /var/lib/postgresql/data/pg_commit_ts /var/lib/postgresql/data/pg_dynshmem /var/lib/postgresql/data/pg_hba.conf /var/lib/postgresql/data/pg_ident.conf /var/lib/postgresql/data/pg_logical /var/lib/postgresql/data/pg_multixact /var/lib/postgresql/data/pg_notify /var/lib/postgresql/data/pg_replslot /var/lib/postgresql/data/pg_serial /var/lib/postgresql/data/pg_snapshots /var/lib/postgresql/data/pg_stat /var/lib/postgresql/data/pg_stat_tmp /var/lib/postgresql/data/pg_subtrans /var/lib/postgresql/data/pg_tblspc /var/lib/postgresql/data/pg_twophase /var/lib/postgresql/data/pg_wal /var/lib/postgresql/data/pg_xact /var/lib/postgresql/data/postgresql.auto.conf /var/lib/postgresql/data/postgresql.conf /var/lib/postgresql/data/postmaster.opts
+ export PGPORT=11000
+ PGPORT=11000
+ psql -d postgresql://oc_nextcloud:<pg_pass>@127.0.0.1:11000/nextcloud_database -c 'select now()'
+ exec docker-entrypoint.sh postgres
psql: error: connection to server at "127.0.0.1", port 11000 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
+ echo 'Waiting for the database to start.'
Waiting for the database to start.
+ sleep 5
chmod: /var/run/postgresql: Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Europe/Sofia
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... sh: locale: not found
2024-06-19 09:57:04.578 EEST [31] WARNING:  no usable system locales were found
+ psql -d postgresql://oc_nextcloud:<pg_pass>@127.0.0.1:11000/nextcloud_database -c 'select now()'
psql: error: connection to server at "127.0.0.1", port 11000 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
+ echo 'Waiting for the database to start.'
Waiting for the database to start.
+ sleep 5
+ psql -d postgresql://oc_nextcloud:<pg_pass>@127.0.0.1:11000/nextcloud_database -c 'select now()'
psql: error: connection to server at "127.0.0.1", port 11000 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
+ echo 'Waiting for the database to start.'
Waiting for the database to start.
+ sleep 5
+ psql -d postgresql://oc_nextcloud:<pg_pass>@127.0.0.1:11000/nextcloud_database -c 'select now()'
psql: error: connection to server at "127.0.0.1", port 11000 failed: Connection refused
        Is the server running on that host and accepting TCP/IP connections?
+ echo 'Waiting for the database to start.'
Waiting for the database to start.
+ sleep 5
...

I’ve forgotten to mention that I was using the option to have the newer version (Nextcloud Hub 8) on this instance. I have another instance that was running Hub 7 and it updated without any problems.

szaimen · June 19, 2024, 1:44pm

Hm, this is weird. Can you run sudo docker stop nextcloud-aio-database; sudo docker rm nextcloud-aio-database; sudo docker image prune -af and afterwards try to restore the backup again and start the containers back up?

If it still doesnt work, please post the database container logs again.

svethitov · June 19, 2024, 4:54pm

Unfortunately, the same outcome.

root@docker-host-01:~# docker stop nextcloud-aio-database
nextcloud-aio-database
root@docker-host-01:~# docker rm nextcloud-aio-database
nextcloud-aio-database
root@docker-host-01:~# docker image prune -af
Deleted Images:
untagged: nextcloud/aio-postgresql:latest
untagged: nextcloud/aio-postgresql@sha256:f14eb96e3e60ae424f335c0681997d2eb963983f6100d43dee78d5902996ce6d
deleted: sha256:150f097d530f21dd43d536a95b3e03e3af89bf7a00dd8b4d4e7b9411670a85f5
deleted: sha256:afc0f8a27468ff069f6a16638a40946b7f2f71f4763b28d9e1b1e737e9386890
deleted: sha256:55feb393878a9b596857f0b9e744beae891e825d58bcfa9f8f03a8d0fcb8f179
deleted: sha256:5e1bdd0d00f1e6438c410cf20f3b4d9416401af95eef53e094ef4bfffcd0230c
deleted: sha256:44e95697e0d97c5456573270f079ca15b62404b1922b006eab19f13afe4a4e89
deleted: sha256:935f808473aa3e15cf00fb8464832b3c209a344f09b138b395ac4520bf8ab3ac
deleted: sha256:0c5fecccd6bb5b8311d988aa7d9f390a64573ba01bcc902d3d9fca36d254a25d
deleted: sha256:5ad58308412f50e53c2aab23800356bd6de33ba4c6d935971b4f7e9fde8f8d7c
deleted: sha256:b943670a71d05ad2e4bfef6ed9920ef158adfd6bc14958a3ba278b1ed8e780b8
deleted: sha256:473dd00489c3e46d6fcd36be6a0e0fcf98dfbf8e8a216bba051ab12f2d01894d
deleted: sha256:408e20803e1e253d71d03284e9c11b2f7c097ef9de34b920ab9eff4a07e46b9d
deleted: sha256:4b45dcaff7a63ff0ea514b5e75850e9008595c775fa3af6f65888fd1a5fdc087
deleted: sha256:46a3cda5ae5f5e7cebf68b030d2464b740566ec9d4dc81a7c462b649784515f4

Total reclaimed space: 241.8MB

And the log of the container:

Restoring from database dump.
+ rm -rf /var/lib/postgresql/data/PG_VERSION /var/lib/postgresql/data/base /var/lib/postgresql/data/global /var/lib/postgresql/data/pg_commit_ts /var/lib/postgresql/data/pg_dynshmem /var/lib/postgresql/data/pg_hba.conf /var/lib/postgresql/data/pg_ident.conf /var/lib/postgresql/data/pg_logical /var/lib/postgresql/data/pg_multixact /var/lib/postgresql/data/pg_notify /var/lib/postgresql/data/pg_replslot /var/lib/postgresql/data/pg_serial /var/lib/postgresql/data/pg_snapshots /var/lib/postgresql/data/pg_stat /var/lib/postgresql/data/pg_stat_tmp /var/lib/postgresql/data/pg_subtrans /var/lib/postgresql/data/pg_tblspc /var/lib/postgresql/data/pg_twophase /var/lib/postgresql/data/pg_wal /var/lib/postgresql/data/pg_xact /var/lib/postgresql/data/postgresql.auto.conf /var/lib/postgresql/data/postgresql.conf /var/lib/postgresql/data/postmaster.opts
+ export PGPORT=11000
+ PGPORT=11000
+ psql -d postgresql://oc_nextcloud:72393e41d8e0b6f8d8269b432f55bcb7a2623e0fbb5042df@127.0.0.1:11000/nextcloud_database -c 'select now()'
+ exec docker-entrypoint.sh postgres
psql: error: connection to server at "127.0.0.1", port 11000 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?
+ echo 'Waiting for the database to start.'
Waiting for the database to start.

5
chmod: /var/run/postgresql: Operation not permitted
The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Europe/Sofia
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... sh: locale: not found
2024-06-19 18:31:29.541 EEST [31] WARNING:  no usable system locales were found
ok
syncing data to disk ... + psql -d postgresql://oc_nextcloud:72393e41d8e0b6f8d8269b432f55bcb7a2623e0fbb5042df@127.0.0.1:11000/nextcloud_database -c 'select now()'
psql: error: connection to server at "127.0.0.1", port 11000 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?
+ echo 'Waiting for the database to start.'
Waiting for the database to start.

5
+ psql -d postgresql://oc_nextcloud:72393e41d8e0b6f8d8269b432f55bcb7a2623e0fbb5042df@127.0.0.1:11000/nextcloud_database -c 'select now()'
psql: error: connection to server at "127.0.0.1", port 11000 failed: Connection refused
	Is the server running on that host and accepting TCP/IP connections?
+ echo 'Waiting for the database to start.'
Waiting for the database to start.

svethitov · June 19, 2024, 6:07pm

@szaimen actually it worked.
I left my workstation for a while and after I returned the DB init process was completed. After the previous restore I left it for several hours.
Thank you for you suggestion. Honestly, I do not understand how the removal of the container helped given that I deleted all of the containers and the volumes previously and then executed docker run ... nextcloud/all-in-one:latest but thank you!
You have my gratitude. I was struggling with this for the better part of the day.