In maintenance mode after upgrading Nextcloud on TrueNAS Scale

ℹ️ Support
, ,
1

Nextcloud version: 29.0.4
Chart version: 2.0.11
Operating system and version: TrueNAS Scale 24.04.2
Apache or nginx version: nginx 1.25.4
PHP version: whatever the TrueNAS Community NextCloud app installer set up

The issue you are facing:

I put off updating my TrueNAS server for several months because of OpenVPN being removed. I recently bit the bullet, with the intention of setting up Zerotier as a replacement for Open VPN. I updated TrueNAS Bluefin to Cobia to Dragonfish, and I correspondingly upgraded NextCloud from the originally installed version (27?) to 29.0.4.

The upgrades proceeded successfully and TrueNAS has NextCloud flagged as Running. It is using a nominal amount of CPU and maybe 66MiB of RAM, and is actively using a few kb/s of network traffic.

Since the upgrade, when I click Web Portal in Apps > Nexcloud > Application Info I get a persistent Maintenance mode window.

Is this the first time you’ve seen this error?: Y

Steps to replicate it:

  1. I don’t have another TrueNAS system lying around in order to figure out if this can be replicated and what steps might be required to do so

I suspect the requests below are not optimised for a TrueNAS installation
The output of your Nextcloud log in Admin > Logging:

I'd love to provide this but there is no logging apparent to me from the TrueNAS Apps > NextCloud app page, and when I click on Web Portal to access NextCloud to find Admin > Logging I get a Maintenance mode window

The output of your config.php file in /path/to/nextcloud :

When I open the shell in TrueNAS and cd to /mnt/tank/nextcloud I get a permission denied error, though this is the admin@truenas$ shell

The output of your Apache/nginx/system log in /var/log/____:

there is no subfolder called '____' in /var/log on my TrueNAS box

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

there is no /var/www/nextcloud.log on my TrueNAS box and I cannot access the Nextcloud portal to access the admin menu because of the maintenance mode

I can provide the following History from the TrueNAS Applications window:

2024-07-25 19:00:12
Job completed
2024-07-25 19:00:12
Saw completed job: nextcloud-nextcloud-cron-28698180, status: Complete
2024-07-25 19:00:02
Created container nextcloud
2024-07-25 19:00:02
Started container nextcloud
2024-07-25 19:00:00
Created job nextcloud-nextcloud-cron-28698180
2024-07-25 19:00:00
Created pod: nextcloud-nextcloud-cron-28698180-9db9q
2024-07-25 19:00:00
Successfully assigned ix-nextcloud/nextcloud-nextcloud-cron-28698180-9db9q to ix-truenas
2024-07-25 19:00:00
Add eth0 [172.16.5.52/16] from ix-net
2024-07-25 19:00:00
Container image "nextcloud:29.0.4" already present on machine
2024-07-25 18:45:12
Job completed
2024-07-25 18:45:12
Saw completed job: nextcloud-nextcloud-cron-28698165, status: Complete
2024-07-25 18:45:02
Started container nextcloud
2024-07-25 18:45:01
Created container nextcloud
2024-07-25 18:45:00
Created job nextcloud-nextcloud-cron-28698165
2024-07-25 18:45:00
Created pod: nextcloud-nextcloud-cron-28698165-nsqnp
2024-07-25 18:45:00
Successfully assigned ix-nextcloud/nextcloud-nextcloud-cron-28698165-nsqnp to ix-truenas
2024-07-25 18:45:00
Add eth0 [172.16.5.51/16] from ix-net
2024-07-25 18:45:00
Container image "nextcloud:29.0.4" already present on machine
2024-07-25 18:30:10
Saw completed job: nextcloud-nextcloud-cron-28698150, status: Complete
2024-07-25 18:30:10
Job completed
2024-07-25 18:30:01
Created container nextcloud
2024-07-25 18:30:01
Started container nextcloud
2024-07-25 18:30:00
Created job nextcloud-nextcloud-cron-28698150
2024-07-25 18:30:00
Created pod: nextcloud-nextcloud-cron-28698150-bcsmv
2024-07-25 18:30:00
Successfully assigned ix-nextcloud/nextcloud-nextcloud-cron-28698150-bcsmv to ix-truenas
2024-07-25 18:30:00
Add eth0 [172.16.5.50/16] from ix-net
2024-07-25 18:30:00
Container image "nextcloud:29.0.4" already present on machine
2024-07-25 18:15:09
Job completed
2024-07-25 18:15:09
Saw completed job: nextcloud-nextcloud-cron-28698135, status: Complete
2024-07-25 18:15:02
Started container nextcloud
2024-07-25 18:15:01
Created container nextcloud
2024-07-25 18:15:00
Created job nextcloud-nextcloud-cron-28698135
2024-07-25 18:15:00
Created pod: nextcloud-nextcloud-cron-28698135-qrsrk
2024-07-25 18:15:00
Successfully assigned ix-nextcloud/nextcloud-nextcloud-cron-28698135-qrsrk to ix-truenas
2024-07-25 18:15:00
Add eth0 [172.16.5.49/16] from ix-net
2024-07-25 18:15:00
Container image "nextcloud:29.0.4" already present on machine

Under Workloads I see the following:
Docker: Accelerated Container Application Development
with the logs:

2024-07-25 16:47:51.358888+12:00e[38;5;6mredis e[38;5;5m04:47:51.35 e[0m
2024-07-25 16:47:51.360318+12:00e[38;5;6mredis e[38;5;5m04:47:51.35 e[0me[1mWelcome to the Bitnami redis containere[0m
2024-07-25 16:47:51.361797+12:00e[38;5;6mredis e[38;5;5m04:47:51.36 e[0mSubscribe to project updates by watching e[1mhttps://github.com/bitnami/containerse[0m
2024-07-25 16:47:51.364573+12:00e[38;5;6mredis e[38;5;5m04:47:51.36 e[0mSubmit issues and feature requests at e[1mhttps://github.com/bitnami/containers/issuese[0m
2024-07-25 16:47:51.366279+12:00e[38;5;6mredis e[38;5;5m04:47:51.36 e[0m
2024-07-25 16:47:51.370834+12:00e[38;5;6mredis e[38;5;5m04:47:51.36 e[0me[38;5;2mINFO e[0m ==> ** Starting Redis setup **
2024-07-25 16:47:51.593833+12:00e[38;5;6mredis e[38;5;5m04:47:51.59 e[0me[38;5;2mINFO e[0m ==> Initializing Redis
2024-07-25 16:47:51.628621+12:00e[38;5;6mredis e[38;5;5m04:47:51.62 e[0me[38;5;2mINFO e[0m ==> Setting Redis config file
2024-07-25 16:47:52.363143+12:00e[38;5;6mredis e[38;5;5m04:47:52.35 e[0me[38;5;2mINFO e[0m ==> ** Redis setup finished! **
2024-07-25 16:47:52.363262+12:002024-07-25T16:47:52.363262292+12:00
2024-07-25 16:47:52.381826+12:00e[38;5;6mredis e[38;5;5m04:47:52.38 e[0me[38;5;2mINFO e[0m ==> ** Starting Redis **
2024-07-25 16:47:52.566463+12:001:C 25 Jul 2024 04:47:52.566 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
2024-07-25 16:47:52.566536+12:001:C 25 Jul 2024 04:47:52.566 # Redis version=7.0.11, bits=64, commit=00000000, modified=0, pid=1, just started
2024-07-25 16:47:52.566566+12:001:C 25 Jul 2024 04:47:52.566 # Configuration loaded
2024-07-25 16:47:52.568217+12:001:M 25 Jul 2024 04:47:52.568 * monotonic clock: POSIX clock_gettime
2024-07-25 16:47:52.591891+12:001:M 25 Jul 2024 04:47:52.591 * Running mode=standalone, port=6379.
2024-07-25 16:47:52.591932+12:001:M 25 Jul 2024 04:47:52.591 # Server initialized
2024-07-25 16:47:52.641226+12:001:M 25 Jul 2024 04:47:52.641 * Creating AOF base file appendonly.aof.1.base.rdb on server start
2024-07-25 16:47:52.727188+12:001:M 25 Jul 2024 04:47:52.726 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
2024-07-25 16:47:52.727811+12:001:M 25 Jul 2024 04:47:52.726 * Ready to accept connections

docker.io/library/nextcloud:29.0.4
with the logs:

2024-07-25 16:47:51.082935+12:002024-07-25T16:47:51.082935847+12:00
2024-07-25 16:47:51.083013+12:00PostgreSQL Database directory appears to contain a database; Skipping initialization
2024-07-25 16:47:51.083026+12:002024-07-25T16:47:51.083026409+12:00
2024-07-25 16:47:52.506503+12:002024-07-25 04:47:52.506 UTC [1] LOG:  starting PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2024-07-25 16:47:52.507105+12:002024-07-25 04:47:52.506 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-07-25 16:47:52.507443+12:002024-07-25 04:47:52.506 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-07-25 16:47:52.583403+12:002024-07-25 04:47:52.583 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-07-25 16:47:52.723775+12:002024-07-25 04:47:52.723 UTC [13] LOG:  database system was shut down at 2024-07-25 04:47:40 UTC
2024-07-25 16:47:52.807488+12:002024-07-25 04:47:52.807 UTC [1] LOG:  database system is ready to accept connections

docker.io/library/nginx:1.25.4
with the logs: (just a taste)

2024-07-25 19:23:31.375682+12:00172.16.0.1 - - [25/Jul/2024:07:23:31 +0000] "GET /status.php HTTP/2.0" 200 168 "-" "kube-probe/1.26"
2024-07-25 19:23:38.043034+12:00192.168.178.49 - jeremy [25/Jul/2024:07:23:38 +0000] "GET /index.php/204 HTTP/1.1" 503 12310 "-" "Mozilla/5.0 (Windows) mirall/3.13.0stable-Win64 (build 20240423) (Nextcloud, windows-10.0.22631 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
2024-07-25 19:23:38.067335+12:00192.168.178.49 - jeremy [25/Jul/2024:07:23:38 +0000] "GET /status.php HTTP/1.1" 200 168 "-" "Mozilla/5.0 (Windows) mirall/3.13.0stable-Win64 (build 20240423) (Nextcloud, windows-10.0.22631 ClientArchitecture: x86_64 OsArchitecture: x86_64)"
2024-07-25 19:23:41.375165+12:00172.16.0.1 - - [25/Jul/2024:07:23:41 +0000] "GET /status.php HTTP/2.0" 200 168 "-" "kube-probe/1.26"
2024-07-25 19:23:41.375468+12:00172.16.0.1 - - [25/Jul/2024:07:23:41 +0000] "GET /status.php HTTP/2.0" 200 168 "-" "kube-probe/1.26"
2024-07-25 19:23:51.375269+12:00172.16.0.1 - - [25/Jul/2024:07:23:51 +0000] "GET /status.php HTTP/2.0" 200 168 "-" "kube-probe/1.26"
2024-07-25 19:23:51.375498+12:00172.16.0.1 - - [25/Jul/2024:07:23:51 +0000] "GET /status.php HTTP/2.0" 200 168 "-" "kube-probe/1.26"
2024-07-25 19:24:01.375718+12:00172.16.0.1 - - [25/Jul/2024:07:24:01 +0000] "GET /status.php HTTP/2.0" 200 168 "-" "kube-probe/1.26"
2024-07-25 19:24:01.375781+12:00172.16.0.1 - - [25/Jul/2024:07:24:01 +0000] "GET /status.php HTTP/2.0" 200 168 "-" "kube-probe/1.26"

docker.io/library/postgres:13.1
with the logs:

2024-07-25 16:47:51.082935+12:002024-07-25T16:47:51.082935847+12:00
2024-07-25 16:47:51.083013+12:00PostgreSQL Database directory appears to contain a database; Skipping initialization
2024-07-25 16:47:51.083026+12:002024-07-25T16:47:51.083026409+12:00
2024-07-25 16:47:52.506503+12:002024-07-25 04:47:52.506 UTC [1] LOG:  starting PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2024-07-25 16:47:52.507105+12:002024-07-25 04:47:52.506 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2024-07-25 16:47:52.507443+12:002024-07-25 04:47:52.506 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2024-07-25 16:47:52.583403+12:002024-07-25 04:47:52.583 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2024-07-25 16:47:52.723775+12:002024-07-25 04:47:52.723 UTC [13] LOG:  database system was shut down at 2024-07-25 04:47:40 UTC
2024-07-25 16:47:52.807488+12:002024-07-25 04:47:52.807 UTC [1] LOG:  database system is ready to accept connections
2

unfortunately you didn’t provide the most interesting logs :man_facepalming:

contains postgres logs, please replace.

if you in fact upgraded NC27 to NC29 directly which is not supported you likely hit the “upgrade over multiple versions” but please provide container and nextcloud.log (from your data directory) to verify.

3

I didn’t, I upgraded via NC28. You’re reading too much about me upgrading from 27 to 29, which is what I did, via 28.

You might face-palm at me for providing postgres logs instead of NC logs, but that is literally the content of the log:

Please provide more detail about your request re: ‘providing container and nextcloud.log from data directory’ - what container would you like (surely you do not mean I should provide an entire docker container?) and where can I find the data directory you speak of, given TrueNAS does not provide a file management tool to browse the pools, and I do not already know the location of the information you want

4

in this case you are running postgres container named Nextcloud which is not intended.

docker logs of you Nextcloud container and nextcloud.log stored in the data directory (by default)

5

Well, I didn’t do anything to make that happen, that’s what the Nextcloud installer did. Perchance, you can tell me how to make it to what it is supposed to be doing?

Where. Can. I. Find. These.

I installed the Nextcloud TrueNAS community Nextcloud app installer and let it do it’s thing. I am not intrinsically aware of what that installer did and where it stuck anything. I’m only peripherally aware that these TrueNAS community apps are some sort of abstraction or sandbox using docker containers, pretty much only because after installing the app package the Workloads pane lists a section called Containers and has those docker io links I provided earlier. This is apparently an awesome capability but this abstraction has thus far only served to make it less clear where anything I might be interested in navigating to is actually to be found.

again, I cannot cd into /mnt/tank/nextcloud because it says permisson denied, though I am in as admin.

When I go to Apps > nextcloud > Workloads > and click the shell button for the docker io/library/nextcloud container, and ls, I do not see a data directory.

$ ls
bin  boot  dev  docker-entrypoint-initdb.d  docker-entrypoint.sh  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

I am assuming this is the content of the sandbox in the docker container created for nextcloud…?

I’m not playing dumb, it just isn’t clear to me what you are actually asking for based on what I can actually see in front of me.

6

I’m sorry I have no clue about TrueNAS specifics. It looks like they you “normal” Docker community containers but for specifics e.g. where/how they mount volumes I would suggest you reach out to TrueNAS community/support.

You must analyze what is wrong with your Nextcloud container - either there is some issues displaying right log or in fact it runs postgres… same for nextcloud.log - by default it is stored together with data and I can’t tell you how to access it. once you solve the problem of your software we can continue with Nextcloud issues.