Problem updating nextcloud on nextcloudpi (Weird stuff happening, eg : "nextcloud is not installed")


Here is a TL;DR of the weird stuff. Imagine I have a nextcloudpi installation working, and I don’t how (that’s my question) but:

  • fail2Ban is not working
  • nextcloud is not installed (as per the updater)
  • ssh has been disabled on the rpi after a reboot

I have an outdated version of NC running on my rpi4, NC I saw on that my instance was outdated, and I wanted to check whether fail2Ban caught something or not. But, fail2Ban actually failed:

sudo fail2ban-client status ssh
2023-06-14 00:38:00,405 fail2ban                [18140]: ERROR   NOK: ('ssh',)
Sorry but the jail 'ssh' does not exist

whereas this used to work. I tried to reactivate it from ncp-config but no luck (nor error messages).

I guessed that, updating might help. So I ran the update with ncp-config > updates to NC 25.0.7

update log with ERROR
Running nc-update-nextcloud
Current   Nextcloud version
Available Nextcloud version 25.0.7
Download Nextcloud 25.0.7...
Back up current instance...
check free space...
Maintenance mode enabled
backup database...
backup files...
backup /var/www/nextcloud-bkp_20230612_1686605226.tar.gz generated
Maintenance mode disabled
Storing backup at '/var/www/nextcloud-bkp_20230612_1686605226-'...
Install Nextcloud 25.0.7...
Fix permissions...
Nextcloud is not installed - only a limited number of commands are available

  Command "upgrade" is not defined.  

Clean up...
Rolling back to backup /var/www/nextcloud-bkp_20230612_1686605226-
check free space...
extracting backup file /var/www/nextcloud-bkp_20230612_1686605226-
restore files...
Restoring old '/var/www/nextcloud/data' to '/var/www/nextcloud/data'...
restore database...

No datadir found in backup
Maintenance mode disabled
Starting scan for user 1 out of 7 (user1)
Starting scan for user 2 out of 7 (user2)
Starting scan for user 3 out of 7 (user3)
Starting scan for user 4 out of 7 (user4)
Starting scan for user 5 out of 7 (user5)
Starting scan for user 6 out of 7 (user6)
Starting scan for user 7 out of 7 (user7)
| Folders | Files   | Elapsed time |
| 137256  | 2003538 | 14:53:11     |
Installing template 'php/'...
System config value tempdirectory set to string /media/USBdrive/ncdata/tmp
sed: can't read /etc/php/8.1/cli/php.ini: No such file or directory
Rollback failed! Data left at /var/www/recovery/ncp-data.QH4qUP
Done. Press any key...

as you can see it failed. Do you have any idea why ? I suspect that there might be a link between the fail2ban error and the Nextcloud does not exist error. Also the recovery data is empty.

After this failed update and failed rollback, the load on the rpi was high (around 5 or 6), caused mainly by mariadb being quite active.

At this point, nextcloud was running and working, although slowly, due to the high load on the rpi.

So I rebooted, but then I could ssh back in. I had to touch ssh on the boot partition of the SD card. MariaDB was still running a lot, using all cpus. Then I did sudo systemctl status and saw that nextcloudpi was in a degraded state with 2 failed services (I don’t know how to see which).

Then I started to try stuff, I remember from a bug I had few years back where I had to change the innoDB_buffer_size to 2GB, I did that again, and restarted the mariadb service, it took some time because the load was still high at this point, but did restart properly. Then the load became normal. I also cleared the PHP OPCache using ncp-config > tools.
After all that, nextcloudpi is still in a degraded state but with 1 failed service… later coming back at 2 failed services.

As you can see, there is also a pb of date.

However nextcloud seems to work, since the load is normal, the access to the website is fast, and new doc and event are properly sync.

Yet, fail2ban still fails although the jail.conf has the correct jails in it. So I suspect that there is some weird stuff going, but it’s way above my skills to track it down.

here is the nextcloudpi report in case :

NextcloudPi diagnostics

NextcloudPi version  v1.52.0
NextcloudPi image    NextCloudPi_03-28-20
OS                   Raspbian GNU/Linux 11. 5.15.76-v7l+ (armv7l)
automount            yes
USB devices          sda sdb 
datadir              /media/USBdrive/ncdata
data in SD           no
data filesystem      ext2/ext3
data disk usage      579G/916G
rootfs usage         20G/29G
swapfile             /var/swap
dbdir                /var/lib/mysql
Nextcloud check      ok
Nextcloud version
HTTPD service        up
PHP service          up
MariaDB service      up
Redis service        up
HPB service          up
Postfix service      up
Internet check       ok
Public IPv4          ***REMOVED SENSITIVE VALUE***
Public IPv6          ***REMOVED SENSITIVE VALUE***
Port 80              open
Port 443             open
IP                   ***REMOVED SENSITIVE VALUE***
Gateway              ***REMOVED SENSITIVE VALUE***
Interface            eth0
Certificates         ***REMOVED SENSITIVE VALUE***
NAT loopback         no
Uptime               1:17

Nextcloud configuration

    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "0": "localhost",
            "5": "nextcloudpi.local",
            "7": "nextcloudpi",
            "8": "nextcloudpi.lan",
            "1": "",
            "4": "",
            "20": "_",
            "21": "_",
            "22": "_",
            "11": "2a01:e34:ec17:a810:a4da:f6c6:25c6:2136",
            "12": "",
            "3": "",
            "": "",
            "14": "nextcloudpi"
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "",
        "overwrite.cli.url": "https:\/\/\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        "mail_smtpmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "overwriteprotocol": "https",
        "maintenance": false,
        "logfile": "\/media\/USBdrive\/ncdata\/nextcloud.log",
        "loglevel": "2",
        "log_type": "file",
        "theme": "",
        "mail_sendmailmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpsecure": "ssl",
        "mail_smtpauth": 1,
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "twofactor_enforced": "false",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "jpeg_quality": "60",
        "trashbin_retention_obligation": "15, 30",
        "data-fingerprint": "a1c395d2ed29c4c09d49dc3af20e514c",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "tempdirectory": "\/media\/USBdrive\/ncdata\/tmp"

HTTPd logs

[Wed Jun 14 00:01:18.127923 2023] [ssl:warn] [pid 732:tid 3066511808] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Wed Jun 14 00:01:18.194804 2023] [ssl:error] [pid 732:tid 3066511808] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=archlinux / issuer: CN=archlinux / serial: 2A964521A5A0AD28440B39B1D781ABFDD1281F7C / notbefore: Mar 28 20:04:49 2020 GMT / notafter: Mar 26 20:04:49 2030 GMT]
[Wed Jun 14 00:01:18.194837 2023] [ssl:error] [pid 732:tid 3066511808] AH02604: Unable to configure certificate localhost:4443:0 for stapling
[Wed Jun 14 00:01:18.197164 2023] [mpm_event:notice] [pid 732:tid 3066511808] AH00489: Apache/2.4.54 (Raspbian) OpenSSL/1.1.1n configured -- resuming normal operations
[Wed Jun 14 00:01:18.197191 2023] [core:notice] [pid 732:tid 3066511808] AH00094: Command line: '/usr/sbin/apache2'

Database logs

2022-12-10 13:51:32 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
2022-12-10 13:51:32 0 [Note] Event Scheduler: Purging the queue. 0 events
2022-12-10 13:51:32 0 [Note] InnoDB: FTS optimize thread exiting.
2022-12-10 13:51:32 0 [Note] InnoDB: Starting shutdown...
2022-12-10 13:51:32 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2022-12-10 13:51:32 0 [Note] InnoDB: Instance 0, restricted to 16382 pages due to innodb_buf_pool_dump_pct=25
2022-12-10 13:51:32 0 [Note] InnoDB: Buffer pool(s) dump completed at 221210 13:51:32
2022-12-10 13:51:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2022-12-10 13:51:35 0 [Note] InnoDB: Shutdown completed; log sequence number 155482689802; transaction id 129105092
2022-12-10 13:51:35 0 [Note] /usr/sbin/mysqld: Shutdown complete

debug2: channel 0: window 999062 sent adjust 49514

If there is anyone nice enough to help, I’d be happy !

Could it be because of PHP ? I’m still on php7.4.33

Any ideas otherwise ? @nachoparker @devnull @OliverV or else ?

Thanks in advance

I’m trying to investigate why fail2ban is not working, so maybe it will point to the problem ?

Apparently, the error comes from the backend used for the ssh jail, which is systemd.
When starting fail2ban, I got :

2023-06-24 10:25:29,155 fail2ban.server         [25682]: INFO    Starting Fail2ban v0.11.2
2023-06-24 10:25:29,156 fail2ban.server         [25682]: INFO    Daemon started
2023-06-24 10:25:29,158       [25682]: INFO    Observer start...
2023-06-24 10:25:29,371 fail2ban.database       [25682]: INFO    Connected to fail2ban persistent database '/var/lib/fail2ban/fail2ban.sqlite3'
2023-06-24 10:25:29,381 fail2ban.jail           [25682]: INFO    Creating new jail 'ssh'
2023-06-24 10:25:29,389 fail2ban.jail           [25682]: ERROR   Backend 'systemd' failed to initialize due to No module named 'systemd'
2023-06-24 10:25:29,389 fail2ban.jail           [25682]: ERROR   Failed to initialize any backend for Jail 'ssh'

But I don’t get why, I think I have systemd :

$ systemd --version
systemd 247 (247.3-7+rpi1+deb11u1)

Any help would be greatly appreciated ! :pray:

fail2ban is not your real problem. Deactivate it and solve Nextcloud version issue.

Yes I know, but how can I investigate?

Coming back to the ncp-update-nc script, I tried the command that failed. It failed after the Upgrade message from this block :

# upgrade
echo "Upgrade..."
ncc='sudo -u www-data php nextcloud/occ'
$ncc upgrade      # && false # test point
$ncc | grep -q db:add-missing-indices && $ncc db:add-missing-indices -n
$ncc | grep -q db:add-missing-columns && $ncc db:add-missing-columns -n
$ncc | grep -q db:add-missing-primary-keys && $ncc db:add-missing-primary-keys -n
$ncc | grep -q db:convert-filecache-bigint && $ncc db:convert-filecache-bigint -n

with this error

Nextcloud is not installed - only a limited number of commands are available

  Command "upgrade" is not defined.  

But when I try this command :

pi@nextcloudpi:~ $ sudo -u www-data php nextcloud/occ check
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
pi@nextcloudpi:~ $ sudo -u www-data php nextcloud/occ upgrade -h
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
  run upgrade routines after installation of a new release. The release has to be installed before.


  -h, --help            Display this help message
  -q, --quiet           Do not output any message
  -V, --version         Display this application version
      --ansi            Force ANSI output
      --no-ansi         Disable ANSI output
  -n, --no-interaction  Do not ask any interactive question
      --no-warnings     Skip global warnings, show command output only
  -v|vv|vvv, --verbose  Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug

I have a message that tells me that it seems to work but I should upgrade (I know, right!).

But why did it fail ?

Thanks for helping !