Hello,
Here is a TL;DR of the weird stuff. Imagine I have a nextcloudpi installation working, and I don’t how (that’s my question) but:
- fail2Ban is not working
- nextcloud is not installed (as per the updater)
- ssh has been disabled on the rpi after a reboot
I have an outdated version of NC running on my rpi4, NC 24.0.8.2. I saw on scan.nextcloud.com that my instance was outdated, and I wanted to check whether fail2Ban caught something or not. But, fail2Ban actually failed:
sudo fail2ban-client status ssh
2023-06-14 00:38:00,405 fail2ban [18140]: ERROR NOK: ('ssh',)
Sorry but the jail 'ssh' does not exist
whereas this used to work. I tried to reactivate it from ncp-config but no luck (nor error messages).
I guessed that, updating might help. So I ran the update with ncp-config > updates to NC 25.0.7
update log with ERROR
Running nc-update-nextcloud
Current Nextcloud version 24.0.8.2
Available Nextcloud version 25.0.7
Download Nextcloud 25.0.7...
Back up current instance...
check free space...
Maintenance mode enabled
backup database...
backup files...
backup /var/www/nextcloud-bkp_20230612_1686605226.tar.gz generated
Maintenance mode disabled
Storing backup at '/var/www/nextcloud-bkp_20230612_1686605226-24.0.8.2.tar.gz'...
Install Nextcloud 25.0.7...
Fix permissions...
Upgrade...
Nextcloud is not installed - only a limited number of commands are available
Command "upgrade" is not defined.
Abort
Clean up...
Rolling back to backup /var/www/nextcloud-bkp_20230612_1686605226-24.0.8.2.tar.gz...
check free space...
extracting backup file /var/www/nextcloud-bkp_20230612_1686605226-24.0.8.2.tar.gz...
restore files...
Restoring old '/var/www/nextcloud/data' to '/var/www/nextcloud/data'...
restore database...
No datadir found in backup
Maintenance mode disabled
Starting scan for user 1 out of 7 (user1)
Starting scan for user 2 out of 7 (user2)
Starting scan for user 3 out of 7 (user3)
Starting scan for user 4 out of 7 (user4)
Starting scan for user 5 out of 7 (user5)
Starting scan for user 6 out of 7 (user6)
Starting scan for user 7 out of 7 (user7)
+---------+---------+--------------+
| Folders | Files | Elapsed time |
+---------+---------+--------------+
| 137256 | 2003538 | 14:53:11 |
+---------+---------+--------------+
Installing template 'php/opcache.ini.sh'...
System config value tempdirectory set to string /media/USBdrive/ncdata/tmp
sed: can't read /etc/php/8.1/cli/php.ini: No such file or directory
Cleanup...
Rollback failed! Data left at /var/www/recovery/ncp-data.QH4qUP
Done. Press any key...
as you can see it failed. Do you have any idea why ? I suspect that there might be a link between the fail2ban error and the Nextcloud does not exist error. Also the recovery data is empty.
After this failed update and failed rollback, the load on the rpi was high (around 5 or 6), caused mainly by mariadb being quite active.
At this point, nextcloud was running and working, although slowly, due to the high load on the rpi.
So I rebooted, but then I could ssh back in. I had to touch ssh on the boot partition of the SD card. MariaDB was still running a lot, using all cpus. Then I did sudo systemctl status and saw that nextcloudpi was in a degraded state with 2 failed services (I don’t know how to see which).
Then I started to try stuff, I remember from a bug I had few years back where I had to change the innoDB_buffer_size to 2GB, I did that again, and restarted the mariadb service, it took some time because the load was still high at this point, but did restart properly. Then the load became normal. I also cleared the PHP OPCache using ncp-config > tools.
After all that, nextcloudpi is still in a degraded state but with 1 failed service… later coming back at 2 failed services.
As you can see, there is also a pb of date.
However nextcloud seems to work, since the load is normal, the access to the website is fast, and new doc and event are properly sync.
Yet, fail2ban still fails although the jail.conf has the correct jails in it. So I suspect that there is some weird stuff going, but it’s way above my skills to track it down.
here is the nextcloudpi report in case :
NextcloudPi diagnostics
NextcloudPi version v1.52.0
NextcloudPi image NextCloudPi_03-28-20
OS Raspbian GNU/Linux 11. 5.15.76-v7l+ (armv7l)
automount yes
USB devices sda sdb
datadir /media/USBdrive/ncdata
data in SD no
data filesystem ext2/ext3
data disk usage 579G/916G
rootfs usage 20G/29G
swapfile /var/swap
dbdir /var/lib/mysql
Nextcloud check ok
Nextcloud version 24.0.8.2
HTTPD service up
PHP service up
MariaDB service up
Redis service up
HPB service up
Postfix service up
Internet check ok
Public IPv4 ***REMOVED SENSITIVE VALUE***
Public IPv6 ***REMOVED SENSITIVE VALUE***
Port 80 open
Port 443 open
IP ***REMOVED SENSITIVE VALUE***
Gateway ***REMOVED SENSITIVE VALUE***
Interface eth0
Certificates ***REMOVED SENSITIVE VALUE***
NAT loopback no
Uptime 1:17
Nextcloud configuration
{
"system": {
"passwordsalt": "***REMOVED SENSITIVE VALUE***",
"secret": "***REMOVED SENSITIVE VALUE***",
"trusted_domains": {
"0": "localhost",
"5": "nextcloudpi.local",
"7": "nextcloudpi",
"8": "nextcloudpi.lan",
"1": "192.168.0.4",
"4": "nc.jcjm.fr",
"20": "_",
"21": "_",
"22": "_",
"11": "2a01:e34:ec17:a810:a4da:f6c6:25c6:2136",
"12": "nc.jcjm.fr",
"3": "nc.jcjm.fr",
"": "nc.jcjm.fr",
"14": "nextcloudpi"
},
"datadirectory": "***REMOVED SENSITIVE VALUE***",
"dbtype": "mysql",
"version": "24.0.8.2",
"overwrite.cli.url": "https:\/\/nc.jcjm.fr\/",
"dbname": "***REMOVED SENSITIVE VALUE***",
"dbhost": "***REMOVED SENSITIVE VALUE***",
"dbport": "",
"dbtableprefix": "oc_",
"mysql.utf8mb4": true,
"dbuser": "***REMOVED SENSITIVE VALUE***",
"dbpassword": "***REMOVED SENSITIVE VALUE***",
"installed": true,
"instanceid": "***REMOVED SENSITIVE VALUE***",
"memcache.local": "\\OC\\Memcache\\Redis",
"memcache.locking": "\\OC\\Memcache\\Redis",
"redis": {
"host": "***REMOVED SENSITIVE VALUE***",
"port": 0,
"timeout": 0,
"password": "***REMOVED SENSITIVE VALUE***"
},
"mail_smtpmode": "smtp",
"mail_from_address": "***REMOVED SENSITIVE VALUE***",
"mail_domain": "***REMOVED SENSITIVE VALUE***",
"overwriteprotocol": "https",
"maintenance": false,
"logfile": "\/media\/USBdrive\/ncdata\/nextcloud.log",
"loglevel": "2",
"log_type": "file",
"theme": "",
"mail_sendmailmode": "smtp",
"mail_smtphost": "***REMOVED SENSITIVE VALUE***",
"mail_smtpport": "465",
"mail_smtpsecure": "ssl",
"mail_smtpauth": 1,
"mail_smtpauthtype": "LOGIN",
"mail_smtpname": "***REMOVED SENSITIVE VALUE***",
"mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
"twofactor_enforced": "false",
"twofactor_enforced_groups": [],
"twofactor_enforced_excluded_groups": [],
"jpeg_quality": "60",
"trashbin_retention_obligation": "15, 30",
"data-fingerprint": "a1c395d2ed29c4c09d49dc3af20e514c",
"trusted_proxies": "***REMOVED SENSITIVE VALUE***",
"tempdirectory": "\/media\/USBdrive\/ncdata\/tmp"
}
}
HTTPd logs
[Wed Jun 14 00:01:18.127923 2023] [ssl:warn] [pid 732:tid 3066511808] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Wed Jun 14 00:01:18.194804 2023] [ssl:error] [pid 732:tid 3066511808] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=archlinux / issuer: CN=archlinux / serial: 2A964521A5A0AD28440B39B1D781ABFDD1281F7C / notbefore: Mar 28 20:04:49 2020 GMT / notafter: Mar 26 20:04:49 2030 GMT]
[Wed Jun 14 00:01:18.194837 2023] [ssl:error] [pid 732:tid 3066511808] AH02604: Unable to configure certificate localhost:4443:0 for stapling
[Wed Jun 14 00:01:18.197164 2023] [mpm_event:notice] [pid 732:tid 3066511808] AH00489: Apache/2.4.54 (Raspbian) OpenSSL/1.1.1n configured -- resuming normal operations
[Wed Jun 14 00:01:18.197191 2023] [core:notice] [pid 732:tid 3066511808] AH00094: Command line: '/usr/sbin/apache2'
Database logs
2022-12-10 13:51:32 0 [Note] /usr/sbin/mysqld (initiated by: unknown): Normal shutdown
2022-12-10 13:51:32 0 [Note] Event Scheduler: Purging the queue. 0 events
2022-12-10 13:51:32 0 [Note] InnoDB: FTS optimize thread exiting.
2022-12-10 13:51:32 0 [Note] InnoDB: Starting shutdown...
2022-12-10 13:51:32 0 [Note] InnoDB: Dumping buffer pool(s) to /var/lib/mysql/ib_buffer_pool
2022-12-10 13:51:32 0 [Note] InnoDB: Instance 0, restricted to 16382 pages due to innodb_buf_pool_dump_pct=25
2022-12-10 13:51:32 0 [Note] InnoDB: Buffer pool(s) dump completed at 221210 13:51:32
2022-12-10 13:51:35 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2022-12-10 13:51:35 0 [Note] InnoDB: Shutdown completed; log sequence number 155482689802; transaction id 129105092
2022-12-10 13:51:35 0 [Note] /usr/sbin/mysqld: Shutdown complete
debug2: channel 0: window 999062 sent adjust 49514
If there is anyone nice enough to help, I’d be happy !
Thanks
