So I have a strange issue:
I use Nextcloudpi on a Raspberry PI 3B+ with a DDNS (no-ip).
It has worked flawlessly for months now, until last week.
I use Phonetrack to track my location. Sometimes I get the notification that phontrack canât connect to my server. When I then try to access the web interface, I canât connect either, but here is the thing:
I can connect using SSH (be it with Putty or Terminus).
When I restart the RPI via terminal it starts working again.
Does anyone know what the issue could be?
I actually canât use âsudo ncp-reportâ either, before I restart and the logs after the reboot donât show anything of note.
I also canât do any other Nextcloud specific commands either. It tries to execute them but gets stuck.
All I can do is shutdown etc.
Can you connect to ncp web interface? There is a system check on initial page, it should contain infos that may help you finding the culprit, probably redis or mariadb issue?
So then you can check if your webserver and database server are running. Is this the case? Do the webserver logs show anything? Or are they blocking up all resources?
To be honest, I donât know how to check if they are running.
What would I have to enter in the terminal to find out?
Also, I would have to wait for the server to crash again, which usually takes between one and two days. So I canât get back to you with the info right away.
Here is the log:
HTTPd logs
[Sat Aug 08 00:00:03.010540 2020] [ssl:warn] [pid 672] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sat Aug 08 00:00:03.011192 2020] [http2:warn] [pid 672] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are pr ocessed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will co ntinue to work, but the HTTP/2 protocol will be inactive.
[Sat Aug 08 00:00:03.012567 2020] [mpm_prefork:notice] [pid 672] AH00163: Apache /2.4.38 (Raspbian) OpenSSL/1.1.1d configured -- resuming normal operations
[Sat Aug 08 00:00:03.012589 2020] [core:notice] [pid 672] AH00094: Command line: '/usr/sbin/apache2'
[Sat Aug 08 10:17:05.663983 2020] [mpm_prefork:notice] [pid 672] AH00169: caught SIGTERM, shutting down
[Sat Aug 08 10:19:03.175657 2020] [ssl:warn] [pid 438] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sat Aug 08 10:19:03.275614 2020] [ssl:warn] [pid 640] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sat Aug 08 10:19:03.276386 2020] [http2:warn] [pid 640] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are pr ocessed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will co ntinue to work, but the HTTP/2 protocol will be inactive.
[Sat Aug 08 10:19:03.285808 2020] [mpm_prefork:notice] [pid 640] AH00163: Apache /2.4.38 (Raspbian) OpenSSL/1.1.1d configured -- resuming normal operations
[Sat Aug 08 10:19:03.285949 2020] [core:notice] [pid 640] AH00094: Command line: '/usr/sbin/apache2'
Database logs
2020-08-08 10:19:22 0 [Note] InnoDB: Initializing buffer pool, total size = 384M , instances = 1, chunk size = 128M
2020-08-08 10:19:23 0 [Note] InnoDB: Completed initialization of buffer pool
2020-08-08 10:19:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-08-08 10:19:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active .
2020-08-08 10:19:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ib tmp1"
2020-08-08 10:19:23 0 [Note] InnoDB: Creating shared tablespace for temporary ta bles
2020-08-08 10:19:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Phys ically writing the file full; Please wait ...
2020-08-08 10:19:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-08-08 10:19:23 0 [Note] InnoDB: 10.3.22 started; log sequence number 355160 3777; transaction id 8711043
2020-08-08 10:19:23 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ ib_buffer_pool
2020-08-08 10:19:23 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-08-08 10:19:23 0 [Note] Recovering after a crash using tc.log
2020-08-08 10:19:23 0 [Note] Starting crash recovery...
2020-08-08 10:19:23 0 [Note] Crash recovery finished.
2020-08-08 10:19:23 0 [Note] Server socket created on IP: '127.0.0.1'.
2020-08-08 10:19:23 0 [Note] Reading of all Master_info entries succeeded
2020-08-08 10:19:23 0 [Note] Added new Master_info '' to hash table
2020-08-08 10:19:23 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.3.22-MariaDB-0+deb10u1' socket: '/run/mysqld/mysqld.sock' port: 3 306 Raspbian 10
2020-08-08 10:19:28 0 [Note] InnoDB: Buffer pool(s) load completed at 200808 10: 19:28
So, the server has crashed once again. I canât access the web interface and I canât upload or download anything.
I can only access it via SSH.
I wonât restart it, so I can do some troubleshooting. How should I proceed?
Edit: I canât actually reach the server via the https://192.168.1.15:4443 to some point. It lets me enter username and password but then never forwards me to the panel page. Itâs just a blank, white page.
However, hhtpd log does not seem to contain any problems, other than a server reboot at 10:17:05 and normal startup 2 mins laterâŚ
I am not an apache expert, but i think there may be an error log as well, but probably it needs to be enabled somehow.
The database log is also starting at the time of the reboot and thus i guess can not contain the crash problem.
Did you get these logs while the page was not reachable? If no, try to do soâŚ
Did you already try to get the mariadb log before restarting the server/mysql-service directly after the crash? The logs you provided earlier were not very helpfull as i sayed already.
Why cant you get rhe logs, while you can execute other shell commands?
FadeFx, turns out the systemctl enable mariadb.service only worked once. Itâs still crashing and I canât get it back up with taht command afterwards.
as to why I canât get the logs while Iâm able to do other shell commands: I have no clue. It doesnât give me an error message, it just starts the process and never shows any results.
Itâs really weird.
Iâll get back to you, once it crashes again. (which is gonna be within the next two days, I assume)
Turns out it wasnât MariaDB after all, itâs still running, even though the server ist down again.
And this is what I got from my ncp-report commandâŚa whole lot of nothing.
I still edited the mariadb.service file to include the error log to file.
First of all i dont see mysql in that top output om top of the screen, how you come to the conclusion mariadb is in fact running?
Second you had been too early witb the screenshot for ncp-report, as it shows rhe 3dots that say ot is still working to collect data, or does that stay tlike this forever?
And to answer the latter, it stays like this forever. At the very least for 15 min. Under normal circumstances it takes less than a minute.
BTW the warning message: âWarning: The unit file, source configuration file or drop-ins of mariadb.service changed on disk. Run âsystemctl daemon-reloadâ to reload units.â always comes back after a server crash, even if I do the systemctl daemon-reload.
But you might be right, itâs something with mysql, but why does it randomly crash?
Here is what happened when I followed the recommendations:
EDIT:
Here is another weird bit of information; to only sure-fire way to get the server back up is to use sudo shutdown now, unplug the powercord from RPI and then plug it back in. sudo reboot now causes a reboot without the server ever starting up again. This includes SSH, so I canât access it afterwards.
