So I have a strange issue:
I use Nextcloudpi on a Raspberry PI 3B+ with a DDNS (no-ip).
It has worked flawlessly for months now, until last week.
I use Phonetrack to track my location. Sometimes I get the notification that phontrack can’t connect to my server. When I then try to access the web interface, I can’t connect either, but here is the thing:
I can connect using SSH (be it with Putty or Terminus).
When I restart the RPI via terminal it starts working again.
Does anyone know what the issue could be?
I actually can’t use “sudo ncp-report” either, before I restart and the logs after the reboot don’t show anything of note.
I also can’t do any other Nextcloud specific commands either. It tries to execute them but gets stuck.
All I can do is shutdown etc.
Thank you in advance!
Can you connect to ncp web interface? There is a system check on initial page, it should contain infos that may help you finding the culprit, probably redis or mariadb issue?
Nope, unfortunately I can’t once it has “crashed”.
After a restart I can, though.
What kind of terminal is this, just locally without network?
I can do it both locally or via the DDNS address. So I can access it from anywhere.
The DDNS is not the problem, unfortuantely.
So then you can check if your webserver and database server are running. Is this the case? Do the webserver logs show anything? Or are they blocking up all resources?
To be honest, I don’t know how to check if they are running.
What would I have to enter in the terminal to find out?
Also, I would have to wait for the server to crash again, which usually takes between one and two days. So I can’t get back to you with the info right away.
Here is the log:
[Sat Aug 08 00:00:03.010540 2020] [ssl:warn] [pid 672] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sat Aug 08 00:00:03.011192 2020] [http2:warn] [pid 672] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are pr ocessed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will co ntinue to work, but the HTTP/2 protocol will be inactive.
[Sat Aug 08 00:00:03.012567 2020] [mpm_prefork:notice] [pid 672] AH00163: Apache /2.4.38 (Raspbian) OpenSSL/1.1.1d configured -- resuming normal operations
[Sat Aug 08 00:00:03.012589 2020] [core:notice] [pid 672] AH00094: Command line: '/usr/sbin/apache2'
[Sat Aug 08 10:17:05.663983 2020] [mpm_prefork:notice] [pid 672] AH00169: caught SIGTERM, shutting down
[Sat Aug 08 10:19:03.175657 2020] [ssl:warn] [pid 438] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sat Aug 08 10:19:03.275614 2020] [ssl:warn] [pid 640] AH01909: localhost:4443:0 server certificate does NOT include an ID which matches the server name
[Sat Aug 08 10:19:03.276386 2020] [http2:warn] [pid 640] AH10034: The mpm module (prefork.c) is not supported by mod_http2. The mpm determines how things are pr ocessed in your server. HTTP/2 has more demands in this regard and the currently selected mpm will just not do. This is an advisory warning. Your server will co ntinue to work, but the HTTP/2 protocol will be inactive.
[Sat Aug 08 10:19:03.285808 2020] [mpm_prefork:notice] [pid 640] AH00163: Apache /2.4.38 (Raspbian) OpenSSL/1.1.1d configured -- resuming normal operations
[Sat Aug 08 10:19:03.285949 2020] [core:notice] [pid 640] AH00094: Command line: '/usr/sbin/apache2'
2020-08-08 10:19:22 0 [Note] InnoDB: Initializing buffer pool, total size = 384M , instances = 1, chunk size = 128M
2020-08-08 10:19:23 0 [Note] InnoDB: Completed initialization of buffer pool
2020-08-08 10:19:23 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2020-08-08 10:19:23 0 [Note] InnoDB: 128 out of 128 rollback segments are active .
2020-08-08 10:19:23 0 [Note] InnoDB: Removed temporary tablespace data file: "ib tmp1"
2020-08-08 10:19:23 0 [Note] InnoDB: Creating shared tablespace for temporary ta bles
2020-08-08 10:19:23 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Phys ically writing the file full; Please wait ...
2020-08-08 10:19:23 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2020-08-08 10:19:23 0 [Note] InnoDB: 10.3.22 started; log sequence number 355160 3777; transaction id 8711043
2020-08-08 10:19:23 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ ib_buffer_pool
2020-08-08 10:19:23 0 [Note] Plugin 'FEEDBACK' is disabled.
2020-08-08 10:19:23 0 [Note] Recovering after a crash using tc.log
2020-08-08 10:19:23 0 [Note] Starting crash recovery...
2020-08-08 10:19:23 0 [Note] Crash recovery finished.
2020-08-08 10:19:23 0 [Note] Server socket created on IP: '127.0.0.1'.
2020-08-08 10:19:23 0 [Note] Reading of all Master_info entries succeeded
2020-08-08 10:19:23 0 [Note] Added new Master_info '' to hash table
2020-08-08 10:19:23 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.3.22-MariaDB-0+deb10u1' socket: '/run/mysqld/mysqld.sock' port: 3 306 Raspbian 10
2020-08-08 10:19:28 0 [Note] InnoDB: Buffer pool(s) load completed at 200808 10: 19:28
So, the server has crashed once again. I can’t access the web interface and I can’t upload or download anything.
I can only access it via SSH.
I won’t restart it, so I can do some troubleshooting. How should I proceed?
Edit: I can’t actually reach the server via the https://192.168.1.15:4443 to some point. It lets me enter username and password but then never forwards me to the panel page. It’s just a blank, white page.
The database log looks like it starts when restarting after a db crash, is there any log before that? It may contain the actual crash reason.
Tha httpd log looks cut to the right and every line only contains part of the log text, hard to say anything from that…
You can run top in ssh to see if mysqld, php-fpm7.x and apache2 or ngnx processes are running
The problem is that I can’t query the log (ncp-report) once it’s crashed. I’ll run top next time it crashes and tell you what happened.
I copy-pasted the log, btw. If you scroll to the right, I think you can see all of it. No clue, why this happened, sorry about that.
Thank you for your help!
Ok i c, looks funny…
However, hhtpd log does not seem to contain any problems, other than a server reboot at 10:17:05 and normal startup 2 mins later…
I am not an apache expert, but i think there may be an error log as well, but probably it needs to be enabled somehow.
The database log is also starting at the time of the reboot and thus i guess can not contain the crash problem.
Did you get these logs while the page was not reachable? If no, try to do so…
At first i would check the raspi ip configuration.
Consider to set an fixed ip address instead a dhcp lease.
I already have a static up, how else could I use port-forwarding? O.o
Or am I missing something?
Anyway, I unfortunately can’t get the logs once nextcloud is down.
But I’ve now found out that this command fixed the issue:
systemctl enable mariadb.service
I just don’t understand why the service has started crashing regularly in the first place.
Did you already try to get the mariadb log before restarting the server/mysql-service directly after the crash? The logs you provided earlier were not very helpfull as i sayed already.
Why cant you get rhe logs, while you can execute other shell commands?
FadeFx, turns out the systemctl enable mariadb.service only worked once. It’s still crashing and I can’t get it back up with taht command afterwards.
as to why I can’t get the logs while I’m able to do other shell commands: I have no clue. It doesn’t give me an error message, it just starts the process and never shows any results.
It’s really weird.
I’ll get back to you, once it crashes again. (which is gonna be within the next two days, I assume)
You probably need to enable logging to a file:
Turns out it wasn’t MariaDB after all, it’s still running, even though the server ist down again.
And this is what I got from my ncp-report command…a whole lot of nothing.
I still edited the mariadb.service file to include the error log to file.
First of all i dont see mysql in that top output om top of the screen, how you come to the conclusion mariadb is in fact running?
Second you had been too early witb the screenshot for ncp-report, as it shows rhe 3dots that say ot is still working to collect data, or does that stay tlike this forever?
To answer your first question, see the attached screenshot.
And to answer the latter, it stays like this forever. At the very least for 15 min. Under normal circumstances it takes less than a minute.
BTW the warning message: “Warning: The unit file, source configuration file or drop-ins of mariadb.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.” always comes back after a server crash, even if I do the systemctl daemon-reload.
But you might be right, it’s something with mysql, but why does it randomly crash?
Here is what happened when I followed the recommendations:
Here is another weird bit of information; to only sure-fire way to get the server back up is to use sudo shutdown now, unplug the powercord from RPI and then plug it back in. sudo reboot now causes a reboot without the server ever starting up again. This includes SSH, so I can’t access it afterwards.