Empty response error starts at midnight or server startup

Support intro

Sorry to hear you’re facing problems :slightly_frowning_face:

help.nextcloud.com is for home/non-enterprise users. If you’re running a business, paid support can be accessed via portal.nextcloud.com where we can ensure your business keeps running smoothly.

In order to help you as quickly as possible, before clicking Create Topic please provide as much of the below as you can. Feel free to use a pastebin service for logs, otherwise either indent short log examples with four spaces:

example

Or for longer, use three backticks above and below the code snippet:

longer
example
here

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can :heart:

Nextcloud version (eg, 20.0.5): 27.0.0 and earlier versions over roughly the last year
Operating system and version (eg, Ubuntu 20.04): Ubuntu 22.04 and Mint 21.1
Apache or nginx version (eg, Apache 2.4.25): 2.4.52
PHP version (eg, 7.4): 8.1.2

The issue you are facing:

Nextcloud will get into a mode of crashing, which shows up as an empty response to the browser. The server must be restarted to clear the problem.

This problem happens either occasionally (intermittently) or at Apache startup.

This problem has been happening to me for quite some time over a number of Nextcloud releases. I have been running Nextlcoud for almost a year now.

Ubuntu will tell me that Apache crashed and want to send a report. If I look at the details, the stack trace will imply a crash in Zend. It used to be something like zend slow auth, but I saw something else happen recently but still seemed related to zend.

I think I have reduced the severity of the problem by telling Linux to not look for updates, and by disabling the update notification in Nextlcoud. I have also tried to change PHP settings and limit “apps” as much as I can (I still use a few). PHP should have plenty of memory, etc. The machines I run these on are new and have plenty of memory. I have several installations and they all have had the problem at one time or another. I’m assuming there is a configuration issue (I use a similar approach on all of them), or there’s something about this release of Ubuntu that is odd.

Recently, the problem only starts happening at midnight. I’m not sure what kind of maintenance might be happening then that I could try to reschedule? It would be more convenient for this to happen just before I get up in the morning. I have LibreNMS monitoring the system. By the time I see this in the morning, the system has written dozens of terabytes to my SSD, which is just basically burning up the longevity of the drive.

Should I try disabling the opcache? Is the problem maybe in my Redis configuration? I use Redis for caching and file locking.

I have run other Apache applications (e.g. Subversion, Trac, Wikimedia) for many years and never had to cope with this kind of problem. Also, when this occurs, other things running in Apache are fine. I’m competent at Linux but I’m not an expert. I an engineer, but I have been having problems finding a root cause for the problem, or a mitigation of the writing of data to the drive when the system is in this state. Also, unfortunately, I’m not a PHP expert.

Thanks for any advice you could give!!

Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it:

  1. Happens at midnight or server startup. Not sure why. Doesn’t always happen.

The output of your Nextcloud log in Admin > Logging:

There is no log message when this occurs.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

I'm using Redis for caching and file locking, and PHP opcache.

The output of your Apache/nginx/system log in /var/log/____:

It just says it crashed.

It happened again last night. Another 40TB or so written during the crash state.

I’m trying disabling opcache in the PHP configuration. That seems to fix the crash at startup. Or at least make it a lot better. I suppose I should have tried this before, but the message on the admin page is a constant reminder to enable it.

No more details? Nothing for php?
What is the last thing Nextcloud does before the crash, does it happen when the cronjob runs?
What is strange that you get the notification to send details to Ubuntu, but you don’t find the details in the logs…

Thanks for your reply! :smiley:

Apache log basically says signal 11, which is a segfault crash.

The fact that it’s happening at midnight tells me it’s triggered by something like the cron job, but I’m not sure how narrow that down properly. Maybe there are some other kinds of logs I need to look at or turn on. I think the Ubuntu crash reporter is finding something it can get the stack trace out of, which it will then show if you look at the information it’s going to send. That tool is apparently specific to Ubuntu versus Mint. It’s maybe also a clue as to why so much gets written to disk – all the crash dumps the system is trying to save. I’m not sure how to best take advantage of that, or possibly turn it off. I tried some time with internet searches, but I was getting a lot of un-useful information.

However, the crash also seems to happen when starting Apache, at least fairly regularly. So I think that can help, since it doesn’t happen very often any more while it’s running. And I’m not sure if it’s because of some activity like adding files or something like that, which maybe triggers something in the cron.

I decided to take a different tack and try to update PHP. As maybe I mentioned, I’m not really a PHP expert, but I can do enough to be dangerous lol.

I tried using the latest PHP updates via the instructions on the following web page:

One of the three Nextcloud instances I’m using is just for testing, and I updated it on that instance. So now rather than 8.1.2 it’s running 8.1.21.

I haven’t seen any failures after restarting Apache several times with the opcache enabled! That’s a really good sign.

… Which leads me to my next thought. Why don’t these LTS versions of Linux use the latest bug fix releases of theses packages? This is just a minor version bump. My test system is running Mint 21.1. One of my others is running that, and the other one Ubuntu 22.04. Those both are equipped with 8.1.2 right now it appears.

Anyway … I guess I’m learning.

If this holds up … which I have a feeling it will …

Is using ppa:ondrej/php a proper long-term solution? Are there better alternatives? This was simple to implement at least.

Thanks to any and all of you with better experience at this; your expertise is valued.

I learned you have to be careful about the things that get pulled in when you add the PPA because they are in a state ready to be autoremoved. Oops. It’s just a matter of actually installing them, not just receiving them partly installed, because then “apt” thinks they’re specifically wanted by you.

I thus ended up putting one of my 27.0.0 servers on PHP 8.2.8. However, there is a strange issue that manifests after some duration of being on a page. The page can no longer access the server. Also the server has a lot of load; apparently there’s some kind of thrashing going on, and it can get quite bad. I think I’ll have to finagle getting that one back to PHP 8.1.21. Or see if 27.0.1 can help. It’s just a day away, I think.

  • What apps exactly are activated?

  • Do you have the configuration parameter
    ‘maintenance_window_start’ => 1,
    defined in your setup?

Thanks much for that tip. I feel a little embarrassed that I wasn’t patient enough to look at the example config for that. I definitely would have tried that.

The apps I was de-activating were things that seemed to run in the background, like check for updates, news, federation, weather, external storage, etc. Plus, trying to stick to a core group of functional apps like Files, Talk, and Deck. On some systems I also needed Calendar and Notes and Passwords.

However, the update to PHP 8.1.21 and 8.2.8 seems to have fixed that crashing problem with its aggravating consequences. Time will tell, but I think that was the issue.

Last night I also found out that the problems I was having running PHP 8.2.8 weren’t due to the server per se. Brave browser (my primary) was apparently getting confused about connections to the server. Apps like Notes, Calendar, Deck, and Serverinfo would, after maybe five or ten minutes, become confused and not be able to contact the server. However, the server was obviously getting overloaded with some kind of requests, as based on both the CPU load and nethogs (network activity). I found out that clearing the cookies and browser cache fixed the problem. It seems kinda weird, but I’m glad it worked.

I’m really hoping to get to the point of not needing to bother with the administration duties and just enjoy using Nextcloud. Maybe now I will be able to do that. … Although with just active development, there will always be issues to be solved. I suppose my next biggest concern is being able to limit myself to well-kept apps. Since the apps need to be actively maintained for each version of Nextcloud, that is a burden for some authors and the plugins are either slow to support new Nextcloud versions or are essentially abandoned.

Still holding up well. No failures.