Internal server error. Untraceable! How is that even possible?

ℹ️ Support 📦 Appliances (Docker, Snappy, VM, NCP, AIO)
1

My Nextcloud died! All it does is report:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

Really? More details can be found in the server log? Not by me. I have tried everything and looked everywhere. Here is my sledge hammer approach:

First I make sure to exclude all proxies by accessing it locally by IP. I can and do access my server through a global domain name but that’s farmed out through a local reverse proxy and so I can and do access it by an internal .lan name or it’s LAN IP as well without problem, at will, regularly. All no dramas. Been working for years.

What’s caused this? At a mild loss, but suspect I did an update, with ncp-update. Not 100% sure. But that’s a candidate. Cause becomes by the by though as no matter what broke it, I should be able to diagnose it. So here goes:

I’m on a bash terminal on the nextcloud server and I do this:

  1. Store a time reference: touch /tmp/foobar
  2. force a page reload in my case I have https://192.168.0.14 open and CTRL+F5 in Firefox and the page reloads.
  3. Search for any filesystem changes since the reference time: sudo find / -regextype posix-extended -regex "/(sys|srv|proc|dev)" -prune -o -newer /tmp/foobar

Alas all this yields is:

/run/sudo/ts/cirrus
/dev
/sys
/var/log/apache2/other_vhosts_access.log
/var/log/apache2/nc-access.log
/var/log/auth.log
/proc
/srv

in short the ONLY logs I see are access logs and the sudo auth logs.

How can this be? How can apache be issuing an internal server error and leave no error log?

What gives? What can I learn from this?

2

Update: If I reboot it goes away for a bit, then reappears later. Aaargh. So I might find some joy if I can establish a mean time till failure (like I have all day to experiment with this!) then I can examine all the sys logs since boot to see if any clues emerge (any errors). It remains a supreme mystery how a page load can generate a displayed error but leave no logging. Is this some banal Apache feature?

3

Well, well, well, 53 people viewed the problem, not one hint at a solution. So I:

  1. Conclude that the issue is a tad too complicated for the NextCloud community so far and no guiru has wandered by with a helpful streak :wink:
  2. Posted it on Server Fault: https://serverfault.com/questions/993422/apache-php-internal-server-error-untraceable-how-is-that-even-possible and it has only 22 views so far but at least one comment that tried to help.
  3. The same box has another silly issue: https://www.raspberrypi.org/forums/viewtopic.php?f=36&t=253702&p=1547953#p1547953, which looks to have been solved with sudo systemctl disable wicd. It seem wicd is a system service that runs on the pi 3 that has the daft side effect of killing wired connections in bizarre ill understood (by me) ways … and losing it (given I haze zero interest in the wlan on that box) fixes that problem AND seemingly this one as well! As in the box has now been running running days without this Internal server error. it’s not clear how they relate, but it seems possible that what happens is wicd trashes the lan IP address that DHCP issued, trying to fetch on over the wlan and that it does this at odd times, and that possibly lacking an IP address configured on the box exhibits itself with this reported Internal server error. As int he server is still reached (the DNS knows its IP address) but something goes wrong with Apache at a level so low it is a) after the request is logged in the access log but b) Apache does not even bother writing to the error log for some reason - suggesting an internal hernia of such proportions as to break the logging as well that happens after the request is received and logged as an access!

A most bizarre set of circumstances in any case that seem to have been righted by finding and disabling an errant service that runs on Raspbery Pi 3 systems that seems to stubbornly try wlan connections breaking lan connections!

4

Hi
When the error says “server logs” it is referring to nextcloud logs and not apache2.

Can you change your log level to 0
(In nextcloud/config/config.php)
And also post your nextcloud.log from your nextcloud data folder ?

5

Aaargh, it’s back. The wicd service is still disabled and I’m looking this in the face again:

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

So OK, as requested, I set the loglevel to 0. A little more clarification is always helpful so I’ll provide it. I find the config in:

/var/www/nextcloud/config/config.php

loglevel was 2 and I set it 0. These are documented as:

  • 0: DEBUG: All activity; the most detailed logging.
  • 1: INFO: Activity such as user logins and file activities, plus warnings, errors, and fatal errors.
  • 2: WARN: Operations succeed, but with warnings of potential problems, plus errors and fatal errors.
  • 3: ERROR: An operation fails, but other services and operations continue, plus fatal errors.
  • 4: FATAL: The server stops.

I restart Apache:

sudo systemctl restart apache2

I find the log file in:

/media/USBdrive/ncdata/nextcloud.log

given in config.php: 'datadirectory' => '/media/USBdrive/ncdata',

Then I reload my Nextcloud page in the browser. Same error.

But it’s now 21:50 and:

cirrus@nephele:/media/USBdrive/ncdata $ ll nextcloud.log
-rw-r----- 1 www-data www-data 54938221 Dec  3 18:37 nextcloud.log
cirrus@nephele:/media/USBdrive/ncdata $ date
Tue  3 Dec 21:51:40 AEDT 2019

So ZERO degugging output or any output.

To be 100% clear on what I’m getting:

$ wget --server-response https://mynextcloud.server
--2019-12-03 21:59:19--  https://mynextcloud.server/
Resolving mynextcloud.server (mynextcloud.server)... MyIP
Connecting to mynextcloud.server (mynextcloud.server)|MyIP|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 500 Internal Server Error
  Date: Tue, 03 Dec 2019 10:59:19 GMT
  Server: Apache
  Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
  Referrer-Policy: no-referrer
  X-Content-Type-Options: nosniff
  X-Download-Options: noopen
  X-Frame-Options: SAMEORIGIN
  X-Permitted-Cross-Domain-Policies: none
  X-Robots-Tag: none
  X-XSS-Protection: 1; mode=block
  Upgrade: h2,h2c
  Content-Length: 289
  Content-Type: text/plain; charset=utf-8
2019-12-03 21:59:19 ERROR 500: Internal Server Error.

and there remains zero useful output in any log anywhere. Note:

cirrus@nephele:~ $ touch /tmp/foobar
cirrus@nephele:~ $ wget --server-response https://mynextcloud.server
--2019-12-03 21:59:19--  https://mynextcloud.server/
Resolving mynextcloud.server (mynextcloud.server)... MyIP
Connecting to mynextcloud.server (mynextcloud.server)|MyIP|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 500 Internal Server Error
  Date: Tue, 03 Dec 2019 10:59:19 GMT
  Server: Apache
  Strict-Transport-Security: max-age=15768000; includeSubDomains; preload
  Referrer-Policy: no-referrer
  X-Content-Type-Options: nosniff
  X-Download-Options: noopen
  X-Frame-Options: SAMEORIGIN
  X-Permitted-Cross-Domain-Policies: none
  X-Robots-Tag: none
  X-XSS-Protection: 1; mode=block
  Upgrade: h2,h2c
  Content-Length: 289
  Content-Type: text/plain; charset=utf-8
2019-12-03 21:59:19 ERROR 500: Internal Server Error.
cirrus@nephele:~ $ sudo find / -regextype posix-extended -regex "/(sys|srv|proc|dev)" -prune -o -newer /tmp/foobar
/run/sudo/ts/cirrus
/dev
/sys
/var/log/apache2/other_vhosts_access.log
/var/log/apache2/error.log
/var/log/apache2/nc-error.log
/var/log/apache2/nc-access.log
/var/log/auth.log
/var/log/syslog
/proc
/tmp
/tmp/21184a148693539f45e8857900db59f6/var/www/nextcloud/config
/tmp/21184a148693539f45e8857900db59f6/var/www/nextcloud/config/config.php.bin
/srv

remembering (and having double checked) that the apache2 logs which are listed contain zero useful information. The access request appears and the error logs are just full of Authorization requests as I have Apache in debug mode too.

6

Hi Bernd_Wechner,
Can you please post your Config.php ?
(Remember to redact Sensitive information like Database User ID/Password etc)

Further, as of now, ignore Apache2 logs. As the error is “server log” and not “webserver log” we are not concerned with Aapche2 logs.

Also, rename the nextcloud.log to something else like backup_nextcloud.log. Then reboot the system and try accessing the website. This way a new nextcloud.log will be created. Post this new nextcloud.log as it is without applying any sort of filters.

7

OK, happy to oblige. Take not that after a reboot the nextcloud works fine … for a while, could me minutes, hours, days then it’s down again with the internal error. Hard to post files here so I plopped them here:

https://mega.nz/#F!JWZwGQDY!MoHN4sVM4IMtk1aVvklTig

Should be easy to access there. Thanks very kindly for your suggestions and support in scanning them.

I should add there is one oddity visible, in that the logo top left disappeared a while back, around the same time likely that these issues appeared which if memory serves me was after I upgraded it by accepting the prompts suggesting as much when I went to the web interface (which I rarely access to be honest).

image

8

Hi Bernd_Wechner,
I went through the Config.php(Redacted) and nextcloud.log
I didnt find anything that points to an error resulting into an “Internal Server Failure”
Perhaps, as you mentioned, the server was working fine at the time you extracted the logs, because you rebooted it.

Further, I would like to share my thoughts on what might be happening. Taking a wild guess here. There may be some issue with your memcache(redis server) that you have set up in your config.php.
When you reboot your instance your cache is cleared and the instance runs fine. Until the cache memory is some how not accessible or is full or something on those lines.
So I think you should check your cache configuration. Check whether redis is enabled and is recognized by the php (do phpinfo()). Also check which php versions are installed in your system and whether the modules you need for caching are enabled in the version you are currently using.
In my case, I had 2 php versions installed (7.2 and 7.3). All my caching modules were existing and enabled in 7.3 but my apache config was using 7.2 (where no such modules existed) hence I was getting internal server error.
So just make sure which version of php is running and whether that running version consists all modules required for configuring nextcloud cache.

1 Like
9

Thanks so much for checking. Part of me is so annoyed by all this I’m looking at blowing it all away and building a new Nextcloud server. What I’d need for that is a relatively comfortable way of ensuring I can migrate all data from the old instance (which may not even run long enough for the job Grrrr) to anew instance.

But I am also always interested in learning and building my skill set. So if you have sufficient patience I will match it and do all those checks. Alas it’s all a tad over my head. Specifically I’m going to have to read up on:

  1. How to check if a redis is enabled.
  2. How check its cache configuration
  3. How do do phpinfo()
  4. How to check php versions.

Part of thinks you have a very good chance of being right on the money here though because I have recollections of a php upgrade and multitude of php config folders. I will do the required reading now to answer these questions if I can and am not dragged off, in which case it may come another day.

1 Like
10

Moved to separate thread.

This is happening to me too. “Internal Server Error” with no related log entries – I’ve checked nextcloud.log, Apache’s access_log and error_log, and even a php error log (logErrors=On in php.ini).

Mine is a fresh install:
Nextcloud version (eg, 12.0.2): 17.0.1.1
Operating system and version (eg, Ubuntu 17.04): FreeBSD 11.3
Apache or nginx version (eg, Apache 2.4.25): Apache 2.4
PHP version (eg, 7.1): 7.2

The issue has now occurred following two command-line installations. After running into this the first time, I wiped out the installation and installed from scratch a second time. This time, the error didn’t appear at first. In an attempt to fix another problem, I added the following to config/config.php:

'overwritewebroot' => '/nextcloud',
'htaccess.RewriteBase' => '/nextcloud',

I saw the error after doing this but that’s probably a coincidence since the error persists after removing these lines from config/config.php.

Other threads about this error suggest this has to do with memcache which my shared hosting service doesn’t provide by default. Besides, the docs say, with emphasis, “A memcache is not required and you may safely ignore the warning if you prefer.”

Here’s my config without the sensitive lines:

  'trusted_domains' => 
  array (
    0 => 'mycloud.example.com',
  ),
  'datadirectory' => '/path/to/data',
  'dbtype' => 'mysql',
  'version' => '17.0.1.1',
  'overwrite.cli.url' => 'https://mycloud.example.com/nextcloud',
  'installed' => true,
  'twofactor_enforced' => 'true',
  'twofactor_enforced_groups' => 
  array (
  ),
  'twofactor_enforced_excluded_groups' => 
  array (
  ),
  'maintenance' => false,
11 
sudo systemctl status redis

it is in your config.php, line 36

  'redis' => 
  array (
    'host' => '/var/run/redis/redis.sock',
    'port' => 0,
    'timeout' => 0.0,
    'password' => 'password',
  ),

just put this at the top of your index.php file. in fact it’s only phpinfo();. the other two lines should be already there. (and but make a backup of index.php before to restore the original. :wink: )

<?php
phpinfo();
?>

and browse to your web page.

it’s in the phpinfo() output.

12

Make sure that the Redis configuration contains the socket related parameters:

# grep "^unix" <path-to-your-redis-config>/redis.conf
unixsocket /var/run/redis/redis.sock
unixsocketperm 0770
13

since nextcloud is working some minutes this shouldn’t be the problem.

@Bernd_Wechner you may start top (or htop) and watch your free memory.

14

Hi 4slam,
Welcome to the community.

Can you add the following line to your config.php and recheck if logs are generated or not?
'loglevel' = 0

This line enables Debugging Mode for logs.

15

Hi, friend!

First of all: my english is very bad. I’m a brazillian. So, be patient, ok?

Apparently, i have the same issue here. My nextcloud server is hosted in an service provider.

I renamed the nextcloud.log file to nextcloudold.log and try access the web interface again. And the same internal server error happens again.

But, no have a new nextcloud.og file on /nexclouddata directory.

Help me, please. The NC client not connect. The nc web interface not connect. The error message is ```
Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server

16

Thanks so much for all the tips! But unbelievably I did follow all that advice and in the end this caught my eye:

$ find /etc/apache2/ -name '*php*'
/etc/apache2/mods-available/php7.3.load
/etc/apache2/mods-available/php7.3.conf
/etc/apache2/conf-available/php7.3-fpm.conf
/etc/apache2/conf-available/phpmyadmin.conf
/etc/apache2/conf-available/php7.0-fpm.conf
/etc/apache2/mods-enabled/php7.3.load
/etc/apache2/mods-enabled/php7.3.conf
/etc/apache2/conf-enabled/php7.3-fpm.conf
/etc/apache2/conf-enabled/phpmyadmin.conf
/etc/apache2/conf-enabled/php7.0-fpm.conf

Notice that conf-enabled includes the two files php7.3-fpm.conf and php7.0-fpm.conf?

I removed the second one, and since that moment the whole system seems (touch wood) to be stable and operational.

What blows my mind to this day is that apache is so lame that with two conflicting mods enabled it will issue a 500 error to the client but leave no log trace anywhere on the system contrary to claims the 500 error makes! Thus making it impossible to diagnose and leaving us fishing in the dark.

I am decisively not enamoured of apache as a consequence. I’ve been using lighttpd in two other contexts now for years and while I’ve had problems there too, I always had the logs to provide reasons and clues and a path to repair. It would seem to be in get top few guiding light paradigms of web server design to always leave a trace when issuing an error to the client. Even if it’s a daft “Sent 500 error to client on account of unknown crash and a stack trace” message, at least we can find it, see the pulse of life and server side record …

Anyhow, I hope this does fix it for me. I may in the end not need to rush to a sever rebuild!

3 Likes
17

Moved to separate thread.

Thanks for responding.

I set the log level to 0 earlier and also enabled Memcached. My config/config.php now (redacted):

  'trusted_domains' =>                                                                                        
  array (                                                                                                     
    0 => 'mycloud.example.com',                                                                              
  ),                                                                                                          
  'datadirectory' => '/path/to/data',                                                               
  'log_type' => 'file',                                                                                       
  'logfile' => '/path/to/data/nextcloud.log',                                                       
  'logfilemode' => 0660,                                                                                      
  'loglevel' => 0,                                                                                            
  'dbtype' => 'mysql',                                                                                        
  'version' => '17.0.1.1',                                                                                    
  'overwrite.cli.url' => 'https://mycloud.example.com/nextcloud',                                                   
  'memcache.local' => '\OC\Memcache\Memcached',                                                               
  'installed' => true,                                                                                        
  'twofactor_enforced' => 'true',                                                                             
  'twofactor_enforced_groups' =>                                                                              
  array (                                                                                                     
  ),                                                                                                          
  'twofactor_enforced_excluded_groups' =>                                                                     
  array (                                                                                                     
  ),                                                                                                          
  'maintenance' => false,

There’s nothing in my nextcloud.log after the installation related entries. Apache’s access_log shows that the 500 error code is coming from nextcloud/status.php. However, the error/exception details are not getting written to nextcloud.log – its permissions are +rw for user and group (660).

18

Guys, I bring news.

My hosting service provider has informed me that they are renewing their cPanel license. The reason for the renewal is that some license limits have been reached. The webmail service has also stopped.

I believe this is interfering with Nextcloud. I will wait for them to normalize all services. After that, I check the behavior of nextcloud.

1 Like
19

I knew it was something related to php . Kudos on solving your issue. Consider marking today’s post of yours as a solution.

Request 4slam to open a new thread if your issue is not resolved by solution posted by Bernd_Wechner.

1 Like
20

I am indeed relieved too. Also the logo in top left is now there again and not a black box. I wonder if it was somehow inexorably related as well. All quite bizarre. Sometime in the coming year methinks I will migrate the nextcloud to a faster server as the Rpi3 seems a little sluggish in response on the web services. Non issue for sync, it can trickle at leisure, but the wait times on web interaction with it as not exemplary.

1 Like