Bad Gateway after everything was running fine

I’m having some problems figuring this out for myself so I’m asking here.
It was running fine for about a week now.

Nextcloud version: 13.0.2.1
Operating system and version: Raspian Stretch Lite build from 18. April 18
Apache or nginx version: nginx 1.10.3
PHP version: 7.0

The issue you are facing:
I can’t access NC over my DNS-Server; I’m getting 502 Bad Gateway
Accessing via IP while in the same LAN works.
I didn’t make any changes to my DNS-Server so I’m not sure why it stopped working when it was working just 2 days ago.
I also checked that the dns directs to the correct ip.
The desktop client returns this error:

Error transferring https://mydns.org/remote.php/webdav/ - server replied: Bad Gateway

Is this the first time you’ve seen this error?: Yes

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

$CONFIG = array (
  'instanceid' => 'xx',
  'passwordsalt' => 'xx',
  'secret' => 'xx',
  'trusted_domains' =>
  array (
    0 => 'mydns',
  ),
  'datadirectory' => '/var/www/html/data',
  'overwrite.cli.url' => 'https://mydns',
  'dbtype' => 'pgsql',
  'version' => '13.0.2.1',
  'dbname' => 'nextcloud',
  'dbhost' => 'localhost',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'xx',
  'dbpassword' => 'xx',
  'installed' => true,
  'memcache.local' => '\OC\Memcache\APCu',
);

The output of your Apache/nginx/system log in /var/log/____:

nginx error.log:

2018/05/13 10:07:15 [error] 32740#32740: *13785 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:07:46 [error] 32740#32740: *13788 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:08:18 [error] 32740#32740: *13791 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:08:50 [error] 32740#32740: *13794 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:09:35 [error] 32740#32740: *13799 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:09:54 [error] 32740#32740: *13802 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:10:26 [error] 32740#32740: *13805 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:10:58 [error] 32740#32740: *13808 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:11:30 [error] 32740#32740: *13811 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:12:04 [error] 32740#32740: *13817 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:13:26 [error] 32740#32740: *13819 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:13:27 [error] 32740#32740: *13819 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "GET /favicon.ico HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 10:13:28 [error] 32740#32740: *13819 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "GET /favicon.ico HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 11:39:59 [error] 32740#32740: *13827 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "GET /favicon.ico HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 11:41:26 [error] 32740#32740: *13830 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 11:57:44 [error] 32740#32740: *13837 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "PROPFIND /remote.php/webdav HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 11:57:45 [error] 32740#32740: *13837 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "GET /ocs/v2.php/cloud/user?format=json HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 11:57:45 [error] 32740#32740: *13837 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "GET /ocs/v1.php/cloud/capabilities?format=json HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"
2018/05/13 11:57:45 [error] 32740#32740: *13837 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 192.168.2.1, server: mydns.org, request: "GET /ocs/v2.php/cloud/activity?format=json HTTP/2.0", upstream: "fastcgi://unix:/var/run/php/php7.0-fpm.sock:", host: "mydns.org"

Hi,

It actually means, that the upstream doesn’t provide a response.
Could you check your php-fpm service? Maybe there was some php(-fpm) update and something failed? Or an update changed the config.
Just make sure

/var/run/php/php7.0-fpm.sock

exists and can be accessed,

@Schmu I seem to be running into this issue. I checked /var/run/php7.2-fpm.sock and it does not exist (I’m running php 7.2 for this server). What does that mean?

Hi,

You need to check your php-fpm configuration then. Look for the php-fpm.d folder and check the contained config files.
For it is:

 /etc/php/php-fpm.d/www.conf

You wanna look for the parameter: listen =
and check what the actual socket is and where it can be found. For me it is:

/run/php-fpm/php-fpm.sock

So make sure your php-fpm is set to socket mode (if you want to use it that way) instead of a port and compare this setup with your webserver config to match the socket file configuration.

If that’s fine check the file permissions as well and make sure your web server user, can access the socket.
While my web server user is http my permissions are:
srw-rw---- 1 http http 0 Oct 18 00:09 /run/php-fpm/php-fpm.sock

Let me know if that helped or if you need some more information.

Hey Schmu,

Thank you for the reply. I ended up finding the issue. It had to do with me putting an array in the global section of the php-fpm config files where it wasn’t supposed to go.

1 Like

Since I have not seen my problem with the very same symptoms this may help others with the same search terms: nginx calls php-fpm via a local socket connection (here: /var/run/php/php7.0-fpm.sock). FPM creates a pool of processes for handling requests. And here the things went bad. For debugging such issues, the easiest way is to enable stderr/stdout via:
/etc/php/7.0/fpm/pool.d/www.conf

; Redirect worker stdout and stderr into main error log. If not set, stdout and
; stderr will be redirected to /dev/null according to FastCGI specs.
; Note: on highloaded environement, this can cause some delay in the page
; process time (several ms).
; Default Value: no
catch_workers_output = yes

FPM logs usually into /var/log/php7.0-fpm.log:

WARNING: [pool www] child 16089 said into stderr: “*** Error in `php-fpm: pool www’: malloc(): memory corruption: 0x00000055b32f9ad0 ***”
WARNING: [pool www] child 16089 said into stderr: “Magick: abort due to signal 6 (SIGABRT) “Abort”…”
WARNING: [pool www] child 16089 exited on signal 6 (SIGABRT) after 36.319160 seconds from start

This let’s the connection die and causes the 502 Bad Gateway without any network connection.
So the cause was the updated version of php-imagick (experimental cross-compiled for ARM, so could have been expected). Since nextcloud provides a fallback, calling “apt remove php-imagick” fixed my issue. Note that this impacts performance. The real fix would be to properly patch php-imagick if there’s time left…

1 Like

Thank you very much, to disable php-imagick solved my problem :slight_smile: php-imagick should be disabled because of security risk reasons.