Nextcloud 27 [solved]Zend OPcache is generating 502 Bad gateway error

ℹ️ Support
,
1

My nextcloud displays error 502 when OPcache is enabled.

I am running nextcloud on centos 8
nginx version: nginx/1.22.1
PHP 8.2.8
[Nextcloud] 27.0.1

journalctl

php-fpm[2425]: segfault at 55dc6e620990 ip 00007fb3569688bc sp 00007ffdcb291870 error 4 in opcache.so[7fb35694c000+e3000]
Code: 49 89 d4 55 53 48 83 ec 18 48 8b 9f f0 00 00 00 48 85 db 75 15 e9 fc 00 00 00 0f 1f 40 00 48 8b 1b 48 85 db 0f 84 ec 00 00 00 <4c> 3b 6b 10 75 ee 41 8b 86 a4 01 00 00 41 03 86 a0 >
php-fpm[2426]: segfault at 55dc6e620990 ip 00007fb3569688bc sp 00007ffdcb291870 error 4 in opcache.so[7fb35694c000+e3000]

opcache.ini settings (current)

zend_extension=opcache
opcache.enable=0
opcache.enable_cli=0
opcache.memory_consumption=256
opcache.interned_strings_buffer=100
opcache.max_accelerated_files=10000
opcache.optimization_level=0x7FFFBFFF
opcache.blacklist_filename=/etc/php.d/opcache*.blacklist
opcache.huge_code_pages=0

nginx logging

2023/07/17 04:45:32 [error] 733353#0: *93228 recv() failed (104: Connection reset by peer) while reading response header from upstream, client

2

Perhaps it’s the problem that it isn’t enabled.

3

when
opcache.enable=1
opcache.enable_cli=1
my nextcloud is ending up in 502 bad gateway

4

Could you provide some information about your system resources, like the output of free?

Without hanging out too far out of the window (I am not a centos specialist), I would like to recommend following changes for you to try:

zend_extension=opcache
opcache.jit=off
opcache.enable=1
opcache.enable_cli=1
opcache.memory_consumption=192
opcache.interned_strings_buffer=32
opcache.max_accelerated_files=32530
opcache.max_wasted_percentage=10
opcache.use_cwd=0
opcache.validate_timestamps=1
opcache.revalidate_freq=1
opcache.enable_file_override=1
; opcache.optimization_level=0x7FFFBFFF
; opcache.blacklist_filename=/etc/php.d/opcache*.blacklist
; to log specific opcache errors use temporary:
opcache.error_log=/var/log/php/opcache_fpm_error.log
; opcache.huge_code_pages=0

Hope this helps,
much luck!

5

unfortunately changing to the supposed settings didn’t work, again 502 bad gateway.
I’ve disabled tOPcache to get it going. see asked details below
ohw, in the mean while I am building a new server on CentOS stream 9, also this installation, which is new is having the same sympthoms 502 bad gateway on an enabled OPcache.(php 8.2)
An other bit information, it is a VM (2 vcpu on 1 socket and 4Gb mem) this makes that a process can consume more then 2 Gb without NUMA effects.
Sorry to say that my php experience is not on that level, but it looks like a php-fpm problem.

numastat -n

numastat -n

Per-node numastat info (in MBs):
                          Node 0           Total
                 --------------- ---------------
Numa_Hit               572860.38       572860.38
Numa_Miss                   0.00            0.00
Numa_Foreign                0.00            0.00
Interleave_Hit             82.86           82.86
Local_Node             572860.38       572860.38
Other_Node                  0.00            0.00
opcache.ini settings (current) 
# cat /etc/php.d/10-opcache.ini | grep -v "^;" | grep -v "^$"
zend_extension=opcache
opcache.enable=0
opcache.enable_cli=1
opcache.memory_consumption=192
opcache.interned_strings_buffer=32
opcache.max_accelerated_files=32530
opcache.jit=off
opcache.use_cwd=0
opcache.validate_timestamps=1
opcache.revalidate_freq=1
opcache.enable_file_override=1
opcache.protect_memory=0
opcache.huge_code_pages=0
free -h 
# free -h
              total        used        free      shared  buff/cache   available
Mem:          3.6Gi       1.2Gi       726Mi       0.0Ki       1.6Gi       2.1Gi
Swap:         2.1Gi        42Mi       2.0Gi
6

I agree. There’s not really much NC can do about this.

Even most variations of opcache settings shouldn’t trigger this (I don’t think) unless there is a deeper underlying problem.

Likely an upstream (GitHub - php/php-src: The PHP Interpreter) matter or possibility some weird CentOS Stream build/packaging/runtime interaction. Or maybe a hardware or VM platform issue.

The 502 bad gateway is sort of a red herring. It’s just a generic error occurring because php-fpm is bombing out.

Since PHP 8.2.8 is < 30 days old, a PHP bug does seem a possibility.

7

Out of curiosity, does this happen if you just use opcache 100% without any config customizations (i.e. all defaults).

Also, I did just notice in your first config you used

opcache.optimization_level=0x7FFFBFFF

But 0x7FFEBFFF is the present default.

Changed from 0x7FFFBFFF in PHP 7.3.0

https://www.php.net/manual/en/opcache.configuration.php

8

I did try with this configuration, but also when I enable OPcache it ended up in a 502 bad gateway.
The fact I am running 8.2 was due a problem I had with 8.0, after upgrade to 8.1 it was even worse, then upgrade to 8.2 what is running more smooth than 8.0 except the problem with OPcache.

Now the challenge is how to bring this issue to the php developers, because I am not a developer, also not that familiar with the php language. I can deliver all kinds of ligging if needed.

/etc/php.d/10-opcache.ini 
# cat /etc/php.d/10-opcache.ini | grep -v "^;" | grep -v "^$"
zend_extension=opcache
opcache.enable=0
opcache.enable_cli=1
nginx/error.log 
2023/07/30 19:49:35 [error] 135826#0: *27 recv() failed (104: Connection reset by peer) while reading      response header from upstream, client: <ip_address>, server: <host., request: "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: <host>"
/var/log/php-fpm/error.log 
[30-Jul-2023 19:49:35] WARNING: [pool www] child 135848 exited on signal 11 (SIGSEGV) after 6.420728 seconds from start
[30-Jul-2023 19:49:35] NOTICE: [pool www] child 135853 started
9

What problem were you having?

10

I had some problems with calendar and Deck, where I dis some searches on the internet witch indicated that the problems should be solved in php 8.1. At that time php was also recommended by NC.
Right now php8.2 is recommended, maybe more users are experiencing this error when running dualstack ip configuration.

After a lot of testing following logs, at the end it was a configuration in config.config.php involving the trusted proxies setting. I dont know why but I exchanged both addresses for the DNS hostname

With 502 error 
 'trusted_proxies' =>
  array (
    0 => 'xxx.xxx.xxx.xxx',
    1=>'[xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx]',
),
without 502 bad gateway 
  'trusted_proxies' =>
  array (
    0 => 'hostname.domain.tld',
),

It seems to work fine now.

1 Like