Segmentation fault and "502 Bad Gateway (nginx)"

ℹ️ Support
1

Nextcloud version (eg, 20.0.5): 26.0.7.0
Operating system and version (eg, Ubuntu 20.04): Debian (version unknown)
Apache or nginx version (eg, Apache 2.4.25): 2.4.56
PHP version (eg, 7.4): 8.1.23
Web-Hoster: Webgo www.webgo.de

The issue you are facing:
I only see “502 Bad Gateway (nginx)” when opening my URL.
In the apache log I see “child pid xxxxx exit signal Segmentation fault (11)”
The nextcloud log doesn’t contain any information - even when I set DEBUG loglevel (“loglevel => 0”)

Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it: unable to replicate it - currently it’s permanent, sometimes it only comes once and sometimes it lasts a few hours and vanishes afterwards…

The output of your Nextcloud log in Admin > Logging:

nothing special / related in there…

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'instanceid' => '<instanceid>',
  'passwordsalt' => '<passwordsalt>',
  'trusted_domains' =>
  array (
    0 => '<mydomain1>',
    1 => '<mydomain2>',
    2 => '<mydomain3>',
    3 => '<mydomain4>',
    4 => '<mydomain5>',
    5 => '<mydomain6>',
    6 => '<mydomain7>',
    7 => '<mydomain8>',
    8 => '<mydomain9>',
    9 => '<mydomain10>',
  ),
  'activity_expire_days' => '120',
  'datadirectory' => '/home/www/nextcloud_data/data',
  'filelocking.debug' => false,
  'filelocking.enabled' => true,
  'filelocking.ttl' => 600,
  'overwrite.cli.url' => 'https://<mymainurl>',
  'version' => '26.0.7.0',
  'installed' => true,
  'dbtype' => 'mysql',
  'dbname' => '<dbname>',
  'dbhost' => '127.0.0.1',
  'dbtableprefix' => 'oc_',
  'dbuser' => '<dbuser>',
  'dbpassword' => '<dbpasswd>',
  'defaultapp' => 'files',
  'default_language' => 'de',
  'force_language' => 'de',
  'default_phone_region' => 'DE',
  'integrity.check.disabled' => false,
  'knowledgebaseenabled' => true,
  'enable_avatars' => true,
  'allow_user_to_change_display_name' => true,
  'theme' => '',
  'secret' => '<something>',
  'log_type' => 'file',
  'loglevel' => '0',
  'logdateformat' => 'Y-m-d H:i:s',
  'logtimezone' => 'Europe/Berlin',
  'log_rotate_size' => 20971520,
  'skeletondirectory' => '/home/www/nextcloud_data/skeleton',
  'updatechecker' => true,
  'trashbin_retention_obligation' => 'auto',
  'versions_retention_obligation' => 'disabled',
  'appstoreenabled' => true,
  'appstore.experimental.enabled' => true,
  'singleuser' => false,
  'maintenance' => false,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'htaccess.RewriteBase' => '/',
  'mail_smtpmode' => 'smtp',
  'updater.release.channel' => 'stable',
  'mail_smtpsecure' => 'ssl',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_from_address' => 'Cloud',
  'mail_smtpauth' => 1,
  'mail_domain' => 'mymail.domain',
  'mail_smtphost' => 'mymailhost',
  'mail_smtpport' => '465',
  'mail_smtpname' => 'somewhere@the.internet',
  'mail_smtppassword' => '<something>',
  'mysql.utf8mb4' => true,
  'jpeg_quality' => '80',
  'share_folder' => '/',
  'simpleSignUpLink.shown' => false,
  'data-fingerprint' => '<something>',
  'mail_sendmailmode' => 'smtp',
  'memories.exiftool' => '/home/www/nextcloud/apps/memories/exiftool-bin/exiftool-amd64-glibc',
  'enabledPreviewProviders' =>
  array (
    0 => 'OC\\Preview\\Image',
    1 => 'OC\\Preview\\HEIC',
    2 => 'OC\\Preview\\TIFF',
    3 => 'OC\\Preview\\Movie',
    4 => 'OC\\Preview\\MKV',
    5 => 'OC\\Preview\\MP4',
    6 => 'OC\\Preview\\AVI',
  ),
  'enable_previews' => false,
  'preview_max_memory' => 2048,
  'preview_max_filesize_image' => 256,
  'preview_max_x' => 1024,
  'preview_max_y' => 1024,
  'preview_concurrency_new' => '1',
  'memories.ffmpeg_path' => '/usr/bin/ffmpeg',
  'memories.ffprobe_path' => '/usr/bin/ffprobe',
  'memories.transcoder' => '/home/www/nextcloud/apps/memories/exiftool-bin/go-vod-amd64',
  'memories.no_transcode' => false,
  'memories.qsv' => false,
  'memories.vod.path' => '/home/www/nextcloud/apps/memories/exiftool-bin/go-vod-amd64',
  'memories.vod.ffmpeg' => '/usr/bin/ffmpeg',
  'memories.vod.ffprobe' => '/usr/bin/ffprobe',
  'updater.secret' => '<something>',
);

The output of your Apache/nginx/system log in /var/log/____:

[Fri Sep 22 13:35:39 2023] [notice] [client AH00163: Apache/2.4.56 (Debian) SVN/1.14.1 OpenSSL/1.1.1n configured -- resuming normal operations
[Fri Sep 22 13:35:39 2023] [notice] [client AH00094: Command line: '/usr/sbin/apache2'
[Fri Sep 22 13:35:44 2023] [notice] [client AH00052: child pid 1342800 exit signal Segmentation fault (11)
[Fri Sep 22 13:35:49 2023] [notice] [client AH00052: child pid 1342882 exit signal Segmentation fault (11)
[Fri Sep 22 13:35:57 2023] [notice] [client AH00052: child pid 1342801 exit signal Segmentation fault (11)
[Fri Sep 22 13:35:58 2023] [notice] [client AH00052: child pid 1342911 exit signal Segmentation fault (11)

I had the problem nearly every time after an upgrade since v22 (if I can remember correctly) - which then suddenly vanishes after a certain time (sometimes after a few hours, sometimes after minutes).
Yesterday I upgraded my instance from 25.0.10 to 25.0.12 and then to 26.0.7 - after that upgrade I had the error once - but after a restart of the apache server everything worked.
Today I suddenly found out, that the failure reappears - now permanently.
When I use the command line occ command, everything seems to be O.K. - even the cron job runs in it’s cycle.
But the web GUI is totally inaccessible…

What’s going on here?
Too less memory?
Database problems?

2

Could be. I’d check if less apps or less apache/php processes change this. You can also try to monitor the memory or in general the resource consumption (top, iotop etc).
You don’t have redis, so for the file locking there can be a considerable load on the database especially with no optimization.

3

Well, it’s a share hoster so everything is limited… :wink:
I can use 2048M on RAM memory and I’v a performance of “12 customers per CPU” - whatever that means.
So I set “memory_limit = 2048M” in my php.ini…

I also have ssh access - but that is limited, too and unfortunately I’m not able to execute top, iotop etc. - so it’s a bit a search in the dark…

Where can I change the number of php processes / which apache parameter is this?
(ulimit -u is 130 if that matters…)

I’m also unable to use / install redis - that’s not part of the share hoster’s offer… (I already asked for that but they didn’t want to install it).

So I assume that memory consumption got bigger for every NC version?

4

Well, if you use all the RAM for your php processes, how much is left for your database and webserver?

I haven’t kept track of this, but if you add features you are likely to use more resources. If it is just regarding apps, you can disable them.

Yes, you can try to optimize everything a bit and try to make it work. If you fail at some point, a vserver or similar will be better since you have better control over the system to optimize (and also adapt the needed resources). For shared hoster, it’s optimized for “normal” websites whereas Nextcloud is a bit of a special case.

5

The hosters says that that RAM is for my “scripts” and software - so database and apache etc. should not be included…

I tried to disable as much as possible - but that didn’t help… :frowning:

I already tried to optimize things - nothing helps.
I assume a vserver would help - but…
…currently the webhosting costs me about 13 Eur/month (including 3 domains and 300 GB harddisk space).
A vserver would cost me at least 120 Eur / month at my current provider - that’s way too expensive for me (it’s just for personal use - used by me, my family and friends in a small departement from my sports club).

When I read the system requirements from the nextcloud documentation it’s said that 128MB RAM per process is the minimum and 512MB recommended.
So 2048MB RAM should be enough to run a server with most of the time only one connected user…
How can I find out, what the “real” requirements are?

And what other (despite a vserver) possibilities do I have now that my server does not work at all?
I need at least a fast temporary solution to make it work again.
Would it be possible to downgrade back to v25?
I know it’s not supported - but would it be possible?
(however I didn’t make a backup of the database before the upgrade :frowning: - would the v26 database structure be compatible to v25?)

6

I had a longer chat session with the support team of my hosting provider.

After long tries and checks they found out, that changing the PHP version from 8.1 to 8.2 makes the installation work again! :open_mouth:
I have no idea why (as regarding the nextcloud documentation 8.1 is the recommended version for v26) - but I’m very happy that it works again :upside_down_face:

And also regarding the support team the memory limit of 2048M could not be the problem - as it was never ever reached during the checks (they told me that the PHP session processes most of the time user less than 512M and sometimes go up to 768M - but not higher - don’t know how they checked this…)

So now the big question: should I upgrade to v27.1 or stay at v26 as long as possible? :thinking:

EDIT:
(I ask about the upgrade because I still have problems - when I enable the photos app it doesn’t take long before I got an “internal server error” because the number of database connections to my mariadb exceed the maximum - I then can wait until the sessions ran into a timeout and kill themselve - after that I’m able to disable the photos app again at command line…)

7

At some point you need to upgrade, version are only supported for a limited time (NC 26 until March 2024). If you knew why it got faster for php 8.2, you could check if they worked in further direction to improve this. But this is hard to figure out, such experience probably depend on the environment of your setup, so perhaps your hoster has some experience themselves.
If you can take a snapshot of the current setup, you could do a backup just before the upgrade. In case it is much worse, you can revert to the 26 version and figure out a different solution.

There are cheaper offers, the storage is probably the critical factor, but perhaps for about 20€/month. Or even a dedicated server (not last generation), you can get for 30-50€/month. However, you have then to manage a system and that is something I’d try and checkout locally first (either on a raspberry pi, virtual machine, old computer, …).
Or if you happen to live where you can have a fiber connection at home (or otherwise a fast upstream connection), that might be an idea as well.

8

The offers here in Germany are very expensive - I found some for 59 Eur / month with enough space - but that’s way too expensive for a private page!

I really would like to host it at a server at home - but my Internet connection (in upload) is not good enough (40/10 MBit/s); and I can’t get more - not possible :frowning:

The big question about my Nextcloud installation is if the current problems are because of some limits of my hoster or because something went wrong and e.g. my installation/database is somehow corrupt.
Would it be better when I try to upgrade to v27 or would it be better when I try (if possible) to downgrade to v25 (where everything was OK and working).
My calendars are only working partially, I’m unable to enable Photos (or the old Gallery+ app) without segmentation faults, “occ preview:generate-all” or similar hangs when I execute it,…
…the whole installation is corrupt and only working in parts - and that’s really annoying!!!

9

You can have older dedicated servers (Serverbörse | Server günstig ersteigern), ovh has similar offers. For vServers there are several, if they are too cheap, you tend to share with more users that impacts the performance.

This is hard to tell for sure. For servers, we can say that optimizing the database caches, and using all the caching mechanisms, change a lot the performance of your server. And the default settings that might be good for some webapplications are perhaps not the best for Nextcloud. And worst case, you should get something that is slow at some point (might also happen once you use it with too many users) but conflicts and corruption in the database are not good.
We had some people here, where the shared hosting stuff just didn’t work, others got some performance they were happy with, but it were rather small setups (few users).

10

Would it to be possible to make a “fresh” installation based on the existing configuration file with existing data directory but with changed database (a new and empty database) and then copy parts from the old database to the new one via SQL?
At least the users and their data should be transferred via SQL; other things (calendars, address books) should be transferred by importing exported files…

When I try to start a new installation it starts all over and asks about database, admin user etc. - but I want to use the existing config.php (as said: with changed database connection information)…

11

I would either migrate all (config.php + data + database), that should be the easiest and shouldn’t take too long. In the end the users can keep all their clients and configurations and it can be totally transparent for them.

Some mixed state is difficult. You should avoid to manually manipulate the database, so if it is just for the users, I’d use a script and use the API to create all the users that you currently have. And then you can copy back the files, adjust permissions and run a occ filescan once. But then all clients need to resync, you still need to care about other app data (calendar, contacts, …).

12

But if my problem are corrupt database entries it would copy that corruption to a new installation…

That why I try to create an “empty” database - but with my existing config.php (e.g. with the old database tables prefix) - then I could put back the database content step for step and recheck when I get “strange” behaviour…
(and yes, I think I’m able to put it back with SQL statements…

13

The bad gateway looks more like a problem between webserver<->php (or could be if you use proxy servers on top of that). Database errors might cause different type of log entries.

You can start fresh, but then I would start with a completely new setup. Or you try to recover your old one, but I wouldn’t do a mix. I usually give myself 1-2h to get the old running and if it fails within that time, I use a new setup.

14

Just do bring this thread to an end…

In-between I did another upgrade to NC v27.1.2 on my existing instance.
After that upgrade everything seems to work again!

I still have no idea what raises this segmentation faults - but when it works again it’d be OK for me…
… I hope it keeps going

@tflidd: thank you very much for your help!

2 Likes
15

With this hint I was able to upgrade from 26.0.5 to 26.0.6:

is the solution.

Adding “(string)” into line 8 of /var/www/html/config/redis.config.php lets start the instance fine!
Maybe vi needs to be installed first: apt update && apt install -y nvi
no change of redis (version 5.*) needed…