[resolved] 502 on Nextcloud 'root' after update

Some or all of the below information will be requested if it isn’t supplied; for fastest response please provide as much as you can. :heart:

The Basics

  • Nextcloud Server version (e.g., 29.x.x):
    • 30.0.0
  • Operating system and version (e.g., Ubuntu 24.04):
    • Alpine Linux 3.22.2
  • Web server and version (e.g, Apache 2.4.25):
    • nginx 1.28.0
  • Reverse proxy and version _(e.g. nginx 1.27.2)
    • nginx 1.28.0
  • PHP version (e.g, 8.3):
    • 8.3.26
  • Is this the first time you’ve seen this error? (Yes / No):
    • Yes
  • When did this problem seem to first start?
    • After updating both, my system, and Nextcloud.
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • manual (from archive)
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • No

Summary of the issue you are facing:

I wanted to add a new (Android) device to my Nextcloud installation. Before doing so, I ssh’ed into my server, ran sudo apk update && sudo apk upgrade, cd’ed into my /usr/bin/webapps directory, stopped my webserver and disabled my Nexcloud’s cronjob, dumped my DB into a backup file, mv’ed my nextcloud dir off for backup as well, downloaded and unpacked the 32.0.0 archive, replaced the default config with a link to mine (I keep it under /etc/nextcloud), adjusted file/directory ownerships/permissions (I have a nextcloud and an nginx user, both in the www_data group), restarted my web server, ran occ upgrade as my nexcloud user, re-enabled the cronjob, and rebooted the system. Once back up, I used occ to disable two-factor authentication for a specific user (as the registered device was broken), and logged in as that user via https://nextcloud.mydomain.tld/login. This worked just fine. As expected (based on my configuration), I was asked to setup two-factor auth for that user and registered the new device’s TOTP app.

At this point I noticed I did a mistake during the update: Several apps (incl. calendar and news) didn’t load their icons and clicking on them they failed to load. This issue was that I forgot to re-link my apps-appstore directory (which I keep separate under /var/lib/nexcloud/apps) before the occ::upgrade. I created that link and the app loaded. I went ahead, logged in as admin (directly accessing https://nextcloud.mydomain.tld/settings/apps from my browers’ history) and updated all apps that had newer versions available. Everything worked fine, so I figured my installation was in good shape.

On the new device, I installed the Nextcloud client app, entered nextcloud.mydomain.tld as server but instead of the browser opening to login, an Invalid URL error popped up. I tried explicitly adding the https:// part, but that didn’t help. I figured something might have changed between 31.x and 32.0.0 that required updating my nginx configuration. Thus, I reviewed NGINX configuration — Nextcloud latest Administration Manual latest documentation and made a few adjustments to my config to match the latest recommendations. I restarted the webserver and tried again, but I still couldn’t login via the app. To rule out a typo or networking issue, I copy/pasted https://nextcloud.mydomain.tld from the app to open the same address in the browser where I hit a 502 Bad Gateway error. I switched network, but got the same error. I checked my nginx logs and saw GET requests from both IPs of the new device (before and after switching networks). Eventually, I tried accessing https://nextcloud.mydomain.tld from my computer (again?) but got the same error. However, in another tab, I still had everything working. After a while, I realised that every access to the Nextcloud instance since the update was through a URL requesting a specific path. Adding /login, I can access my instance from browser on either device. Just adding /index.htmlmakes it work. But directly requesting https://nextcloud.mydomain.tld yields a 502 every time. I figure this is what the app must be doing resulting in the Invalid URL error I saw. Checking another (iOS) device that was linked to the instance before, I noticed that my calendars had stopped syncing as well. So currently, I can access my instance only via the web interface.

Steps to replicate it (hint: details matter!):

  1. Visit https://nextcloud.mydomain.tld and get a 502.

  2. Visit https://nextcloud.mydomain.tld/index.html and get a 502 for https://nextcloud.mydomain.tld/index.php.

  3. Visit https://nextcloud.mydomain.tld/login and get a working login screen.

  4. Log in and convince yourself that everything works.

Log entries

Nextcloud

Please provide the log entries from your Nextcloud log that are generated during the time of problem (via the Copy raw option from Administration settings->Logging screen or from your nextcloud.log located in your data directory). Feel free to use a pastebin/gist service if necessary.

I did not find any related entries in the log.

Web Browser

If the problem is related to the Web interface, open your browser inspector Console and Network tabs while refreshing (reloading) and reproducing the problem. Provide any relevant output/errors here that appear.

There is no issue with the web interface itself.

Web server / Reverse Proxy

The output of your ~Apache/~nginx~/system~ log in /var/log/nginx/nextcloud.mydomain.tld_ssl_access.log:

<IP> - - [<time stamp>] "GET / HTTP/2.0" 502 150 "-" "<user agent>"

And /var/log/nginx/nextcloud.mydomain.tld_ssl_error.loghas

<time stamp> [error] 16637#16637: *861 upstream sent too big header while reading response header from upstream, client: <IP>, server: nextcloud.mydomain.tld, request: "GET / HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm83/fpm.sock:", host: "nexcloud.mydomain.tld"

Configuration

Nextcloud

The output of occ config:list system or similar is best, but, if not possible, the contents of your config.php file from /path/to/nextcloud is fine (make sure to remove any identifiable information!):

{
    "system": {
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "logfile": "\/var\/log\/nextcloud\/nextcloud.log",
        "apps_paths": [
            {
                "path": "\/usr\/share\/webapps\/nextcloud\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/lib\/nextcloud\/apps",
                "url": "\/apps-appstore",
                "writable": true
            }
        ],
        "updatechecker": false,
        "check_for_working_htaccess": false,
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "nextcloud.mydomain.tld"
        ],
        "dbtype": "mysql",
        "version": "32.0.0.13",
        "overwrite.cli.url": "https:\/\/nextcloud.mydomain.tld",
        "overwriteprotocol": "https",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "dbindex": 0,
            "password": "***REMOVED SENSITIVE VALUE***",
            "timeout": 2
        },
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [],
        "twofactor_enforced_excluded_groups": [],
        "maintenance": false,
        "loglevel": 0,
        "app_install_overwrite": [
            "twofactor_totp",
            "notes",
            "news",
            "impersonate",
            "gpxedit"
        ],
        "default_phone_region": "<country code>",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "tls",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "587",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "theme": ""
    }
}

Apps

The output of occ app:list (if possible).

Enabled:
  - activity: 5.0.0-dev.0
  - admin_audit: 1.22.0
  - app_api: 32.0.0
  - bruteforcesettings: 5.0.0-dev.0
  - calendar: 6.0.1
  - cloud_federation_api: 1.16.0
  - contacts: 8.0.3
  - contactsinteraction: 1.13.1
  - dav: 1.34.2
  - federatedfilesharing: 1.22.0
  - files: 2.4.0
  - files_downloadlimit: 5.0.0-dev.0
  - files_pdfviewer: 5.0.0-dev.0
  - files_reminders: 1.5.0
  - files_sharing: 1.24.0
  - files_trashbin: 1.22.0
  - files_versions: 1.25.0
  - impersonate: 3.0.0
  - logreader: 5.0.0-dev.0
  - lookup_server_connector: 1.20.0
  - news: 27.0.0
  - nextcloud_announcements: 4.0.0-dev.0
  - notes: 4.12.3
  - notifications: 5.0.0-dev.0
  - oauth2: 1.20.0
  - password_policy: 4.0.0-dev.0
  - photos: 5.0.0-dev.1
  - privacy: 4.0.0-dev.0
  - profile: 1.1.0
  - provisioning_api: 1.22.0
  - recommendations: 5.0.0-dev.0
  - related_resources: 3.0.0-dev.0
  - serverinfo: 4.0.0-dev.0
  - settings: 1.15.1
  - sharebymail: 1.22.0
  - socialsharing_email: 3.3.0
  - support: 4.0.0-dev.0
  - survey_client: 4.0.0-dev.0
  - text: 6.0.0-dev.0
  - theming: 2.7.0
  - theming_customcss: 1.19.0
  - twofactor_backupcodes: 1.21.0
  - twofactor_totp: 14.0.0
  - twofactor_webauthn: 2.4.1
  - updatenotification: 1.22.0
  - user_status: 1.12.0
  - viewer: 5.0.0-dev.0
  - webhook_listeners: 1.3.0
  - workflowengine: 2.14.0
Disabled:
  - circles: 32.0.0 (installed 27.0.1)
  - comments: 1.22.0 (installed 1.17.0)
  - dashboard: 7.12.0 (installed 7.7.0)
  - encryption: 2.20.0 (installed 2.14.0)
  - end_to_end_encryption: 1.17.0 (installed 1.17.0)
  - external: 7.0.0 (installed 7.0.0)
  - federation: 1.22.0 (installed 1.17.0)
  - files_accesscontrol: 3.0.1 (installed 3.0.1)
  - files_external: 1.24.0 (installed 1.19.0)
  - files_retention: 3.0.0 (installed 3.0.0)
  - files_rightclick: 0.15.1 (installed 1.6.0)
  - firstrunwizard: 5.0.0-dev.0 (installed 2.16.0)
  - gpxedit: 0.0.14 (installed 0.0.14)
  - groupfolders: 20.1.1 (installed 20.1.1)
  - mail: 5.5.7 (installed 5.5.7)
  - maps: 1.6.0 (installed 1.5.0)
  - onlyoffice: 9.11.0 (installed 9.11.0)
  - spreed: 22.0.0 (installed 22.0.0)
  - suspicious_login: 10.0.0-dev.0 (installed 5.0.0)
  - systemtags: 1.22.0 (installed 1.17.0)
  - tasks: 0.16.1 (installed 0.16.1)
  - twofactor_nextcloud_notification: 6.0.0-dev.0
  - user_ldap: 1.23.0
  - weather_status: 1.12.0 (installed 1.7.0)

Tips for increasing the likelihood of a response

  • Use the preformatted text formatting option in the editor for all log entries and configuration output.
  • If screenshots are useful, feel free to include them.
    • If possible, also include key error output in text form so it can be searched for.
  • Try to edit log output only minimally (if at all) so that it can be ran through analyzers / formatters by those trying to help you.

Hi @mschilli

Is there anything intersting too see from php8.3-fpm ?

like

journalctl -feu php8.3-fpm

( untested command please adapt to systemctl status | grep fpm )

Regards

@artlog1 Thank you for assisting me.

I am not sure how to adapt your command to my system.
However, /var/log/php83/error.logonly contains a bunch of instances of the same warning:

[<time stamp>] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

@mschilli this is very fine nothing needed to be adapted in fine, we now just have to do what the WARNING is requesting :wink:

pm.max_children setting should be set accoring RAM you have and average memory a php process takes.
If the first is easy the second is a little more tricky.

the file in question has a high probability to be /etc/php/8.3/fpm/pool.d/www.conf

cat /etc/php/8.3/fpm/pool.d/www.conf

read content comment, those explains many things.

increase it, if later on you get memory shortage … you will have to reduce it …

pm.max_children = 10

and restart php-fm

systemctl restart php8.3-fpm

I adjusted /etc/php83/php-fpm.d/www.conf as suggested and restarted the service.
However, the 502 error persists. Also, I don’t really understand how this could be related to a bad gateway error on the root, while everything else works when accessing apps directlty.

@mschilli so that’s not the problem. Let’s find other clues …

Where do you suggest to look? AFAICT, the only error I get is a rather generic nginx failure complaining about the response (header), that results in the 502. With nothing else in the logs, how can I figure out what is happening?

@mschilli

i have no real suggestion, having a deep look on your nginx configuration is something to do, i didn’t see your nginx conf, did you provide it ?

If i was in your situation, i would review again all my changes, might even go back to preivous version and redo things, play with system by trying to disable each application only one at a time to see thing, and testing with a brand new session in browser, such things…

For what it worth, i used to have /etc/ as a git, so i can spot any changes i, or any other people/system did in it. In facts it is usefull because i always mess up at first and this is a good safe net to go back on a previous working setup. But that’s my way and it is too late now even if we set it up.

I did try different browsers. I did not provide my exact nginx configration but besides the SSL setup, it mirrors the reccomended one I linked in my initial post. Could you point me to a guide on how to downgrade my Nextcloud installation so I can confirm it is the upgrade to 32.0.0 that triggered this issue?

@mschilli

i can’t provide you a correct downgrading method yet. Will just try to find the canonical one. Usulay downgrading causes harm if not down with full backup and system snapshots…

I fixed it!

Following up on the one error message I had, I eventually stumbled upon https://www.cyberciti.biz/faq/nginx-upstream-sent-too-big-header-while-reading-response-header-from-upstream.

First I tried just adding to the (HTTPS) server block corresponding to my Nexcloud instance in my nginx config:

     proxy_busy_buffers_size   512k;
     proxy_buffers   4 512k;
     proxy_buffer_size   256k;

This alone did not do the trick as I could still reproduce the error by visiting the root URL of my instance (after restarting the webserver, of course).

However, after I also added

    fastcgi_buffers 16 32k;
    fastcgi_buffer_size 64k;
    fastcgi_busy_buffers_size 64k;

and restarted the webserver again, the error message and the 502 responses dissappeared. My existing devices started syncing again and adding my new device worked as expected.

@artlog1: Thank you for your support in debugging this.

What would be the right place to suggest adding these options to the recommended nginx configuration in the Nextcloud documentation?

1 Like

@mschilli I would use the ‘Edit on GitHub’ button of documentation.

-side note- i’m an independant doing this on my own as a community member, i’m not part of official answer and support at Nextcloud.

1 Like

I am aware. I am active in other FOSS projects as well an I really appreciate the community you (and others) provide.

That’s also why I asked regarding the documentation update, as I think the least I can do is everything in my power to share this information with others.

I’ll head over to GH and open a PR. Thank you for the pointer.

1 Like

Are there any errors above this entry? Or any funny looking ones that look cut-off?

The Nginx buffer related changes you described later are not normally required.

I’m wondering if something else is going on, like continuous errors going to stderr (that are hitting FastCGI).

This would be exasperated if fastcgi.logging is on (it’s off in the community micro-services Docker image due to it being off in the upstream official PHP image)[1].

Also, having Nextcloud’s own debugging on (or, possibly, the loglevel set to 0 as you do), may be factors too.

[1] PHP-FPM stderr logs cause nginx to return 502 errors to client · Issue #878 · docker-library/php · GitHub

1 Like

@jtr: No, this is the only error message I could find anywhere. If you have further suggestion on how to investigate this issue, let me know.