Caddy & notify_push setup: Proxy stripped X-Forwarded-For Header

The Basics

  • Nextcloud Server version:
    • 33.0.0
  • Operating system and version:
    • Debian Trixie
  • Web server and version:
    • Caddy v2.11.2
  • PHP version:
    • 8.4
  • Is this the first time you’ve seen this error?
    • Yes
  • When did this problem seem to first start?
    • Some days ago
  • Installation method (e.g. AlO, NCP, Bare Metal/Archive, etc.)
    • NextcloudPi, now Bare metal
  • Are you using CloudfIare, mod_security, or similar? (Yes / No)
    • No

Summary of the issue you are facing:

I want to setup notify_push. I dug through the whole internet (~60 different pages visited) but still cant finish the setup.

I followed the installation guide for notify push but when doing the setup, this is the output:

ncc notify_push:setup https://wolke.numerfolt.de/push
✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
🗴 push server is not a trusted proxy by Nextcloud or another proxy in the chain.
  Nextcloud resolved the following client address for the test request: "MY PUBLIC IP" instead of the expected "1.2.3.4" test value.
  The following trusted proxies are currently configured: "127.0.0.1", "::1", "MY PUBLIC IP"
  The following x-forwarded-for header was received by Nextcloud: "MY PUBLIC IP"
    from the following remote: MY PUBLIC IP

✓ All proxies in the chain appear to be trusted by Nextcloud
  One of the proxies is the chain (probably MY PUBLIC IP) seems to have stripped the x-forwarded-for header
  Please configure the reverse proxy at MY PUBLIC IP to not strip the x-forwarded-for header

  If you're having issues getting the trusted proxy setup working, you can try bypassing any existing reverse proxy
  in your setup by setting the `NEXTCLOUD_URL` environment variable to point directly to the internal Nextcloud webserver url
  (You will still need the ip address of the push server added as trusted proxy)

I had a setup before (not sure if it worked), but it somehow broke and now when I am on my cloud website, the browser console logs the following error:
Firefox kann keine Verbindung zu dem Server unter wss://wolke.local/push/ws aufbauen.

Somehow notify_push wants to connect to the old local name of the server. At the moment its address is just wolke not wolke.local. I dont even know why it would use that old value…

Steps to replicate it:

  1. Install the notify push (Client Push) app from the nextcloud app store

  2. Create the systemd service and start it

  3. Set the trusted proxies in config.php

  4. Adding the reverse_proxy entry in the config file for my cloud

  5. Calling the setup function

Log entries

Nextcloud

Please provide the log entries from your Nextcloud log that are generated during the time of problem (via the Copy raw option from Administration settings->Logging screen or from your nextcloud.log located in your data directory). Feel free to use a pastebin/gist service if necessary.

No log entries while calling setup function

Web Browser

If the problem is related to the Web interface, open your browser inspector Console and Network tabs while refreshing (reloading) and reproducing the problem. Provide any relevant output/errors here that appear.

Firefox kann keine Verbindung zu dem Server unter wss://wolke.local/push/ws aufbauen. 2 index.js:91:30

[DEBUG] user_status: Failed sending heartbeat, got: 400 
Object { level: "2", app: "user_status" }
​
app: "user_status"
​
level: "2"
​
<prototype>: Object { … }
index.mjs:38:17

GET
wss://wolke.local/push/ws
NS_ERROR_UNKNOWN_HOST

Web server / Reverse Proxy

The output of your Apache/nginx/system log in /var/log/:

{"level":"info","ts":"2026-04-07T02:07:11.631+0200","logger":"http.log.access.log3","msg":"handled request","request":{"remote_ip":"84.151.237.1","remote_port":"59267","client_ip":"84.151.237.1","proto":"HTTP/2.0","method":"POST","host":"wolke.numerfolt.de","uri":"/index.php/apps/notify_push/pre_auth","headers":{"X-Requested-With":["XMLHttpRequest, XMLHttpRequest"],"Sec-Fetch-Dest":["empty"],"Accept":["application/json, text/plain, */*"],"Accept-Language":["de,en-US;q=0.9,en;q=0.8"],"Accept-Encoding":["gzip, deflate, br, zstd"],"Sec-Fetch-Mode":["cors"],"Te":["trailers"],"Sec-Fetch-Site":["same-origin"],"Requesttoken":["6gZUamr0zxyyShcgV6mKDtmOmp6fBUMkF6dUlWH8RCw=:vzZ/AQSlhWXgZXkSEPHlSImh0/zuPCpReNcT+1S5J0s="],"Sec-Gpc":["1"],"Content-Length":["0"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:149.0) Gecko/20100101 Firefox/149.0"],"Cookie":["REDACTED"],"Dnt":["1"],"Origin":["https://wolke.numerfolt.de"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"wolke.numerfolt.de","ech":false}},"bytes_read":0,"user_id":"","duration":0.181679888,"size":32,"status":200,"resp_headers":{"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Permitted-Cross-Domain-Policies":["none"],"Content-Length":["32"],"Content-Disposition":["inline; filename=\"\""],"Strict-Transport-Security":["max-age=31968000; includeSubDomains; preload"],"Content-Security-Policy":["default-src 'none';base-uri 'none';manifest-src 'self';frame-ancestors 'none'"],"X-User-Id":["numerfolt"],"X-Content-Type-Options":["nosniff"],"Content-Type":["text/html; charset=UTF-8"],"Permissions-Policy":["interest-cohort=()"],"X-Request-Id":["RvdGER9YxESYAAdfPkmC"],"Cache-Control":["no-cache, no-store, must-revalidate"],"X-Robots-Tag":["noindex, nofollow"],"X-Frame-Options":["SAMEORIGIN"],"Feature-Policy":["autoplay 'none';camera 'none';fullscreen 'none';geolocation 'none';microphone 'none';payment 'none'"],"Referrer-Policy":["no-referrer"]}}

Configuration

Caddy

wolke.numerfolt.de {
    root * /var/www/nextcloud

    tls ***REMOVED SENSITIVE VALUE***

    header {
        Permissions-Policy interest-cohort=()
        Strict-Transport-Security "max-age=31968000; includeSubDomains; preload"
    }
    
    @cache {
        path *.js *.css *.csv *.png *.jpg
    }
    
    header @cache {
        Cache-Control "max-age=604800, s-maxage=604800, public"
    }
    
    header -server
    header -via
    
    # .htaccess / data / config / ... shouldn't be accessible from outside
    @forbidden {
        path /.htaccess
        path /data/*
        path /config/*
        path /db_structure
        path /.xml
        path /README
        path /3rdparty/*
        path /lib/*
        path /templates/*
        path /occ
        path /console.php
    }
    respond @forbidden 404

    # Redirect CardDAV and CalDAV endpoints
    redir /.well-known/carddav /remote.php/dav/ 301
    redir /.well-known/caldav /remote.php/dav/ 301
    redir /.well-known/openid-configuration /index.php/apps/oidc/openid-configuration/ 301
    redir /.well-known/webfinger /index.php/.well-known/webfinger 301
    redir /.well-known/nodeinfo /index.php/.well-known/nodeinfo 301

    # Nextcloud push (mobile notifications)
    route /push/* {
        uri strip_prefix /push
        reverse_proxy http://127.0.0.1:7867
    }

    php_fastcgi unix//run/php/php8.4-fpm.sock {
        # Needed to remove index.php from URLs. Does not work with rainloop.
        #env front_controller_active true
        trusted_proxies private_ranges
    }

    file_server

    log {
        format json {
            time_format iso8601
        }
        output file /var/log/caddy/wolke/logfile.log {
        roll_size 10MB
        roll_keep_for 14d
        roll_uncompressed  
        }
    }
}

Nextcloud

The output of occ config:list system or similar is best, but, if not possible, the contents of your config.php file from /path/to/nextcloud is fine (make sure to remove any identifiable information!):

{
    "system": {
        "serverid": 12,
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": {
            "0": "wolke.numerfolt.de",
            "1": "192.168.17.99",
            "2": "127.0.0.1",
            "3": "https:\/\/wolke.numerfolt.de",
            "11": "2003:f3:572a:2300:4946:50c0:3fcc:8a46"
        },
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "33.0.0.16",
        "overwrite.cli.url": "https:\/\/wolke.numerfolt.de\/",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 0,
            "password": "***REMOVED SENSITIVE VALUE***"
        },
        "tempdirectory": "\/opt\/ncdata\/tmp",
        "mail_sendmailmode": "smtp",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauth": 1,
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "ssl",
        "preview_max_x": "2048",
        "preview_max_y": "2048",
        "jpeg_quality": "60",
        "overwriteprotocol": "https",
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false,
        "logfile": "\/opt\/ncdata\/nextcloud.log",
        "loglevel": "2",
        "log_type": "file",
        "htaccess.RewriteBase": "\/",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "default_phone_region": "DE",
        "data-fingerprint": "83701efefe2359b38b341b97f0ab6fe9",
        "memories.exiftool": "\/var\/www\/nextcloud\/apps\/memories\/exiftool-bin\/exiftool-aarch64-glibc",
        "memories.gis_type": 1,
        "memories.vod.path": "\/var\/www\/nextcloud\/apps\/memories\/exiftool-bin\/go-vod-aarch64",
        "maintenance_window_start": 2,
        "app_install_overwrite": [
            "apporder"
        ],
        "theme": "",
        "trashbin_retention_obligation": "30, auto",
        "config_preset": 2,
        "updater.secret": "***REMOVED SENSITIVE VALUE***"
    }
}

Apps

The output of occ app:list (if possible).

Enabled:
  - activity: 6.0.0-dev.0
  - admin_audit: 1.23.0
  - bruteforcesettings: 6.0.0-dev.0
  - calendar: 6.2.2
  - circles: 33.0.0
  - cloud_federation_api: 1.17.0
  - comments: 1.23.0
  - contacts: 8.4.3
  - cookbook: 0.11.6
  - dashboard: 7.13.0
  - dav: 1.36.0
  - federatedfilesharing: 1.23.0
  - files: 2.5.0
  - files_downloadlimit: 5.1.0-dev.0
  - files_pdfviewer: 6.0.0-dev.0
  - files_reminders: 1.6.0
  - files_sharing: 1.25.2
  - files_trashbin: 1.23.0
  - files_versions: 1.26.0
  - forms: 5.2.5
  - guests: 4.6.0
  - logreader: 6.0.0
  - lookup_server_connector: 1.21.0
  - music: 3.0.0
  - nextcloud_announcements: 5.0.0
  - notifications: 6.0.0
  - notify_push: 1.3.1
  - oauth2: 1.21.0
  - password_policy: 5.0.0-dev.0
  - previewgenerator: 5.13.0
  - privacy: 5.0.0-dev.0
  - profile: 1.2.0
  - provisioning_api: 1.23.0
  - recommendations: 6.0.0-dev.0
  - related_resources: 4.0.0-dev.0
  - serverinfo: 5.0.0-dev.0
  - settings: 1.16.0
  - sharebymail: 1.23.0
  - spreed: 23.0.3
  - systemtags: 1.23.0
  - tasks: 0.17.1
  - text: 7.0.0-dev.3
  - theming: 2.8.0
  - twofactor_backupcodes: 1.22.0
  - twofactor_totp: 15.0.0-dev.0
  - updatenotification: 1.23.0
  - user_status: 1.13.0
  - viewer: 6.0.0-dev.0
  - weather_status: 1.13.0
  - workflowengine: 2.15.0
Disabled:
  - app_api: 33.0.0 (installed 33.0.0)
  - bookmarks: 16.0.0 (installed 16.0.0)
  - contactsinteraction: 1.14.1 (installed 1.7.0)
  - encryption: 2.21.0
  - federation: 1.23.0 (installed 1.16.0)
  - files_external: 1.25.1
  - firstrunwizard: 6.0.0-dev.0 (installed 2.15.0)
  - nextcloudpi: 0.0.2 (installed 0.0.1)
  - notes: 4.13.1 (installed 4.13.1)
  - photos: 6.0.0-dev.0 (installed 2.4.0)
  - support: 5.0.0 (installed 1.9.0)
  - survey_client: 5.0.0-dev.0 (installed 1.14.0)
  - suspicious_login: 11.0.0-dev.0
  - twofactor_nextcloud_notification: 7.0.0
  - user_ldap: 1.24.0
  - webhook_listeners: 1.5.0 (installed 1.2.0)
  - whiteboard: 1.5.7 (installed 1.5.7)

Reverse proxies transmit x-forwarded-* headers to relay information about the client request to the backend. Since these headers can be used to spoof the origin of the request, reverse proxies discard them by default. This is likely also the case with Caddy here, you need to configure Caddy to trust headers from notify_push as an “upstream” reverse proxy and accept the headers. (during setup it uses strange IP like 1.2.3.4 to verify the header is passed to NC)

otherwise you can advice notify_push to bypass your Caddy and connect directly to NC (internal hostname) and configure this as trusted proxy.

likely this happens because the notify_push setup doesn’t complete and your old config remains active (which obviously fails because wss://wolke.local/ is not a valid fqdn for public TLS)

Thanks for taking the time to answer :slight_smile:
To be honest, I was a bit frustrated at first, because that’s exactly what I tried yesterday and it just. wouldn’t. work.

But then I just tried it again, as it didnt matter anyway and added

        servers {
                trusted_proxies static private_ranges MY PUBLIC IP
        }

to the caddy config once more.
Yesterday the config file didnt validate, because it didnt recognise the trusted_proxies config value.

But today it just works!
I have literally no clue why it works now, but hey, it does.
Probably I had some errors in my Caddyfile or a reboot fixed it or something like that? Or I updated my Caddy version (it was 2.6 two days ago) and that value was added later?

Anyway, now the output of the setup is:

✓ redis is configured
✓ push server is receiving redis messages
✓ push server can load mount info from database
✓ push server can connect to the Nextcloud server
✓ push server is a trusted proxy
✓ push server is running the same version as the app
  configuration saved

And the browser console states

Has notify_push enabled, slowing polling to 15 minutes NotificationsApp.vue:230:12

and shows no more errors :partying_face: