X-Forward-For HTTP header entry ignored

Nextcloud version (eg, 29.0.5): 28.0.5
Operating system and version (eg, Ubuntu 24.04): Ubuntu 24.04
Apache or nginx version (eg, Apache 2.4.25): I am using the nextcloud:28.0.5-apache Docker image
PHP version (eg, 8.3): Whatever is installed in nextcloud:28.0.5-apache

The issue you are facing:

This is yet another “X-Forwarded-For” stuff being ignored and I have worked through many issues in the past, still have not found a solution in my Docker Swarm installation of NextCloud. Other services work fine.

X-Forward-For HTTP header is not respected → Nextcloud only sees the IP of the reverse proxy, which sometimes changes, depending where it is spawned in the Swarm. Every single log shows the IP of the load balancer / reverse proxy.
The HTTP header contains the x-forward-for entry with the correct IP address and other services in the Docker Swarm are also using that and work correctly. NextCloud ignores it.

Here, the IP of the load balancer (inside the Docker Swarm, an HAProxy service) is 10.0.9.228

cloud_nextcloud  | 10.0.9.228 - - [12/Jul/2024:08:47:43 +0000] "HEAD / HTTP/1.0" 302 1632 "-" "-"
cloud_nextcloud  | 10.0.9.228 - - [12/Jul/2024:08:47:44 +0000] "GET / HTTP/1.1" 302 1658 "-" "Telegraf/1.25.2 Go/1.20"

Is this the first time you’ve seen this error? (Y/N): N

Steps to replicate it:

  1. Deploy nextcloud:28.0.5-apache behind HAProxy

The output of your Nextcloud log in Admin > Logging: nothing relevant.

The output of your config.php file in /path/to/nextcloud (make sure you remove any identifiable information!):

<?php
$CONFIG = array (
  'htaccess.RewriteBase' => '/',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'loglevel' => 2,
  'apps_paths' =>
  array (
    0 =>
    array (
      'path' => '/var/www/html/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 =>
    array (
      'path' => '/var/www/html/custom_apps',
      'url' => '/custom_apps',
      'writable' => true,
    ),
  ),
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'redis',
    'password' => '',
    'port' => 6379,
  ),
  'passwordsalt' => '...',
  'secret' => '...',
  'trusted_domains' =>
  array (
    0 => 'localhost',
    1 => 'cloud.institute.org',
    2 => '10.0.*.*',
    3 => '10.0.9.228',
    4 => '10.0.0.0/16'
  ),
  'trusted_proxies' =>
  array (
    0 => '10.0.*.*',
    1 => '10.0.9.228',
    2 => '10.0.0.0/16'
  ),
  'forwarded_for_headers' =>
  array (
    0 => 'HTTP_X_FORWARDED_FOR',
  ),
  'datadirectory' => '/var/www/html/data',
  'dbtype' => 'mysql',
  'version' => '28.0.5.1',
  'overwrite.cli.url' => 'http://localhost',
  'overwriteprotocol' => 'https',
  'dbname' => '...',
  'dbhost' => 'mariadb',
  'dbport' => '',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => '...',
  'dbpassword' => '...',
  'installed' => true,
  'instanceid' => '...',
  'ldapProviderFactory' => 'OCA\\User_LDAP\\LDAPProviderFactory',
  'mail_smtpmode' => 'smtp',
  'mail_smtpauthtype' => 'PLAIN',
  'mail_sendmailmode' => 'smtp',
  'mail_smtpauth' => 1,
  'mail_smtphost' => '...',
  'mail_smtpsecure' => 'ssl',
  'mail_smtpport' => '465',
  'mail_from_address' => '...',
  'mail_smtpname' => '...',
  'mail_smtppassword' => '...',
  'mail_domain' => 'fau.de',
  'maintenance' => false,
);

The output of your Apache/nginx/system log in /var/log/____:

cloud_nextcloud  | 10.0.9.228 - - [12/Jul/2024:08:47:43 +0000] "HEAD / HTTP/1.0" 302 1632 "-" "-"
cloud_nextcloud  | 10.0.9.228 - - [12/Jul/2024:08:47:44 +0000] "GET / HTTP/1.1" 302 1658 "-" "Telegraf/1.25.2 Go/1.20"

Output errors in nextcloud.log in /var/www/ or as admin user in top right menu, filtering for errors. Use a pastebin service if necessary.

No errors

Are you sure that your proxy is sending the HTTP_X_FORWARDED_FOR? You can test this, for example, by placing a PHP script within the Nextcloud installation that dumps the headers.

1 Like

The log entry you posted is the Apache log. What does the Nextcloud log actually show?

In any case, you’re using the apache Docker image so you need to understand how it functions by default, which is using X-Real-IP. You didn’t post your Compose file, which has the options to control the image’s behavior in this regard, so I’m guessing you may not have seen the docs on this which can be found here.

Also:

  • Please post the output of occ config:list system rather than your config.php since the Docker image merges multiple config files (what you posted isn’t necessarily your live config)
  • What does Administration settings->Overview show?
2 Likes

Thanks for the quick replies!

@mritzmann I hacked the .htaccess to allow a test file to show phpinfo() in the running instance and I can confirm that the HTTP_X_FORWARDED_FOR is set correctly. Below you can see the internal Docker Swarm IP of the load balancer (reverse proxy), the internal address of the NextCloud instance and the external IP of my router at home:

In any case, you’re using the apache Docker image so you need to understand how it functions by default, which is using X-Real-IP . You didn’t post your Compose file, which has the options to control the image’s behavior in this regard, so I’m guessing you may not have seen the docs on this which can be found here.

OK I see, I am pretty sure I missed something :wink:

I am using the nextcloud:28.0.5-apache and only set the following environment variables:

      - MYSQL_HOST=mariadb
      - MYSQL_DATABASE=...
      - MYSQL_USER=...
      - MYSQL_PASSWORD=...
      - NEXTCLOUD_ADMIN_USER=...
      - NEXTCLOUD_ADMIN_PASSWORD=...
      - NEXTCLOUD_TRUSTED_DOMAINS=cloud.institute.org
      - PHP_UPLOAD_LIMIT=15000M
      - REDIS_HOST=redis

Here is the output of occ config:list system:

{
    "system": {
        "htaccess.RewriteBase": "\/",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "loglevel": 2,
        "apps_paths": [
            {
                "path": "\/var\/www\/html\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/html\/custom_apps",
                "url": "\/custom_apps",
                "writable": true
            }
        ],
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "password": "***REMOVED SENSITIVE VALUE***",
            "port": 6379
        },
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "cloud.institute.org",
            "10.0.*.*",
            "10.0.9.228"
        ],
        "trusted_proxies": "***REMOVED SENSITIVE VALUE***",
        "forwarded_for_headers": [
            "HTTP_X_FORWARDED_FOR"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "28.0.5.1",
        "overwrite.cli.url": "http:\/\/localhost",
        "overwriteprotocol": "https",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "ldapProviderFactory": "OCA\\User_LDAP\\LDAPProviderFactory",
        "mail_smtpmode": "smtp",
        "mail_smtpauthtype": "PLAIN",
        "mail_sendmailmode": "smtp",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "ssl",
        "mail_smtpport": "465",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "maintenance": false
    }
}

and finally the Adminsitration settings -> Overview. I must admin, I overlooked the error about the trusted proxies but I guess that’s new, because I again played around with those values. The error is referring to the current setting, so I assume it does not like 10.0.0.0/16?

There are some errors regarding your setup.

Your “trusted_proxies” setting is not correctly set, it should be an array of IP addresses - optionally with range in CIDR notation. For more details see the documentation :arrow_upper_right:.

The DAV system address book sync has not run yet as your instance has more than 1000 users or because an error occurred. Please run it manually by calling “occ dav:sync-system-addressbook”.

310 errors in the logs since 5 July 2024

Server has no maintenance window start time configured. This means resource intensive daily background jobs will also be executed during your main usage time. We recommend to set it to a time of low usage, so users are less impacted by the load caused from these heavy tasks. For more details see the documentation :arrow_upper_right:.

Some columns in the database are missing a conversion to big int. Due to the fact that changing column types on big tables could take some time they were not changed automatically. By running “occ db:convert-filecache-bigint” those pending changes could be applied manually. This operation needs to be made while the instance is offline. file_metadata.id For more details see the documentation :arrow_upper_right:.

The '10.0.*.*' was wrong in the config, now I removed it and the Your "trusted_proxies" setting is not correctly set, it should be an array of IP addresses is gone.

This is what I have in the config.php right now:

  'trusted_proxies' =>
  array (
    0 => '10.0.9.210',
    1 => '10.0.9.228',
    2 => '10.0.0.0/16'
  ),

I’ll now check if the IPs are correctly propagated to NextCloud.

1 Like

I confirm that it works now :slight_smile: Indeed, the problem was the IP range notation…

2 Likes

This topic was automatically closed 8 days after the last reply. New replies are no longer allowed.