How to get the real IPs in logs

ℹ️ Support 📦 Appliances (Docker, Snappy, VM, NCP, AIO)
, ,
1

I run Nextcloud in a docker container (:latest, currently 18.0.4). I access it via a reverse proxy.

When looking at the logs, I see one single IP address as the source (the docker network gateway). It seems that the logs display the IP address of the proxy and not the one from the header X-Forwarded-For, which is the real one (and the only one interesting).

Per the documentation, I updated my configuration file with

'trusted_proxies' => array('172.18.0.0/16'),
'forwarded_for_headers' => array('HTTP_X_FORWARDED_FOR'),

but there are no changes (trusted_proxies is the whole range of the docker network because the proxy (itself a container - but with directly exposed ports) can be anywhere)

Just to be complete with the question, the proxy server specifically sets the X-Forwarded-For header:

By default, Caddy passes thru incoming headers to the backend—including the Host header—without modifications, with two exceptions:

1 Like
2
3

Thanks a lot - but unfortunately none of the settings helped.

I removed my config above and added the environment variables to the docker-compose file, no change in the IP.

environment:
      - APACHE_DISABLE_REWRITE_IP=1
      - TRUSTED_PROXIES='172.18.0.0/16'

The logs show an extra line (Conf remoteip disabled) but it seems that this configuration is not applied (?)

 → docker-compose -f /etc/docker/docker-compose.d/nextcloud.yaml down
 → docker-compose -f /etc/docker/docker-compose.d/nextcloud.yaml up -d
Conf remoteip disabled.
To activate the new configuration, you need to run:
  service apache2 reload
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.15. Set the 'ServerName' directive globally to suppress this message
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.15. Set the 'ServerName' directive globally to suppress this message
[Fri May 29 13:58:52.958141 2020] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.38 (Debian) PHP/7.3.18 configured -- resuming normal operations
[Fri May 29 13:58:52.958192 2020] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
172.18.0.1 - wi [29/May/2020:13:58:56 +0000] "MKCOL /remote.php/webdav/Joplin/.sync/ HTTP/1.1" 405 1526 "-" "Joplin/1.0"
172.18.0.1 - wi [29/May/2020:13:58:56 +0000] "MKCOL /remote.php/webdav/Joplin/.lock/ HTTP/1.1" 405 1516 "-" "Joplin/1.0"
(...)
4

Looks good to me. The IP should be correct with nexcloud (and nextcloud.log / brute force detection) now. If you need the real ip also with the apache2 log (without enabled remoteip again because that will break brute force detection and the ip with nextcloud.log) modify the log format used by apache2. I usually enable the access log with the reverse proxy.

5

@kesselb Sorry but I do not understand (I am not sure which configuration you are talking about). To set up some context:

client (192.168.10.3) → reverse proxy (192.168.10.2 / 172.18.0.1) → [ apache → nextcloud ]
                                                                     nextcloud container (172.18.0.5)

The IP I see in the logs above (from docker logs nextcloud) is from the reverse proxy. This is the incoming IP the nextcloud container sees.

What I would like to see is the IP of the client, added to X-Forwarded-For by the reverse proxy. To take the example above, I would like to have in the logs

192.168.10.3 - wi [29/May/2020:13:58:56 +0000] "MKCOL /remote.php/webdav/Joplin/.sync/ HTTP/1.1" 405 1526 "-" "Joplin/1.0"
192.168.10.3 - wi [29/May/2020:13:58:56 +0000] "MKCOL /remote.php/webdav/Joplin/.lock/ HTTP/1.1" 405 1516 "-" "Joplin/1.0"
6

Or enable remoteip again and tell caddy to send x-forwarded-for as x-real-ip. But overwritehost and overwriteprotcol are mandatory then.

7

My constellation is a Nextcloud docker container behind an Apache Server, set up as a Proxy.

I really searched a lot and tried a lot. The only thing that helped, was editing the enabled-site in the Docker Container to use the remote-ip directives, like it is described here:
https://www.loadbalancer.org/blog/apache-and-x-forwarded-for-headers/
It then looks like this:

SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" forwarded
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" forwarded
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined env=!forwarded
CustomLog ${APACHE_LOG_DIR}/access.log forwarded env=forwarded

I changed the configuration of the Nextclod config.php File like this:

   'trusted_proxies' => array('172.17.0.1','ip.from.apache.proxy',),
   'overwritecondaddr' => '^172\.17\.0\.1$',
   'forwarded_for_headers' => array('HTTP_X_FORWARDED',),

The 172.17.0.1 is the gateway from the Docker container.

In Apache Proxy Config, it was sufficient to switch on the “ProxyPreserve host on”

Now I get the real IPs in the docker logs nextcloud.

Hope that helps someone. Don’t know if everything is really needed, but it works for me so far.

Perhaps this would be super to integrate it already in the Docker image … that would save lots of time.

8

Thanks @dora71 for the detailed answer. I will try this out as I almost abandoned the idea to have the right IP (I was getting the proxy one no matter which combinaison of parameters I tried).

I agree that this should be the default - there is no real interest to have the proxy IP (or maybe as a secondary information)

I have a question regarding your setup: where did you configure the Apache part (= where is the Apache configuration file)?

1 Like
9

By default mod remoteip is configured to read x-real-ip from a trusted network. Tell caddy to send the forward to as x-real-ip and you are good.

10

I put it inside the Docker container in /etc/apache2/sites-available/000-default.conf inside the VirtualHost Configuration and commented out the predefined ErrorLog and CustomLog directives.

11

@kesselb - first of all, thanks a lot for following up, I am quite lost in the settings to be frank.

I set this on caddy and these are the headers which are forwarded to each container (including nextcloud). I removed the non-X ones:

X-Forwarded-For: 192.168.10.72
X-Forwarded-Proto: http
X-Real-Ip: 192.168.10.72

So X-Real-Ip is there, it is the IP of the client calling nextcloud.

I also set in the docker-compose file for nextcloud TRUSTED_PROXIES='172.18.0.0/16' (this is the internal docker network, on which all containers are, including the reverse proxy)

I tried to set and unset (again, in the docker-compose file for nextcloud) APACHE_DISABLE_REWRITE_IP=1.

None of this changed the logging, this is still the IP of the proxy that is displayed.

I then recalled that you wrote (emphasis mine)

Or enable remoteip again and tell caddy to send x-forwarded-for as x-real-ip. But overwritehost and overwriteprotcol are mandatory then.

I am not sure what they mean - the documentation says that these “set the protocol and hostname of the proxy”. I do not know what “the hostname of the proxy” is supposed to be.

My proxy (caddy) has a CNAME nextcloud.example.com (and many others, one for each service behind the proxy). I tried to set this as overwritehost (and https as overwriteprotcol because the full URI is https://nextcloud.example.com - but this did not change anything.

All this was done with a clean nextcloud configuration, the only things I changed there are

'trusted_domains' => 
  array (
    0 => 'nextcloud.example.com',
  ),
  'overwrite.cli.url' => 'https://nextcloud.example.com',
  'overwriteprotocol' => 'https',
  'overwritehost' => 'nextcloud.example.com',

(the actual domain name is mine of course)

Maybe the overwritehost refers to the IP the call comes from (the IP of the proxy, as seen within nextcloud network), so I tried

'overwriteprotocol' => 'http',
'overwritehost' => '172.18.0.1',

Same thing: the IP of the proxy is logged

12

@Wpq
Did you ever get this figured out?

I am having a similar issue. I’ve tried damn near everything, but it seems Nextcloud does not want to respect the X-Real-Ip or X-Forwarded-For at all. No matter what I do I get the IP address of my reverse proxy (traefik).

13

@scottdfedorov No, I also tried everything.

I am not very happy with that but I think this is actually a bug (in Nextcloud probably, as the problem exists with several reverse proxies)

14

Finally, I figured the proper configuration to get the client remote ip logged by Nextcloud. It isn’t a bug. Actually, all is mentioned in the official Nextcloud documentation.

So, I added in the sites-available/nextcloud.conf this additional line:

RemoteIPHeader X-Forwarded-For

Not to mention, you have to enable all belonging modules like remoteip and headers. I run Nextcloud on ubuntu server 20.04, installed through snap. Nextcloud is hidden behind a reverse proxy. The reverse proxy runs Apache 2.4

These two sites appeared as quite helpful:

https://docs.nextcloud.com/server/19/admin_manual/configuration_server/reverse_proxy_configuration.html

https://github.com/nextcloud/nextcloud-snap

‘trusted_proxies’ => [’111.222.333.444’],
‘forwarded_for_headers’ => [‘HTTP_X_FORWARDED_FOR’],
‘overwritehost’ => ‘cloud.example.com’,
‘overwriteprotocol’ => ‘https’,
‘overwrite.cli.url’ => ‘https://cloud.example.com’,
‘trusted_domains’ =>
array (
0 => ‘localhost’,
1 => ‘cloud.example.com’,
),

111.222.333.444 is the ip address of my Apache reverse proxy.

Also, in the settings of Nextcloud the result of the internal security check now is all well. Before, I got the error message that either the reverse proxy header is wrongly configured, or I run Nextcloud behind a trusted reverse proxy. As mentioned, now I only get a green okay confirmation.

15

Thanks for the detailed answer but unfortunately this did not help - I still get the wrong IP.

16

Hey @Wpq ,

There could be an error in your webserver setup. If your Nextcloud configuration remains unchanged with an exception that is what I have added to my previous post, it sounds pretty much like that.

If we assume a setup for cloud.example.com like

browser (a) -> reverse proxy (b) -> Nextcloud with it’s own webserver running https (c)

then the configuration is something like (be aware, you absolutely need to add a few more things regarding security concerns, it just is an excerpt of my configuration):

For http:

# (b)
# http
# 000.conf
…
ProxyRequests Off
ProxyPreserveHost On
RewriteEngine on
RewriteCond %{HTTP_HOST} ^([a-z.]*)?example.com$ [NC]
RewriteRule .? https://%1example.com/%{REQUEST_URI} [R=301,L]
…

And for https:

# (b)
# https
# cloud.conf
<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
                ServerAdmin admin@example.com
                ServerName cloud.example.com

                ProxyRequests Off
                ProxyPreserveHost On

                RemoteIPHeader X-Forwarded-For

                <Proxy *>
                        Require all granted
                </Proxy>

                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined

                SSLEngine on
                SSLProxyEngine on
                SSLProxyVerify none
                SSLProxyCheckPeerCN off
                SSLProxyCheckPeerName off
                SSLProxyCheckPeerExpire off

                SSLCertificateFile …
                SSLCertificateKeyFile …

                ProxyPass / https://111.222.333.444/
                ProxyPassReverse / https://111.222.333.444/

        </VirtualHost>
</IfModule>

You see RemoteIPHeader X-Forwarded-For in the file cloud.conf This setup works pretty well for me, and the Nextcloud logfile includes the remote ip.

17

I have done those header forwards with different products, e.g., Atlassian Confluence. Over more than one reverse proxies. Without any issues.

Nextcloud works pretty well, too. Without any flaws. Even the client is very reliable. Our desktops run ubuntu Linux, our phones Android, or something similar. I don’t know much about Windows or Mac.

If you don’t see any remote ip in your log, I guess you should check the configuration files of all your web servers. Since we talk about web server configuration here, I like to ask you for contacting the community of your web server. Your issue probably has nothing to do with Nextcloud.

If your docker file includes a web server, check the configuration there. You probably also need to add X-Forwarded-For to this one.

18

Except if I am terribly missing something, I still think that the issue is with Nextcloud (probably my configuration).

The reverse proxy is caddy. The exact same configuration of a whoami container yields correct results, with the correct X-Forwarded-For header. Same for a standard application (etherpad, when started with a “trust the proxy” setting).

I am sure Nextcloud itself works fine, there must be something with my configuration but to be frank I have tested everything I found on Internet (including cases of people who were in my situation and gave up because they were only seeing their proxy IP) and I start to loose hope :slight_smile:

19

I would say it is daily business. If an issue comes into your way, you have two options: giving up, or finding a way to get around it.

Don’t know anything about caddy. “X-Forwarded-For” isn’t standardized yet, that means you could find different implementations in different products. Nextcloud snap includes Apache2, and my reverse proxy also runs Apache2. These two web servers work perfectly together.

You go another path, and that is all fine. I have never seen caddy running in professional setups. Nonetheless, you made a decision, and you need to figure out what is going wrong. I would set up a virtualbox with Nextcloud running http, another with Apache2, and a third with caddy. Then, with a gateway logging all traffic to a file, you will be able to see the differences between these two reverse proxies.

Share your insights here so that everybody with caddy gets the piece of information they are looking for.

20

My point is that this is not a problem with caddy, because at least two reasons:

  1. I have 30 containers, many of them log the incoming IP. The IP they log is the client one, retrieved though X-Forwarded-For (which is the de-factor standard: X-Forwarded-For - HTTP | MDN).
    The whoami container also shows the right header being passed.

  2. When you take a network trace in the caddy container, here is what you see when there is a call to nextcloud (172.19.0.23):

    172.19.0.31.56186 > 172.19.0.23.80: Flags [P.], cksum 0x5bd7 (incorrect -> 0xb673), seq 1356374864:1356375716, ack 1753153869, win 618, options [nop,nop,TS val 3060191419 ecr 2639773560], length 852: HTTP, length: 852
        PROPFIND /remote.php/dav/files/michael/ HTTP/1.1
        Host: nextcloud.MYDOMAIN
        User-Agent: Mozilla/5.0 (Windows) mirall/2.6.4stable-Win64 (build 20200303) (Nextcloud)
        Content-Length: 105
        Accept: */*
        Accept-Encoding: gzip, deflate
        Accept-Language: fr-FR,en,*
        Authorization: Basic bWlja...0Q=
        Content-Type: text/xml; charset=utf-8
        Cookie: oc_sessionPassphrase=bbXZJ...wjk; nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; ocflnf1ody3y=7ef75e330054749d600f595009f72a33
        Depth: 0
        X-Forwarded-For: 192.168.10.251
        X-Forwarded-Proto: https
        X-Real-Ip: 192.168.10.251
        X-Request-Id: 5b27c480-e5e6-4b79-a208-4ff6ef237d58

So there is no miracle: the packet that leaves caddy holds the right header.

It then arrives to Nexcloud. There is something which exposes port 80 - Apache I guess. I believe that this is also Apache that logs the call:

172.19.0.31 - michael [28/Aug/2020:11:19:00 +0000] "PROPFIND /remote.php/dav/files/michael/ HTTP/1.1" 207 1103 "-" "Mozilla/5.0 (Windows) mirall/2.6.4stable-Win64 (build 20200303) (Nextcloud)"

At some point Apache must make the decision on which IP to log, this is (again, I believe) decided in the configuration when I write

  'trusted_proxies' => 
  array (
    0 => '172.19.0.31',
  ),
  'proxy' => '172.19.0.31',
  'forwarded_for_headers' => 
  array (
    0 => 'HTTP_X_FORWARDED_FOR',
  ),

According to Reverse proxy — Nextcloud 15 Administration Manual 15 documentation

A reverse proxy can define HTTP headers with the original client IP address, and Nextcloud can use those headers to retrieve that IP address. Nextcloud uses the de-facto standard header ‘X-Forwarded-For’ by default, but this can be configured with the forwarded_for_headers parameter. This parameter is an array of PHP lookup strings, for example ‘X-Forwarded-For’ becomes ‘HTTP_X_FORWARDED_FOR’. Incorrectly setting this parameter may allow clients to spoof their IP address as visible to Nextcloud, even when going through the trusted proxy! The correct value for this parameter is dependent on your proxy software.

I work in IT, I know that Nextcloud is a complex piece of software - but at some point there are objective tests which show that data flows in and that with a specific configuration - the result is not the expected one.

It is probable that the configuration of my docker instance does not have the magical environment, or that it is incompatible with the configuration but taken into account the number of people on Internet who try to get the right IP and fail (or manage though an undocumented combination of configuration items) is at least surprising.

1 Like