How to get the real IPs in logs

ℹ️ Support 📦 Appliances (Docker, Snappy, VM, NCP, AIO)
, ,
11

@kesselb - first of all, thanks a lot for following up, I am quite lost in the settings to be frank.

I set this on caddy and these are the headers which are forwarded to each container (including nextcloud). I removed the non-X ones:

X-Forwarded-For: 192.168.10.72
X-Forwarded-Proto: http
X-Real-Ip: 192.168.10.72

So X-Real-Ip is there, it is the IP of the client calling nextcloud.

I also set in the docker-compose file for nextcloud TRUSTED_PROXIES='172.18.0.0/16' (this is the internal docker network, on which all containers are, including the reverse proxy)

I tried to set and unset (again, in the docker-compose file for nextcloud) APACHE_DISABLE_REWRITE_IP=1.

None of this changed the logging, this is still the IP of the proxy that is displayed.

I then recalled that you wrote (emphasis mine)

Or enable remoteip again and tell caddy to send x-forwarded-for as x-real-ip. But overwritehost and overwriteprotcol are mandatory then.

I am not sure what they mean - the documentation says that these “set the protocol and hostname of the proxy”. I do not know what “the hostname of the proxy” is supposed to be.

My proxy (caddy) has a CNAME nextcloud.example.com (and many others, one for each service behind the proxy). I tried to set this as overwritehost (and https as overwriteprotcol because the full URI is https://nextcloud.example.com - but this did not change anything.

All this was done with a clean nextcloud configuration, the only things I changed there are

'trusted_domains' => 
  array (
    0 => 'nextcloud.example.com',
  ),
  'overwrite.cli.url' => 'https://nextcloud.example.com',
  'overwriteprotocol' => 'https',
  'overwritehost' => 'nextcloud.example.com',

(the actual domain name is mine of course)

Maybe the overwritehost refers to the IP the call comes from (the IP of the proxy, as seen within nextcloud network), so I tried

'overwriteprotocol' => 'http',
'overwritehost' => '172.18.0.1',

Same thing: the IP of the proxy is logged

12

@Wpq
Did you ever get this figured out?

I am having a similar issue. I’ve tried damn near everything, but it seems Nextcloud does not want to respect the X-Real-Ip or X-Forwarded-For at all. No matter what I do I get the IP address of my reverse proxy (traefik).

13

@scottdfedorov No, I also tried everything.

I am not very happy with that but I think this is actually a bug (in Nextcloud probably, as the problem exists with several reverse proxies)

14

Finally, I figured the proper configuration to get the client remote ip logged by Nextcloud. It isn’t a bug. Actually, all is mentioned in the official Nextcloud documentation.

So, I added in the sites-available/nextcloud.conf this additional line:

RemoteIPHeader X-Forwarded-For

Not to mention, you have to enable all belonging modules like remoteip and headers. I run Nextcloud on ubuntu server 20.04, installed through snap. Nextcloud is hidden behind a reverse proxy. The reverse proxy runs Apache 2.4

These two sites appeared as quite helpful:

https://docs.nextcloud.com/server/19/admin_manual/configuration_server/reverse_proxy_configuration.html

https://github.com/nextcloud/nextcloud-snap

‘trusted_proxies’ => [’111.222.333.444’],
‘forwarded_for_headers’ => [‘HTTP_X_FORWARDED_FOR’],
‘overwritehost’ => ‘cloud.example.com’,
‘overwriteprotocol’ => ‘https’,
‘overwrite.cli.url’ => ‘https://cloud.example.com’,
‘trusted_domains’ =>
array (
0 => ‘localhost’,
1 => ‘cloud.example.com’,
),

111.222.333.444 is the ip address of my Apache reverse proxy.

Also, in the settings of Nextcloud the result of the internal security check now is all well. Before, I got the error message that either the reverse proxy header is wrongly configured, or I run Nextcloud behind a trusted reverse proxy. As mentioned, now I only get a green okay confirmation.

15

Thanks for the detailed answer but unfortunately this did not help - I still get the wrong IP.

16

Hey @Wpq ,

There could be an error in your webserver setup. If your Nextcloud configuration remains unchanged with an exception that is what I have added to my previous post, it sounds pretty much like that.

If we assume a setup for cloud.example.com like

browser (a) -> reverse proxy (b) -> Nextcloud with it’s own webserver running https (c)

then the configuration is something like (be aware, you absolutely need to add a few more things regarding security concerns, it just is an excerpt of my configuration):

For http:

# (b)
# http
# 000.conf
…
ProxyRequests Off
ProxyPreserveHost On
RewriteEngine on
RewriteCond %{HTTP_HOST} ^([a-z.]*)?example.com$ [NC]
RewriteRule .? https://%1example.com/%{REQUEST_URI} [R=301,L]
…

And for https:

# (b)
# https
# cloud.conf
<IfModule mod_ssl.c>
        <VirtualHost _default_:443>
                ServerAdmin admin@example.com
                ServerName cloud.example.com

                ProxyRequests Off
                ProxyPreserveHost On

                RemoteIPHeader X-Forwarded-For

                <Proxy *>
                        Require all granted
                </Proxy>

                ErrorLog ${APACHE_LOG_DIR}/error.log
                CustomLog ${APACHE_LOG_DIR}/access.log combined

                SSLEngine on
                SSLProxyEngine on
                SSLProxyVerify none
                SSLProxyCheckPeerCN off
                SSLProxyCheckPeerName off
                SSLProxyCheckPeerExpire off

                SSLCertificateFile …
                SSLCertificateKeyFile …

                ProxyPass / https://111.222.333.444/
                ProxyPassReverse / https://111.222.333.444/

        </VirtualHost>
</IfModule>

You see RemoteIPHeader X-Forwarded-For in the file cloud.conf This setup works pretty well for me, and the Nextcloud logfile includes the remote ip.

17

I have done those header forwards with different products, e.g., Atlassian Confluence. Over more than one reverse proxies. Without any issues.

Nextcloud works pretty well, too. Without any flaws. Even the client is very reliable. Our desktops run ubuntu Linux, our phones Android, or something similar. I don’t know much about Windows or Mac.

If you don’t see any remote ip in your log, I guess you should check the configuration files of all your web servers. Since we talk about web server configuration here, I like to ask you for contacting the community of your web server. Your issue probably has nothing to do with Nextcloud.

If your docker file includes a web server, check the configuration there. You probably also need to add X-Forwarded-For to this one.

18

Except if I am terribly missing something, I still think that the issue is with Nextcloud (probably my configuration).

The reverse proxy is caddy. The exact same configuration of a whoami container yields correct results, with the correct X-Forwarded-For header. Same for a standard application (etherpad, when started with a “trust the proxy” setting).

I am sure Nextcloud itself works fine, there must be something with my configuration but to be frank I have tested everything I found on Internet (including cases of people who were in my situation and gave up because they were only seeing their proxy IP) and I start to loose hope :slight_smile:

19

I would say it is daily business. If an issue comes into your way, you have two options: giving up, or finding a way to get around it.

Don’t know anything about caddy. “X-Forwarded-For” isn’t standardized yet, that means you could find different implementations in different products. Nextcloud snap includes Apache2, and my reverse proxy also runs Apache2. These two web servers work perfectly together.

You go another path, and that is all fine. I have never seen caddy running in professional setups. Nonetheless, you made a decision, and you need to figure out what is going wrong. I would set up a virtualbox with Nextcloud running http, another with Apache2, and a third with caddy. Then, with a gateway logging all traffic to a file, you will be able to see the differences between these two reverse proxies.

Share your insights here so that everybody with caddy gets the piece of information they are looking for.

20

My point is that this is not a problem with caddy, because at least two reasons:

  1. I have 30 containers, many of them log the incoming IP. The IP they log is the client one, retrieved though X-Forwarded-For (which is the de-factor standard: X-Forwarded-For - HTTP | MDN).
    The whoami container also shows the right header being passed.

  2. When you take a network trace in the caddy container, here is what you see when there is a call to nextcloud (172.19.0.23):

    172.19.0.31.56186 > 172.19.0.23.80: Flags [P.], cksum 0x5bd7 (incorrect -> 0xb673), seq 1356374864:1356375716, ack 1753153869, win 618, options [nop,nop,TS val 3060191419 ecr 2639773560], length 852: HTTP, length: 852
        PROPFIND /remote.php/dav/files/michael/ HTTP/1.1
        Host: nextcloud.MYDOMAIN
        User-Agent: Mozilla/5.0 (Windows) mirall/2.6.4stable-Win64 (build 20200303) (Nextcloud)
        Content-Length: 105
        Accept: */*
        Accept-Encoding: gzip, deflate
        Accept-Language: fr-FR,en,*
        Authorization: Basic bWlja...0Q=
        Content-Type: text/xml; charset=utf-8
        Cookie: oc_sessionPassphrase=bbXZJ...wjk; nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; ocflnf1ody3y=7ef75e330054749d600f595009f72a33
        Depth: 0
        X-Forwarded-For: 192.168.10.251
        X-Forwarded-Proto: https
        X-Real-Ip: 192.168.10.251
        X-Request-Id: 5b27c480-e5e6-4b79-a208-4ff6ef237d58

So there is no miracle: the packet that leaves caddy holds the right header.

It then arrives to Nexcloud. There is something which exposes port 80 - Apache I guess. I believe that this is also Apache that logs the call:

172.19.0.31 - michael [28/Aug/2020:11:19:00 +0000] "PROPFIND /remote.php/dav/files/michael/ HTTP/1.1" 207 1103 "-" "Mozilla/5.0 (Windows) mirall/2.6.4stable-Win64 (build 20200303) (Nextcloud)"

At some point Apache must make the decision on which IP to log, this is (again, I believe) decided in the configuration when I write

  'trusted_proxies' => 
  array (
    0 => '172.19.0.31',
  ),
  'proxy' => '172.19.0.31',
  'forwarded_for_headers' => 
  array (
    0 => 'HTTP_X_FORWARDED_FOR',
  ),

According to Reverse proxy — Nextcloud 15 Administration Manual 15 documentation

A reverse proxy can define HTTP headers with the original client IP address, and Nextcloud can use those headers to retrieve that IP address. Nextcloud uses the de-facto standard header ‘X-Forwarded-For’ by default, but this can be configured with the forwarded_for_headers parameter. This parameter is an array of PHP lookup strings, for example ‘X-Forwarded-For’ becomes ‘HTTP_X_FORWARDED_FOR’. Incorrectly setting this parameter may allow clients to spoof their IP address as visible to Nextcloud, even when going through the trusted proxy! The correct value for this parameter is dependent on your proxy software.

I work in IT, I know that Nextcloud is a complex piece of software - but at some point there are objective tests which show that data flows in and that with a specific configuration - the result is not the expected one.

It is probable that the configuration of my docker instance does not have the magical environment, or that it is incompatible with the configuration but taken into account the number of people on Internet who try to get the right IP and fail (or manage though an undocumented combination of configuration items) is at least surprising.

1 Like
21

@Wpq were you able to figure this out? I just followed a similar path. I’m using docker-compose, my reverse proxy is Traefik. I’m using the nextcloud:apache Docker image. I’m seeing correct and expected IP addresses in log lines in nextcloud.log (remoteAddr represents actual client IP addresses), but the Apache log (viewed via docker-compose logs) still shows the proxy IP address. I thought the issue was in the apache config, my proxy’s IP address is not in any of the ranges listed for RemoteIPTrustedProxy. However, I tried customizing /etc/apache2/conf-enabled/remoteip.conf with my proxy IP but the Apache logs still show the proxy IP address, not the real client IP address. It’s not a dealbreaker for me (I can, for example, train fail2ban to watch the nextcloud log) but it is annoying.

22

@meonkeys No, I did not. There is a fundamental problem with logging in Netxcloud, simply due to the fact that half of the people claim “it works” and the other half “it does not work”.

I tried every combination of parameters and did not manage to get the right IP. At the same time in the various 20+ services I run I do not have this problem. I work in hardcore IT for more years I would like to believe I do, so if that can be an indication that something is wrong there, it is.

23

Do you have ProxyPreserveHost set on the proxy?

24

Ugh, bummer. Just to be clear, which log files are you examining? Apache or Nextcloud? Or both?

1 Like
25

The reverse proxy (Traefik) is properly passing on X-Real-Ip and Nextcloud is using it: client IPs show up as expected in the Nextcloud log. I’m pretty sure there’s just something odd going on with the in-container Apache config.

1 Like
26

I had a similar issue that I couldn’t nail down for a bit. In the end I found a vague reference in documentation elsewhere that recommends modifying the apache2.conf LogFormat entries, replacing %h with %a like my example below.

LogFormat "%v:%p %a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%a %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%a %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

Doing this and restarting the apache2 service seemed to fix the issue, although being a Docker container this won’t fix the issue for me longer term. Interested to know if this helps solve the issue for anyone else though!

1 Like
27

I was just trying to solve this problem for a non-docker install on a server behind an nginx reverse proxy, and came across this thread. I played around with the suggestions here and this is what worked for me:

NGINX on gateway/proxy:

server {
listen 443;
listen [::]:443;
server_name cloud.blah.com;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
send_timeout 600;
location / {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_pass https://192.168.1.6:443;
  }
ssl_certificate /etc/letsencrypt/live/cloud.blah.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/cloud.blah.com/privkey.pem; # managed by Certbot
}

I basically had to fully fit out the nextcloud’s nginx config, with its own SSL certificate, in order to get it working properly. There were endless http <-> https problems until I did this.
NGINX on nextcloud server:

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name cloud.blah.com 

ssl_certificate /etc/letsencrypt/live/cloud.blah.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/cloud.blah.com/privkey.pem; # managed by Certbot

#... rest of the full config as if this was in a DMZ

config/config.php:

...
'trusted_domains' =>
array (
  0 => 'cloud.blah.com',
  1 => '192.168.1.6',
),
'trusted_proxies' =>
array (
  0 => '192.168.1.8',  /* the proxy IP */
),
'forwarded_for_headers' => array('HTTP_X_REAL_IP'),
'overwrite.cli.url' => 'https://cloud.blah.com',
'overwriteprotocol' => 'https',
'overwritecondaddr' => '^192.168.1.8$',
...

After I restarted php/nginx on the nextcloud server, it just worked. The nextcloud.log started showing the correct IP whether it was from a local networked machine, or an external one.

29

@dmartin: you withdrew the post because there was problem with what you provided as a solution?

I am asking because I use Caddy as the reverse proxy to nextcloud living in its own container as well. So your information was wonderful news :slight_smile:

One thing though, you mentioned

(…) in the caddy inside the container (…)

Does this mean that you have to rebuild the container every time there is a new version (I update nextcloud though watchtower)

30

I just tried this and it actually works. I wonder what you have to change in the settings to make it work permanently though.

31

So, for other who search for a solution, here is how I solved it:

I’m using reverse proxy with HAproxy, with ssl offloading, running nextcloud in docker
Also using nat reflection so that I can use my external host on my internal network

First added my haproxy to config.php trusted_proxies (nextcloud_config dir)

By enabling X-Forwarded-For header in HA proxy, I did get my client IP’s in nextcloud.log, also verified headers with tcpdump (also easily testet with something like containous/whoami),
apache logs didn’t care about that, first of all, I had to change %h to %a in LogFormat (in apache2.conf), which is the variable that mod_remoteip uses, then I had to add the X-Real-IP header to HAproxy as it doesn’t seem to listen to X-Forwarded-For in this configuration, now I can see the client IP’s of external clients
Clients originates from my local network still just shows the proxy as ip

After some investigation, I did some changes in /etc/apache2/conf-enabled/remoteip.conf
Commented out all RemoteIPTrustedProxy lines and added:
RemoteIPInternalProxy

By doing so, IP’s of clients originating from my local network also hits the apache log

Further I noticed that mod_remoteip only had “RemoteIPHeader X-Real-IP” configured, by changing that to:
RemoteIPHeader X-Forwarded-For
in remoteip.conf, it was unnecessary to add the X-Real-IP header in HAproxy config

I would like to see a persistent solution to this

This is the changes I did from within the container:

sed -i -e 's/RemoteIPHeader X-Real-IP/RemoteIPHeader X-Forwarded-For/g' /etc/apache2/conf-enabled/remoteip.conf
sed -i -e 's/RemoteIPTrustedProxy/#RemoteIPTrustedProxy/g' /etc/apache2/conf-enabled/remoteip.conf
echo "RemoteIPInternalProxy $(env | grep TRUSTED_PROXIES | cut -d= -f2)" >> /etc/apache2/conf-enabled/remoteip.conf
sed -i -e '/LogFormat/s/%h/%a/g' /etc/apache2/apache2.conf

note that TRUSTED_PROXIES env variable has to be set to the reverse proxy ip

Feel free to correct me if I made some mistakes

1 Like