Apache Docker behind reverse proxy

wwe · December 12, 2022, 10:10pm

understanding and configuring reverse proxy for official Nextcloud Apache Docker image

I was experimenting with new option to run official Nextcloud Docker image with custom user (long requested and really appreciated feature improving security of the system) which finally find into official image shortly Allow to run with custom uid by J0WI · Pull Request #1812 · nextcloud/docker · GitHub when I hit very strange issue: Nextcloud logs and Apache logs didn’t show client IP but internal IP of reverse proxy.

Basic overview is like this:

The problem itself is common and discussed very often - a webserver behind a reverse proxy has no direct connection with a client so additional reverse proxy config is required to show client IP within application. This is achieved by adding so called http headers X-Real-Ip and/or X-Forwarded-For which are only used by Nextcloud if this header comes from known TRUSTED_PROXY trusted_proxies. The config in general is not rocket since and was working in my setup with traefik reverseproxy and Nextcloud Apache image for ages…

Apache Docker image is expected to use X-Forwarded-For header by default in Docker (given the fact docker networks are by default in 172.16.0.0/12 range)… If this doesn’t work there are two more variables to add trusted_proxies and APACHE_DISABLE_REWRITE_IP=1 which worked good with container running root mode.

Now it turned APACHE_DISABLE_REWRITE_IP=1 doesn’t work in combination with docker --user parameter Issue 1494 and Issue 772 - luckily good workaround was suggested (which later becomes key to success!!):

  nextcloud:
    ...
    environment:
      ...
      # Use the `remoteip.conf` workaround below instead
      #APACHE_DISABLE_REWRITE_IP: 1
    ...
    volumes:
    - ./remoteip.conf:/etc/apache2/conf-enabled/remoteip.conf:ro
    - ./redis-session.ini:/usr/local/etc/php/conf.d/redis-session.ini

this two volumes are required as Nextcloud image can’t modify this files when running as non-root and fails to start/work…

Further analyzing the problem I found client IP was shown when the client was external and reverse proxy IP was shown for internal clients… The mystery was complete - why doesn’t it work for internal clients? My first assumption was IPv6 - I made some effort to enable IPv6 internally some time ago so this was my first try - remove IPv6 of the server from internal dns and double check… same issue reverse proxy IP was shown for internal clients but worked well from internet…

Troubleshooting Apache logs

Assuming some fail within docker or maybe traefik proxy I changed the logging directive for Apache2 server in the container (mount /etc/apache2/apache2.conf config file as shown above) - added [%{X-Forwarded-For}i] [%{X-Real-Ip}i] at the end of the log line:

LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" [%{X-Forwarded-For}i] [%{X-Real-Ip}i]" combined

this allows me to see incoming the two relevant headers X-Forwarded-For and X-Real-Ip along with other Apache logging (send to STDOUT in Nextcloud Docker by default - use “docker logs %container name%” to view this logs (add -n 10 to see only ten last records) which show right headers arrive within the container but in case of external access the client IP is rewritten and for internal client it doesn’t…

And now the final thought - the problem must be related to the Apache mod_remoteip module… Reading through the docs show the config used by Nextcloud docker is not ideal!

this is how it looks by default:

cat remoteip.conf
RemoteIPHeader X-Real-Ip
RemoteIPTrustedProxy 10.0.0.0/8
RemoteIPTrustedProxy 172.16.0.0/12
RemoteIPTrustedProxy 192.168.0.0/16

the idea is to accept proxy headers from system within IPv4 private address range (RFC 1918)… The problem is RemoteIPTrustedProxy does not accept IPs from private networks!!

Solution

The solution is to use RemoteIPInternalProxy directive:

The RemoteIPInternalProxy directive adds one or more addresses (or address blocks) to trust as presenting a valid RemoteIPHeader value of the useragent IP. Unlike the RemoteIPTrustedProxy directive, any IP address presented in this header, including private intranet addresses, are trusted when passed from these proxies.

changing the file (thankfully we have it local on the docker host): to

cat remoteip.conf
# RFC1918
RemoteIPInternalProxy 10.0.0.0/8
RemoteIPInternalProxy 172.16.0.0/12
RemoteIPInternalProxy 192.168.0.0/16
# unique local addresses 
# fc00::/7 is divided into 2 different /8 networks  fc00::/8 and fd00::/8
RemoteIPInternalProxy fc00::/7
# Link-local addresses have a prefix of FE80::/10
RemoteIPInternalProxy fe80::/10
# reserved for documentation, widely used in tutorials.
RemoteIPInternalProxy 2001:db8::/32

makes real client IP work from local LAN and from internet (for IPv4 and common internal IPv6 if active in your installation).

log screenshots

docker logs:

audit log:

References:

kesselb · December 30, 2022, 9:49pm

Good guide, thank you!

I wonder if we could/should remove APACHE_DISABLE_REWRITE_IP.
Mounting remoteip.conf from host seems mucher nicer.

meonkeys · January 17, 2023, 7:55am

Oh man, finally, real client IPs in the logs! Thank you so much for sharing.

Also, I wasn’t aware of Allow to run with custom uid by J0WI · Pull Request #1812 · nextcloud/docker · GitHub - that is wonderful!

wwe · March 21, 2023, 8:06pm

2 posts were split to a new topic: Reverse proxy config with podman and Caddy

John2 · April 8, 2023, 10:09pm

Will mounting those two volumes (files) on an existing system cause issues? Do I need to have some content in those files or can they be empty? If I add the mount but not the file will it copy the container file to that location so I have the default files? Thanks four your help.

John2 · April 14, 2023, 1:52am

I already had a existing and running nextcloud docker container. I was concerned about messing things up if I changed these volumes. I did some testing and found you have to put the two files (remoteip.conf and redis-session.ini) there or you will get some errors.

This is what I did (less some things that failed):

Run docker-compose down on nextcloud container.
Make remoteip.conf and redis-session.ini files in the same directory as docker-compose.yml file for nextcloud. I did not have to do anything special with permissions. They are the same owner and group as the yml file.
Put the four lines above into the file called remoteip.conf (the RemoteIPHeader line and 3 RemoteIPInternalProxy lines).
Put nothing into the file called redis-session.ini (the container will add some lines).
Add the two volumes above to your docker-compose.yml. They are the lines that start with - ./remoteip.conf... and - ./redis-session.ini.... The location in the file and indent is important. Look at example above.
Run docker-compose up -d on nextcloud container.

I did not figure out what to do with the line that starts with LogFormat.... It sounds like it should go into apache2.conf and that should be mounted but did not see that with the other volumes.

Results:
After updating things still seem to work. Not thoroughly tested but it looks like the log that I see on the Logging page in the Administration section has real ip address now.

Very nice.

Thanks for your help.

Edit: I just reread the post and it does not look like we are suppose to do anything with the LogFormat... line.