Hi,
I have a dockerized reverse proxy nginx sever sitting in front of two Nextcloud and Collabora. Both Nextcloud ans Collabora are in separate containers as well.
I use docker-compose file example from https://github.com/nextcloud/docker/tree/master/.examples/docker-compose/with-nginx-proxy/mariadb-cron-redis/fpm , so I have two different networks :
- on frontend network, I have reverse proxy + collabora
- on backend network, I have nextcloud + DB
- nextcloud nginx server is sitting on both network, so it can pass requests from reverse to nextcloud
Here is a schema of my current setup :
Everything related to Nextcloud itself works fine. Problems arise when I want to make Collabora and Nextcloud communicate. Iptables drops all the packets that are sent between Collabora or Nextcloud and reverse proxy.
I log every iptables dropped packets. Following are examples of dropped packets in different scenarios.
During test connexion from Nextcloud’s collabora application panel :
Nextcloud container (172.18.0.5) tries to reach external vps ip adress = dropped :
IPTables-Dropped: IN=br-network-backend-id OUT= PHYSIN=veth-some-id MAC=02:42... SRC=172.18.0.5 DST=my-vps-ip4-adress LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=7184 DF PROTO=TCP SPT=42554 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Let’s say I allow this connection in iptables. Then iptables blocks a connexion between frontend gateway (172.19.0.1) and reverse proxy server (172.19.0.2) :
IPTables-Dropped: IN= OUT=br-network-frontend-id SRC=172.19.0.1 DST=172.19.0.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=23334 DF PROTO=TCP SPT=35114 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
I then again allow this connexion, and test connexion works fine.
Opening a ods document from Nextcloud
Collabora container (172.19.0.3) tries to reach external vps ip adress = dropped :
IPTables-Dropped: IN=br-network-frontend-id OUT= PHYSIN=veth-some-id MAC=02:42.... SRC=172.19.0.3 DST=my-vps-ip4-adress LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=19007 DF PROTO=TCP SPT=43798 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0
From my point of view, it looks like both Nextcloud and Collabora are not using Proxy reverse nginx, and try instead to directly reach for VPS external ip adress, which is not allowed by Docker iptables custom rules.
I can of course add iptables custom rules for each of the dropped packes, but I feel like this is wrong, and should work otherwise. I don’t want to have to deal with custom rules. I want Collabora and Nexcloud to be able to communicate between each others behind a Proxy reverse server.
I’m certainly doing something wrong. Can someone give me a hint please ? Any help very appreciated
Here are my config files :
docker-compose.yaml
:
version: '3'
services:
proxy:
build: ./proxy
restart: always
ports:
- 80:80
- 443:443
labels:
com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
volumes:
- certs:/etc/nginx/certs:ro
- ./proxy/vhost.d:/etc/nginx/vhost.d:ro
- html:/usr/share/nginx/html
- /var/run/docker.sock:/tmp/docker.sock:ro
networks:
- frontend
nextcloud:
image: nextcloud:19.0.1-fpm
restart: always
volumes:
- code:/var/www/html
- data:/var/www/html/data
env_file:
- .env
depends_on:
- db
- redis
networks:
- backend
webserver:
build: ./web
restart: always
volumes:
- code:/var/www/html:ro
environment:
- VIRTUAL_HOST=cloud.domain.tld
- LETSENCRYPT_HOST=cloud.domain.tld
depends_on:
- app
networks:
- frontend
- backend
collabora:
image: collabora/code
restart: always
expose:
- 9980
cap_add:
- MKNOD
environment:
- "domain=cloud\\.domain\\.tld"
- "VIRTUAL_HOST=office.domain.tld"
- "VIRTUAL_PORT=9980"
- "LETSENCRYPT_HOST=office.doamin.tld"
networks:
- frontend
volumes:
- office:/etc/loolwsd # to be able to edit loolwsd.xml file more easily
letsencrypt-companion:
image: jrcs/letsencrypt-nginx-proxy-companion
restart: always
volumes:
- certs:/etc/nginx/certs:rw
- ./proxy/vhost.d:/etc/nginx/vhost.d:rw
- html:/usr/share/nginx/html
- /var/run/docker.sock:/var/run/docker.sock:ro
networks:
- frontend
depends_on:
- proxy
db:
image: mariadb
command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
restart: always
volumes:
- db:/var/lib/mysql
env_file:
- .env
networks:
- backend
redis:
image: redis:alpine
restart: always
networks:
- backend
cron:
image: nextcloud:19.0.1-fpm
restart: always
volumes:
- code:/var/www/html
- data:/var/www/html/data
entrypoint: /cron.sh
depends_on:
- db
- redis
networks:
- backend
volumes:
db:
code:
data:
certs:
html:
networks:
frontend:
backend:
Tree of working dir :
.
├── docker-compose.yml
├── office
│ └── loolwsd.xml
├── proxy
│ ├── Dockerfile
│ ├── limit_req.conf
│ ├── uploadsize.conf
│ └── vhost.d
│ ├── default
│ └── office.domain.tld
└── webserver
├── Dockerfile
└── nginx.conf
office.domain.tld
nginx vhost file :
## Start of configuration add by letsencrypt container
location ^~ /.well-known/acme-challenge/ {
auth_basic off;
auth_request off;
allow all;
root /usr/share/nginx/html;
try_files $uri =404;
break;
}
## End of configuration add by letsencrypt container
# static files
location ^~ /loleaflet {
proxy_pass http://office.domain.tld;
proxy_set_header Host $http_host;
}
# WOPI discovery URL
location ^~ /hosting/discovery {
proxy_pass http://office.domain.tld;
proxy_set_header Host $http_host;
}
# Capabilities
location ^~ /hosting/capabilities {
proxy_pass http://office.domain.tld;
proxy_set_header Host $http_host;
}
# main websocket
location ~ ^/lool/(.*)/ws$ {
proxy_pass http://office.domain.tld;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $http_host;
proxy_read_timeout 36000s;
}
# download, presentation and image upload
location ~ ^/lool {
proxy_pass http://office.domain.tld;
proxy_set_header Host $http_host;
}
# Admin Console websocket
location ^~ /lool/adminws {
proxy_pass http://office.domain.tld;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $http_host;
proxy_read_timeout 36000s;
}
Nginx for Nextcloud conf file is same as here : https://github.com/nextcloud/docker/blob/master/.examples/docker-compose/with-nginx-proxy/mariadb-cron-redis/fpm/web/nginx.conf
Reverse proxy uploadsize.conf
:
client_max_body_size 10G;
proxy_request_buffering off;
Reverse proxy limit_req.conf
:
limit_req_zone $binary_remote_addr zone=one:10m rate=1r/s;
loolwsd.xml
(relevant parts) :
<server_name desc="External hostname:port of the server running loolwsd. If empty, it's derived from the request (please set it if this doesn't work). Must be specified when behind a reverse-proxy or when the hostname is not reachable directly." type="string" default="">office.domain.tld</server_name>
<file_server_root_path desc="Path to the directory that should be considered root for the file server. This should be the directory containing loleaflet." type="path" relative="true" default="loleaflet/../"></file_server_root_path>
<net desc="Network settings">
<!-- On systems where localhost resolves to IPv6 [::1] address first, when net.proto is all and net.listen is loopback, loolwsd unexpectedly listens on [::1] only.
You need to change net.proto to IPv4, if you want to use 127.0.0.1. -->
<proto type="string" default="all" desc="Protocol to use IPv4, IPv6 or all for both">all</proto>
<listen type="string" default="any" desc="Listen address that loolwsd binds to. Can be 'any' or 'loopback'.">any</listen>
<service_root type="path" default="" desc="Prefix all the pages, websockets, etc. with this path."></service_root>
<proxy_prefix type="bool" default="false" desc="Enable a ProxyPrefix to be passed int through which to redirect requests"></proxy_prefix>
<post_allow desc="Allow/deny client IP address for POST(REST)." allow="true">
<host desc="The IPv4 private 192.168 block as plain IPv4 dotted decimal addresses.">192\.168\.[0-9]{1,3}\.[0-9]{1,3}</host>
<host desc="Ditto, but as IPv4-mapped IPv6 addresses">::ffff:192\.168\.[0-9]{1,3}\.[0-9]{1,3}</host>
<host desc="The IPv4 loopback (localhost) address.">127\.0\.0\.1</host>
<host desc="Ditto, but as IPv4-mapped IPv6 address">::ffff:127\.0\.0\.1</host>
<host desc="The IPv6 loopback (localhost) address.">::1</host>
<host desc="The IPv4 private 172.17.0.0/16 subnet (Docker).">172\.17\.[0-9]{1,3}\.[0-9]{1,3}</host>
<host desc="Ditto, but as IPv4-mapped IPv6 addresses">::ffff:172\.17\.[0-9]{1,3}\.[0-9]{1,3}</host>
</post_allow>
<frame_ancestors desc="Specify who is allowed to embed the LO Online iframe (loolwsd and WOPI host are always allowed). Separate multiple hosts by space."></frame_ancestors>
</net>
<ssl desc="SSL settings">
<enable type="bool" desc="Controls whether SSL encryption between browser and loolwsd is enabled (do not disable for production deployment). If default is false, must first be compiled with SSL support to enable." default="true">false</enable>
<termination desc="Connection via proxy where loolwsd acts as working via https, but actually uses http." type="bool" default="true">true</termination>
<cert_file_path desc="Path to the cert file" relative="false">/etc/loolwsd/cert.pem</cert_file_path>
<key_file_path desc="Path to the key file" relative="false">/etc/loolwsd/key.pem</key_file_path>
<ca_file_path desc="Path to the ca file" relative="false">/etc/loolwsd/ca-chain.cert.pem</ca_file_path>
<cipher_list desc="List of OpenSSL ciphers to accept" default="ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH"></cipher_list>
<hpkp desc="Enable HTTP Public key pinning" enable="false" report_only="false">
<max_age desc="HPKP's max-age directive - time in seconds browser should remember the pins" enable="true">1000</max_age>
<report_uri desc="HPKP's report-uri directive - pin validation failure are reported at this URL" enable="false"></report_uri>
<pins desc="Base64 encoded SPKI fingerprints of keys to be pinned">
<pin></pin>
</pins>
</hpkp>
</ssl>
<storage desc="Backend storage">
<filesystem allow="false" />
<wopi desc="Allow/deny wopi storage. Mutually exclusive with webdav." allow="true">
<host desc="Regex pattern of hostname to allow or deny." allow="true">cloud.domain.tld</host>
<host desc="Regex pattern of hostname to allow or deny." allow="true">10\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}</host>
<host desc="Regex pattern of hostname to allow or deny." allow="true">172\.1[6789]\.[0-9]{1,3}\.[0-9]{1,3}</host>
<host desc="Regex pattern of hostname to allow or deny." allow="true">172\.2[0-9]\.[0-9]{1,3}\.[0-9]{1,3}</host>
<host desc="Regex pattern of hostname to allow or deny." allow="true">172\.3[01]\.[0-9]{1,3}\.[0-9]{1,3}</host>
<host desc="Regex pattern of hostname to allow or deny." allow="true">192\.168\.[0-9]{1,3}\.[0-9]{1,3}</host>
<host desc="Regex pattern of hostname to allow or deny." allow="false">192\.168\.1\.1</host>
<max_file_size desc="Maximum document size in bytes to load. 0 for unlimited." type="uint">0</max_file_size>
<reuse_cookies desc="When enabled, cookies from the browser will be captured and set on WOPI requests." type="bool" default="false">false</reuse_cookies>
<locking desc="Locking settings">
<refresh desc="How frequently we should re-acquire a lock with the storage server, in seconds (default 15 mins) or 0 for no refresh" type="int" default="900">900</refresh>
</locking>
</wopi>
<webdav desc="Allow/deny webdav storage. Mutually exclusive with wopi." allow="false">
<host desc="Hostname to allow" allow="false">cloud.domain.tld</host>
</webdav>
<ssl desc="SSL settings">
<as_scheme type="bool" default="true" desc="When set we exclusively use the WOPI URI's scheme to enable SSL for storage">true</as_scheme>
<enable type="bool" desc="If as_scheme is false or not set, this can be set to force SSL encryption between storage and loolwsd. When empty this defaults to following the ssl.enable setting"></enable>
<cert_file_path desc="Path to the cert file" relative="false"></cert_file_path>
<key_file_path desc="Path to the key file" relative="false"></key_file_path>
<ca_file_path desc="Path to the ca file. If this is not empty, then SSL verification will be strict, otherwise cert of storage (WOPI-like host) will not be verified." relative="false"></ca_file_path>
<cipher_list desc="List of OpenSSL ciphers to accept. If empty the defaults are used. These can be overriden only if absolutely needed."></cipher_list>
</ssl>
</storage>