Reverse Proxy on Kubernetes

mhadibox · November 28, 2019, 10:47am

I have installed Nextcloud on my Kubernetes cluser (running under Rancher2) and it’s working fine excepting I have the following warning:

The reverse proxy header configuration is incorrect, or you are accessing Nextcloud from a trusted proxy. If not, this is a security issue and can allow an attacker to spoof their IP address as visible to the Nextcloud. Further information can be found in the documentation.

I tried to update my config file withe following snippet:

array (
    0 => 'mydomain.com',
    1 => '<ip of my load balancer>',
    2 => '10.42.0.82',
    3 => '127.0.0.1'
)

But still the same warning. Just to be sure, when updating the config file, are changes made instantly (or almost) ?

The ip of my load balancer is the ip on which my domain is pointing to.
The load balancer has been configured by following Rancher2’s documentation and contains the following NGINX configuration:

load_module /usr/lib/nginx/modules/ngx_stream_module.so;

worker_processes 4;
worker_rlimit_nofile 40000;

events {
    worker_connections 8192;
}

stream {
    upstream rancher_servers_http {
        least_conn;
        server <ip node1>:80 max_fails=3 fail_timeout=5s;
        server <ip node2>:80 max_fails=3 fail_timeout=5s;
        server <ip node3>:80 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     80;
        proxy_pass rancher_servers_http;
    }

    upstream rancher_servers_https {
        least_conn;
        server <ip node1>:443 max_fails=3 fail_timeout=5s;
        server <ip node2>:443 max_fails=3 fail_timeout=5s;
        server <ip node3>:443 max_fails=3 fail_timeout=5s;
    }
    server {
        listen     443;
        proxy_pass rancher_servers_https;
    }
}

The 10.42.0.82 ip is the internal ip of my pod. Here is the description of the pod:

Name:           nextcloud-release-76d557b7ff-mnk6x
Namespace:      nextcloud
Priority:       0
Node:           <ip node1>/<ip node1>
Start Time:     Thu, 28 Nov 2019 09:04:45 +0000
Labels:         app.kubernetes.io/instance=nextcloud-release
                app.kubernetes.io/name=nextcloud
                nextcloud-release-redis-client=true
                pod-template-hash=76d557b7ff
Annotations:    <none>
Status:         Running
IP:             10.42.0.82
IPs:            <none>
Controlled By:  ReplicaSet/nextcloud-release-76d557b7ff
Containers:
  nextcloud:
    Container ID:   docker://f5b4195b4b2367139687bc7c9a5aaa343a169da98786ae112cc10a1a99ac04cc
    Image:          nextcloud:17.0.0-apache
    Image ID:       docker-pullable://nextcloud@sha256:96104cb965fc1a9db3757d691533fcec1066953c9c519a2bb92198f045592845
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 28 Nov 2019 09:06:48 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 28 Nov 2019 09:05:42 +0000
      Finished:     Thu, 28 Nov 2019 09:06:47 +0000
    Ready:          True
    Restart Count:  1
    Liveness:       http-get http://:http/status.php delay=30s timeout=5s period=15s #success=1 #failure=3
    Readiness:      http-get http://:http/status.php delay=30s timeout=5s period=15s #success=1 #failure=3
    Environment:
      MYSQL_HOST:                 nextcloud-release-mariadb
      MYSQL_DATABASE:             nextcloud
      MYSQL_USER:                 <set to the key 'db-username' in secret 'nextcloud-release-db'>      Optional: false
      MYSQL_PASSWORD:             <set to the key 'db-password' in secret 'nextcloud-release-db'>      Optional: false
      NEXTCLOUD_ADMIN_USER:       <set to the key 'nextcloud-username' in secret 'nextcloud-release'>  Optional: false
      NEXTCLOUD_ADMIN_PASSWORD:   <set to the key 'nextcloud-password' in secret 'nextcloud-release'>  Optional: false
      NEXTCLOUD_TRUSTED_DOMAINS:  mydomain.com
      NEXTCLOUD_DATA_DIR:         /var/www/html/data
      MAIL_FROM_ADDRESS:          smtp.gmail.com
      MAIL_DOMAIN:                smtp.gmail.com
      SMTP_HOST:                  smtp.gmail.com
      SMTP_SECURE:                ssl
      SMTP_PORT:                  587
      SMTP_AUTHTYPE:              LOGIN
      SMTP_NAME:                  <set to the key 'smtp-username' in secret 'nextcloud-release'>  Optional: false
      SMTP_PASSWORD:              <set to the key 'smtp-password' in secret 'nextcloud-release'>  Optional: false
      REDIS_HOST:                 nextcloud-release-redis-master
      REDIS_HOST_PORT:            6379
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-49m7d (ro)
      /var/www/ from nextcloud-data (rw,path="root")
      /var/www/html from nextcloud-data (rw,path="html")
      /var/www/html/config from nextcloud-data (rw,path="config")
      /var/www/html/custom_apps from nextcloud-data (rw,path="custom_apps")
      /var/www/html/data from nextcloud-data (rw,path="data")
      /var/www/html/themes from nextcloud-data (rw,path="themes")
      /var/www/tmp from nextcloud-data (rw,path="tmp")
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  nextcloud-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  nextcloud-release-nextcloud
    ReadOnly:   false
  default-token-49m7d:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-49m7d
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

I can aslo share the ingress and nginx-controller description:

Name:             nextcloud-ingress
Namespace:        nextcloud
Address:          <ip node1>,<ip node2>,<ip node3>
Default backend:  default-http-backend:80 (<none>)
TLS:
  nextcloud-tls terminates mydomain.com
Rules:
  Host                Path  Backends
  ----                ----  --------
  mydomain.com
                      /   nextcloud-release:8080 (10.42.0.82:80)
Annotations:
  field.cattle.io/ingressState:     {"bmV4dGNsb3VkLWluZ3Jlc3MvbmV4dGNsb3VkL293Y2xvdWQuYWRpc29mdC5iZS8vLzgwODA=":"","bmV4dGNsb3VkLXRscw==":"nextcloud:nextcloud-tls"}
  field.cattle.io/publicEndpoints:  [{"addresses":["<ip node1>","<ip node2>","<ip node3>"],"port":443,"protocol":"HTTPS","serviceName":"nextcloud:nextcloud-release","ingressName":"nextcloud:nextcloud-ingress","hostname":"mydomain.com","path":"/","allNodes":false}]
  kubernetes.io/ingress.class:      nginx
Events:
  Type    Reason  Age                 From                      Message
  ----    ------  ----                ----                      -------
  Normal  UPDATE  51m (x5 over 118m)  nginx-ingress-controller  Ingress nextcloud/nextcloud-ingress
  Normal  UPDATE  51m (x5 over 118m)  nginx-ingress-controller  Ingress nextcloud/nextcloud-ingress
  Normal  UPDATE  51m (x5 over 118m)  nginx-ingress-controller  Ingress nextcloud/nextcloud-ingress

Name:           nginx-ingress-controller-6r78b
Namespace:      ingress-nginx
Priority:       0
Node:           <ip node2>/<ip node2>
Start Time:     Fri, 22 Nov 2019 10:01:22 +0000
Labels:         app=ingress-nginx
                controller-revision-hash=58496446c5
                pod-template-generation=1
Annotations:    prometheus.io/port: 10254
                prometheus.io/scrape: true
Status:         Running
IP:             <ip node2>
IPs:            <none>
Controlled By:  DaemonSet/nginx-ingress-controller
Containers:
  nginx-ingress-controller:
    Container ID:  docker://e0080d2dd4cf507edf37f9d860cb96153e12990b71a012221bdd3d6e301b8451
    Image:         rancher/nginx-ingress-controller:nginx-0.25.1-rancher1
    Image ID:      docker-pullable://rancher/nginx-ingress-controller@sha256:b5b59d579b7e5ff3a032db0f6dfc405dd4b7e1961307dd0926574d7182e7caf6
    Ports:         80/TCP, 443/TCP
    Host Ports:    80/TCP, 443/TCP
    Args:
      /nginx-ingress-controller
      --default-backend-service=$(POD_NAMESPACE)/default-http-backend
      --configmap=$(POD_NAMESPACE)/nginx-configuration
      --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      --annotations-prefix=nginx.ingress.kubernetes.io
    State:          Running
      Started:      Fri, 22 Nov 2019 10:01:40 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:10254/healthz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      POD_NAME:       nginx-ingress-controller-6r78b (v1:metadata.name)
      POD_NAMESPACE:  ingress-nginx (v1:metadata.namespace)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from nginx-ingress-serviceaccount-token-fjc2h (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  nginx-ingress-serviceaccount-token-fjc2h:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx-ingress-serviceaccount-token-fjc2h
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     :NoExecute
                 :NoSchedule
                 node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/network-unavailable:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/pid-pressure:NoSchedule
                 node.kubernetes.io/unreachable:NoExecute
                 node.kubernetes.io/unschedulable:NoSchedule
Events:          <none>

And last but not least. I’m not sure those messages are related but I also have

Your web server is not properly set up to resolve “/.well-known/caldav”. Further information can be found in the documentation.

Your web server is not properly set up to resolve “/.well-known/carddav”. Further information can be found in the documentation.

Under nextcloud settings page.

My Nextcloud version is 17.0.1.

Any help is welcome.

Feel free to ask more information on my setup.

Thanks.

Reiner_Nippes · November 29, 2019, 10:14am

is that the external or internal ip of your ingress controller?

is that the trusted domain or the trusted proxy entry in your config.php?

i think you have to include this into the config of your nginx ingress controller:

    location = /.well-known/carddav {
      return 301 $scheme://$host:$server_port/remote.php/dav;
    }
    location = /.well-known/caldav {
      return 301 $scheme://$host:$server_port/remote.php/dav;
    }

mhadibox · November 29, 2019, 12:42pm

This isn’t the ip of my ingress-controller. This is the ip of my VM hosting the NGINX configuration that is behind Rancher.

This was effectively trusted_domains. But I’ve tried with the following snippets too without any success:

 'trusted_domains' =>
  array (
          0 => 'mydomain.com',
          1 => '<load balancer ip>',
          2 => '127.0.0.1',
          3 => '10.42.2.184', // = Pod IP
          4 => '<ip node1>',
          5 => '<ip node2>',
          6 => '<ip node3>'
  ),
  'trusted_proxies' =>
  array (
          0 => 'mydomain.com',
          1 => '<load balancer ip>',
          2 => '127.0.0.1',
          3 => '10.42.2.184', // = Pod IP
          4 => '<ip node1>',
          5 => '<ip node2>',
          6 => '<ip node3>'
  )

Do you have any tips on how I could add this to my nginx-controller ?
I’m new to Kubernetes and as far as I know the config file is generated by following ingress rules ?

Thanks for your help.

Reiner_Nippes · November 29, 2019, 1:15pm

i think you have to put it in the nginx configuration which you posted below:

trusted: i only use the nextcloud fqdn, the name of the nginx container (ngninx as trusted domains and the internal/backend ip of the ingress controller (in my case traefik)

github.com

ReinerNippes/nextcloud_on_docker/blob/575c96b248cafb01b621ea3f6787c64418baa514/roles/nextcloud_config/defaults/main.yml#L6


      
          ---
          # defaults file for prep_nextcloud
          
          nextccloud_system_config:
            - "overwrite.cli.url --value=https://{{ nextcloud_server_fqdn }}"
            - "trusted_domains 0 --value={{ nextcloud_server_fqdn }}"
            - "trusted_domains 1 --value=nginx"
            - "trusted_proxies 0 --value={{ traefik_facts.ansible_facts.docker_container.NetworkSettings.Networks.backend.IPAddress }}"
            - "redis host --value='redis'"
            - "redis port --value='6379' --type=integer"
            - "redis timeout --value='0' --type=integer"
            - "redis password --value={{ lookup('password', '{{ nextcloud_credential_store }}/redis_secret chars=ascii_letters,digits length=32') }}"
            - "memcache.local --value='\\OC\\Memcache\\Redis'"
            - "memcache.locking --value='\\OC\\Memcache\\Redis'"
            - "filelocking.enabled --value=true --type=boolean"
            - "enable_previews --value=true --type=boolean"

mhadibox · November 29, 2019, 1:47pm

Where would you put

location = /.well-known/carddav {
          return 301 $scheme://$host:$server_port/remote.php/dav;
        }
        location = /.well-known/caldav {
          return 301 $scheme://$host:$server_port/remote.php/dav;
        }

?

I tried on the root of my config file, under “stream” and under both “server” annotations but I always get a

 "location" directive is not allowed here in /etc/nginx/nginx.conf:33

I already have FQDN in my trusted proxies: this is mydomain.com.
And the IP of my ingress-controllers seems to be the IP of my nodes:

kubectl get pods -n ingress-nginx -o wide
NAME                                    READY   STATUS    RESTARTS   AGE    IP               NODE             NOMINATED NODE   READINESS GATES
default-http-backend-5bcc9fd598-xmpr8   1/1     Running   0          7d3h   10.42.2.3        149.202.36.244   <none>           <none>
nginx-ingress-controller-6r78b          1/1     Running   1          7d3h   149.202.36.244   149.202.36.244   <none>           <none>
nginx-ingress-controller-d7vlx          1/1     Running   0          7d3h   149.202.36.245   149.202.36.245   <none>           <none>
nginx-ingress-controller-mxqs5          1/1     Running   0          7d3h   149.202.36.243   149.202.36.243   <none>           <none>

Which I already add to my trusted_proxies.
I’m surely misunderstanding something as Kubernetes and NGINX are quite new to me…

Reiner_Nippes · November 29, 2019, 2:06pm

imho it would be in the server section.

And you may look at the examples at the rancher doc page in your first post. They include some proxy header stuff. That is missing in your config.

jsalatiel · April 9, 2020, 2:36pm

@mhadibox Have you got any success on this ? I am facing the same problem.