How to fix authorization through a load balancer?

Support intro

Good afternoon, masters! :slight_smile:
So, we have the following scheme on board:
х1 Load Balancer (HAProxy) 172.17.70.55
x2 Nextcloud servers 172.17.70.58 and 172.17.70.59
x1 Redis
x1 Minio
x2 Mysql master + Salve

Nextcloud servers work synchronously, separately for each web interface it enters without problems, under the same login, too, LDAP authorization is screwed on, but has not yet enabled it

Configured the LB HAProxy server, if we go to 172.17.70.55 - we get to the authorization page of one of the NC server, and then nothing happens - we enter the login password, and the deliberately false one - the page is updated and that’s it, nothing else happens

SSL not screwed on yet

Logs below

Nextcloud version (eg, 20.0.5): 21.0.0.
Operating system and version (eg, Ubuntu 20.04): 18.04
Apache or nginx version (eg, Apache 2.4.25): 2.4.29
PHP version (eg, 7.4): 7.4

The issue you are facing:

Is this the first time you’ve seen this error? (Y/N): Y

Steps to replicate it:

  1. Follow the link 172.17.70.55, the authorization window opens
  2. Enter the login and password of the administrator or user
  3. Loading takes 2 seconds and again returns to the authorization window. I did not see errors in the logs

The output of your Nextcloud log in Admin > Logging:

Warning	appstoreFetcher	GuzzleHttp\Exception\RequestException: cURL error 60: SSL certificate problem: unable to get local issuer certificate (see https://curl.haxx.se/libcurl/c/libcurl-errors.html) for https://apps.nextcloud.com/api/v1/apps.json
<?php
$CONFIG = array (
  'instanceid' => 'oceetf******',
  'passwordsalt' => 't2Zu4RO5cDNtVFzkmucO***********',
  'secret' => '9Gah9dRITODJgiS53OVY2XjgdALe*****************',
  'trusted_domains' =>
  array (
    0 => '172.17.70.58',
    1 => '172.17.70.55',
    2 => '172.17.70.59',

  ),
  'trusted_proxies' => '172.17.70.55',
  'datadirectory' => '/var/www/html/nextcloud/data',
  'dbtype' => 'mysql',
  'version' => '21.0.0.18',
  'overwrite.cli.url' => 'http://172.17.70.55',
  'dbname' => 'nextcloud',
  'dbhost' => '172.17.70.53',
  'dbport' => '3306',
  'dbtableprefix' => 'oc_',
  'dbuser' => '*******',
  'dbpassword' => '*********',
  'installed' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.distributed' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => '172.17.70.57',
    'port' => 6379,
  ),
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'theme' => '',

  'loglevel' => 2,
  'maintenance' => false,
  'updater.release.channel' => 'stable',
  'ldapProviderFactory' => 'OCA\\User_LDAP\\LDAPProviderFactory',
  'overwritecondaddr' => '^172\.17\.70\.55',

);

The output of your Apache/nginx/system log in /var/log/**error.log**:

[Wed Aug 04 16:26:25.866313 2021] [mpm_prefork:notice] [pid 3157] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 16:26:26.006529 2021] [mpm_prefork:notice] [pid 3214] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 16:26:26.006582 2021] [core:notice] [pid 3214] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 16:29:46.998388 2021] [mpm_prefork:notice] [pid 3214] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 16:29:47.135403 2021] [mpm_prefork:notice] [pid 3266] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 16:29:47.135465 2021] [core:notice] [pid 3266] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 16:33:23.490513 2021] [mpm_prefork:notice] [pid 3266] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 16:33:23.637955 2021] [mpm_prefork:notice] [pid 3344] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 16:33:23.638013 2021] [core:notice] [pid 3344] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 16:44:39.746637 2021] [mpm_prefork:notice] [pid 3344] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 16:44:39.890987 2021] [mpm_prefork:notice] [pid 3516] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 16:44:39.891046 2021] [core:notice] [pid 3516] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 16:45:23.759109 2021] [mpm_prefork:notice] [pid 3516] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 16:45:23.896707 2021] [mpm_prefork:notice] [pid 3581] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 16:45:23.896775 2021] [core:notice] [pid 3581] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 16:45:48.731059 2021] [mpm_prefork:notice] [pid 3581] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 16:45:48.867791 2021] [mpm_prefork:notice] [pid 3645] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 16:45:48.867855 2021] [core:notice] [pid 3645] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 16:47:18.915010 2021] [mpm_prefork:notice] [pid 3645] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 16:47:19.054039 2021] [mpm_prefork:notice] [pid 3714] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 16:47:19.054106 2021] [core:notice] [pid 3714] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 18:49:22.570900 2021] [mpm_prefork:notice] [pid 3714] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 18:49:22.719264 2021] [mpm_prefork:notice] [pid 4345] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 18:49:22.719348 2021] [core:notice] [pid 4345] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 18:53:45.418517 2021] [mpm_prefork:notice] [pid 4345] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 18:53:45.563446 2021] [mpm_prefork:notice] [pid 4400] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 18:53:45.563521 2021] [core:notice] [pid 4400] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 18:58:00.235018 2021] [mpm_prefork:notice] [pid 4400] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 18:58:00.368183 2021] [mpm_prefork:notice] [pid 4462] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 18:58:00.368240 2021] [core:notice] [pid 4462] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 19:00:50.982692 2021] [mpm_prefork:notice] [pid 4462] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 19:00:51.149140 2021] [mpm_prefork:notice] [pid 4520] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 19:00:51.149208 2021] [core:notice] [pid 4520] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 04 19:10:15.377388 2021] [mpm_prefork:notice] [pid 4520] AH00169: caught SIGTERM, shutting down
[Wed Aug 04 19:10:15.523625 2021] [mpm_prefork:notice] [pid 4695] AH00163: Apache/2.4.29 (Ubuntu) configured -- resuming normal operations
[Wed Aug 04 19:10:15.523691 2021] [core:notice] [pid 4695] AH00094: Command line: '/usr/sbin/apache2'

The output of your Apache/nginx/system log in /var/log/**access.log**:

172.19.2.18 - - [04/Aug/2021:19:25:08 +0700] "GET /index.php/apps/logreader/poll?lastReqId=tTfLhKmhdAP591NWdPDD HTTP/1.1" 200 801 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, l$
172.19.2.18 - - [04/Aug/2021:19:25:39 +0700] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 286 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)$
172.19.2.18 - - [04/Aug/2021:19:25:39 +0700] "PUT /index.php/apps/user_status/heartbeat HTTP/1.1" 204 696 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4$
172.19.2.18 - - [04/Aug/2021:19:25:39 +0700] "GET /ocs/v2.php/apps/user_status/api/v1/user_status HTTP/1.1" 200 934 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chr$
172.19.2.18 - - [04/Aug/2021:19:25:41 +0700] "GET /index.php/settings/admin/logging HTTP/1.1" 200 9095 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515$
172.19.2.18 - - [04/Aug/2021:19:25:42 +0700] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 1814 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko$
172.19.2.18 - - [04/Aug/2021:19:25:42 +0700] "PUT /index.php/apps/user_status/heartbeat HTTP/1.1" 204 696 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4$
172.19.2.18 - - [04/Aug/2021:19:25:42 +0700] "GET /ocs/v2.php/search/providers?from=%2Fsettings%2Fadmin%2Flogging HTTP/1.1" 200 1089 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML$
172.19.2.18 - - [04/Aug/2021:19:25:42 +0700] "GET /core/img/favicon.ico HTTP/1.1" 200 3563 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/$
172.19.2.18 - - [04/Aug/2021:19:25:42 +0700] "GET /ocs/v2.php/apps/user_status/api/v1/user_status HTTP/1.1" 200 935 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chr$
172.19.2.18 - - [04/Aug/2021:19:25:42 +0700] "GET /index.php/apps/logreader/settings HTTP/1.1" 200 887 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515$
172.19.2.18 - - [04/Aug/2021:19:25:42 +0700] "GET /index.php/apps/logreader/get?offset=0&count=50&levels=11111 HTTP/1.1" 200 6141 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, l$
172.19.2.18 - - [04/Aug/2021:19:25:42 +0700] "GET /cron.php HTTP/1.1" 200 906 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.131 Safari/537.36"
172.19.2.18 - - [04/Aug/2021:19:25:28 +0700] "GET /index.php/apps/logreader/poll?lastReqId=tTfLhKmhdAP591NWdPDD HTTP/1.1" 200 1863 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, $
172.19.2.18 - - [04/Aug/2021:19:25:42 +0700] "GET /index.php/apps/logreader/poll?lastReqId=tTfLhKmhdAP591NWdPDD HTTP/1.1" 200 1863 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, $
172.19.2.18 - - [04/Aug/2021:19:25:45 +0700] "GET /index.php/apps/logreader/poll?lastReqId=ZaiHmKP82I4HH1zUUAp3 HTTP/1.1" 200 801 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, l$
172.19.2.18 - - [04/Aug/2021:19:26:13 +0700] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 286 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)$
172.19.2.18 - - [04/Aug/2021:19:26:05 +0700] "GET /index.php/apps/logreader/poll?lastReqId=ZaiHmKP82I4HH1zUUAp3 HTTP/1.1" 200 801 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, l$
172.19.2.18 - - [04/Aug/2021:19:26:43 +0700] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 286 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)$
172.19.2.18 - - [04/Aug/2021:19:26:25 +0700] "GET /index.php/apps/logreader/poll?lastReqId=ZaiHmKP82I4HH1zUUAp3 HTTP/1.1" 200 801 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, l$
172.19.2.18 - - [04/Aug/2021:19:26:45 +0700] "GET /index.php/apps/logreader/poll?lastReqId=ZaiHmKP82I4HH1zUUAp3 HTTP/1.1" 200 801 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, l$
172.19.2.18 - - [04/Aug/2021:19:27:12 +0700] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 286 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)$
172.19.2.18 - - [04/Aug/2021:19:27:05 +0700] "GET /index.php/apps/logreader/poll?lastReqId=ZaiHmKP82I4HH1zUUAp3 HTTP/1.1" 200 801 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, l$
172.19.2.18 - - [04/Aug/2021:19:27:43 +0700] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 286 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)$
172.19.2.18 - - [04/Aug/2021:19:27:25 +0700] "GET /index.php/apps/logreader/poll?lastReqId=ZaiHmKP82I4HH1zUUAp3 HTTP/1.1" 200 801 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, l$
172.19.2.18 - - [04/Aug/2021:19:27:45 +0700] "GET /index.php/apps/logreader/poll?lastReqId=ZaiHmKP82I4HH1zUUAp3 HTTP/1.1" 200 801 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, l$
172.19.2.18 - - [04/Aug/2021:19:28:13 +0700] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 304 286 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)$

I miss information about the load balancer…
You have to make sure, that the user will be pointed to the same nextcloud server after login. This is called stickysessions sometimes.

Config HAProxy

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        # An alternative list with additional directives can be obtained from
        #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?server=haproxy
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend Local_Server
    bind 172.17.70.55:80
    mode http
    default_backend My_Web_Servers

backend My_Web_Servers
    mode http
    balance roundrobin
    option forwardfor
   # http-request set-header X-Forwarded-Port %[dst_port]
   # http-request add-header X-Forwarded-Proto https if { ssl_fc }
   # option httpchk HEAD / HTTP/1.1rnHost:localhost
    server web1  172.17.70.59:80
    server web2  172.17.70.58:80

I guess the problem is, that your user is getting balanced to server 1, tries to login and is redirected to server 2. The two servers don’t share the session information so the user seems to have login again. Try to setup stickysession cookies this could resolv this:
https://thisinterestsme.com/haproxy-sticky-sessions/

Maybe another way could be to use redis as session handler:
https://www.the-cake-shop.de/redis-als-php-session-handler-nutzen/

2 Likes

The cookie method helped! Thank you so much, your karma is sparkling! :innocent: