Collabora not responding on 9980

ChiaraG · November 8, 2018, 12:36pm

Hi!

I’ve succesfully installed NextCloud as a snap on my Ubuntu Server 18.04. It works perfectly over https on a public domain cloud.chiara.org (not the real one ! ).
Now I want to integrate the Collabora application. I’ve installed it with the following command:

docker pull collabora/code
docker run -t -d -p 127.0.0.1:9980:9980 -e 'domain=cloud\\.chiara\\.org' --restart always --cap-add MKNOD collabora/code

The container is up and running.

docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                      NAMES
8398beb8a077        collabora/code      "/bin/sh -c 'bash st…"   2 hours ago         Up 2 hours          127.0.0.1:9980->9980/tcp   infallible_allen

Collabora should be on listening on port 9980, but if I try

curl 127.0.0.1:9980/hosting/discovery
curl: (52) Empty reply from server

uhm, that’s strange, maybe because it doesn’t reply to ip address. So I go to setup the reverse proxy.
I have a dedicated machine in my internal network that works as a reverse proxy of all the traffic coming from outside.
Below a schema of the network:

So the ReverseProxy is on the machine 192.168.1.132 which routes the traffic, accordingly to the domain name, to the server 192.168.1.30 which contains a NextCloud SNAP and the Docker container of CollaboraOnline.
There is no Apache service on 192.168.1.30, I mean outside the SNAP.

I write the virtual host, on 192.168.1.132 reverseProxy server, setting for Collabora in addition to what I already have for NextCloud:

<VirtualHost *:443>
    ServerName cloud.chiara.org
    ProxyPreserveHost On
    ProxyPass        / https://192.168.1.30/
    ProxyPassReverse / https://192.168.1.30/

    SSLProxyEngine on
    SSLCertificateFile /etc/letsencrypt/live/cloud.chiara.org/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/cloud.chiara.org/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

<VirtualHost *:443>
    ServerName docs.chiara.org:443

    # Encoded slashes need to be allowed
    AllowEncodedSlashes NoDecode

    # keep the host
    ProxyPreserveHost On

    # static html, js, images, etc. served from loolwsd
    # loleaflet is the client part of LibreOffice Online
    ProxyPass           /loleaflet https://192.168.1.30:9980/loleaflet retry=0
    ProxyPassReverse    /loleaflet https://192.168.1.30:9980/loleaflet

    # WOPI discovery URL
    ProxyPass           /hosting/discovery https://192.168.1.30:9980/hosting/discovery retry=0
    ProxyPassReverse    /hosting/discovery https://192.168.1.30:9980/hosting/discovery

    # Main websocket
    ProxyPassMatch "/lool/(.*)/ws$" wss://192.168.1.30:9980/lool/$1/ws nocanon

    # Admin Console websocket
    ProxyPass   /lool/adminws wss://192.168.1.30:9980/lool/adminws

    # Download as, Fullscreen presentation and Image upload operations
    ProxyPass           /lool https://192.168.1.30:9980/lool
    ProxyPassReverse    /lool https://192.168.1.30:9980/lool

    SSLProxyEngine on
    SSLProxyVerify None
    SSLProxyCheckPeerCN Off
    SSLProxyCheckPeerName Off

    # Let's Encrypt Certificate
    SSLCertificateFile /etc/letsencrypt/live/docs.chiara.org/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/docs.chiara.org/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf

    LogLevel debug
    ErrorLog ${APACHE_LOG_DIR}/error.log
</VirtualHost>

The certificate for cloud.chiara.org has been generated both on the ReverseProxy Server and on the Cloud Server using CertBot and Let’s Encrypt.
The certificate for docs.chiara.org has been generated only on the ReverseProxy Server using CertBot and Let’s Encrypt.

The problem is that CollaboraOnline is not responding at all, not reachable.
I have configured in the NextCloud app, under settings, by inserting the URL “https://docs.chiara.org” but when I try to edit an odt doc, I get the error “unable to complete the request…blablabla… remote address: 192.168.1.132…”
Why 192.168.1.132??? It’s the ReverseProxy Server!!
I check the ReverseProxy Apache log and I find the following:

[Thu Nov 08 13:27:25.297531 2018] [socache_shmcb:debug] [pid 16524] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0xdf -> subcache 31)
[Thu Nov 08 13:27:25.297561 2018] [socache_shmcb:debug] [pid 16524] mod_socache_shmcb.c(728): AH00842: expiring 1 and reclaiming 0 removed socache entries
[Thu Nov 08 13:27:25.297567 2018] [socache_shmcb:debug] [pid 16524] mod_socache_shmcb.c(747): AH00843: we now have 0 socache entries
[Thu Nov 08 13:27:25.297572 2018] [socache_shmcb:debug] [pid 16524] mod_socache_shmcb.c(845): AH00847: insert happened at idx=0, data=(0:32)
[Thu Nov 08 13:27:25.297576 2018] [socache_shmcb:debug] [pid 16524] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/207
[Thu Nov 08 13:27:25.297584 2018] [socache_shmcb:debug] [pid 16524] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Thu Nov 08 13:27:25.298306 2018] [ssl:debug] [pid 16524] ssl_engine_kernel.c(354): [client 2.231.118.170:54518] AH02034: Initial (No.1) HTTPS request received for child 10 (server docs.chiara.org:443)
[Thu Nov 08 13:27:25.298330 2018] [authz_core:debug] [pid 16524] mod_authz_core.c(835): [client 2.231.118.170:54518] AH01628: authorization result: granted (no directives)
[Thu Nov 08 13:27:25.298358 2018] [proxy:debug] [pid 16524] mod_proxy.c(1160): [client 2.231.118.170:54518] AH01143: Running scheme https handler (attempt 0)
[Thu Nov 08 13:27:25.298368 2018] [proxy:debug] [pid 16524] proxy_util.c(2160): AH00942: HTTPS: has acquired connection for (192.168.1.30)
[Thu Nov 08 13:27:25.298378 2018] [proxy:debug] [pid 16524] proxy_util.c(2213): [client 2.231.118.170:54518] AH00944: connecting https://192.168.1.30:9980/hosting/discovery to 192.168.1.30:9980
[Thu Nov 08 13:27:25.298440 2018] [proxy:debug] [pid 16524] proxy_util.c(2422): [client 2.231.118.170:54518] AH00947: connected /hosting/discovery to 192.168.1.30:9980
[Thu Nov 08 13:27:25.298618 2018] [proxy:error] [pid 16524] (111)Connection refused: AH00957: HTTPS: attempt to connect to 192.168.1.30:9980 (192.168.1.30) failed
[Thu Nov 08 13:27:25.298642 2018] [proxy:error] [pid 16524] AH00959: ap_proxy_connect_backend disabling worker for (192.168.1.30) for 0s
[Thu Nov 08 13:27:25.298649 2018] [proxy_http:error] [pid 16524] [client 2.231.118.170:54518] AH01114: HTTP: failed to make connection to backend: 192.168.1.30
[Thu Nov 08 13:27:25.298656 2018] [proxy:debug] [pid 16524] proxy_util.c(2175): AH00943: HTTPS: has released connection for (192.168.1.30)
[Thu Nov 08 13:27:25.298752 2018] [ssl:debug] [pid 16524] ssl_engine_io.c(1017): [client 2.231.118.170:54518] AH02001: Connection closed to child 10 with standard shutdown (server docs.chiara.org:443)
[Thu Nov 08 13:27:25.395640 2018] [socache_shmcb:debug] [pid 16511] mod_socache_shmcb.c(491): AH00831: socache_shmcb_store (0x1b -> subcache 27)
[Thu Nov 08 13:27:25.395670 2018] [socache_shmcb:debug] [pid 16511] mod_socache_shmcb.c(845): AH00847: insert happened at idx=0, data=(0:32)
[Thu Nov 08 13:27:25.395676 2018] [socache_shmcb:debug] [pid 16511] mod_socache_shmcb.c(850): AH00848: finished insert, subcache: idx_pos/idx_used=0/1, data_pos/data_used=0/207
[Thu Nov 08 13:27:25.395681 2018] [socache_shmcb:debug] [pid 16511] mod_socache_shmcb.c(512): AH00834: leaving socache_shmcb_store successfully
[Thu Nov 08 13:27:25.396624 2018] [ssl:debug] [pid 16511] ssl_engine_kernel.c(354): [client 2.231.118.170:53990] AH02034: Initial (No.1) HTTPS request received for child 2 (server docs.chiara.org:443)
[Thu Nov 08 13:27:25.396649 2018] [authz_core:debug] [pid 16511] mod_authz_core.c(835): [client 2.231.118.170:53990] AH01628: authorization result: granted (no directives)
[Thu Nov 08 13:27:25.396678 2018] [proxy:debug] [pid 16511] mod_proxy.c(1160): [client 2.231.118.170:53990] AH01143: Running scheme https handler (attempt 0)
[Thu Nov 08 13:27:25.396688 2018] [proxy:debug] [pid 16511] proxy_util.c(2160): AH00942: HTTPS: has acquired connection for (192.168.1.30)
[Thu Nov 08 13:27:25.396697 2018] [proxy:debug] [pid 16511] proxy_util.c(2213): [client 2.231.118.170:53990] AH00944: connecting https://192.168.1.30:9980/hosting/discovery to 192.168.1.30:9980
[Thu Nov 08 13:27:25.396757 2018] [proxy:debug] [pid 16511] proxy_util.c(2422): [client 2.231.118.170:53990] AH00947: connected /hosting/discovery to 192.168.1.30:9980
[Thu Nov 08 13:27:25.396988 2018] [proxy:error] [pid 16511] (111)Connection refused: AH00957: HTTPS: attempt to connect to 192.168.1.30:9980 (192.168.1.30) failed
[Thu Nov 08 13:27:25.397007 2018] [proxy:error] [pid 16511] AH00959: ap_proxy_connect_backend disabling worker for (192.168.1.30) for 0s
[Thu Nov 08 13:27:25.397014 2018] [proxy_http:error] [pid 16511] [client 2.231.118.170:53990] AH01114: HTTP: failed to make connection to backend: 192.168.1.30
[Thu Nov 08 13:27:25.397021 2018] [proxy:debug] [pid 16511] proxy_util.c(2175): AH00943: HTTPS: has released connection for (192.168.1.30)
[Thu Nov 08 13:27:25.397112 2018] [ssl:debug] [pid 16511] ssl_engine_io.c(1017): [client 2.231.118.170:53990] AH02001: Connection closed to child 2 with standard shutdown (server docs.chiara.org:443)

So the error is that the connection is refused because the attempt to connect to 192.168.1.30:9980 failed.
Please note that all firewalls are temporarily disabled!

Have I missed something during the installation and configuration?
How can I debug better why Collabora is not responding on 9980?

Many many thanks for your help

Chiara

Starfish · November 8, 2018, 12:49pm

I am not a 100% sure, but I think that is the problem. For your reverse proxy to work, you would need to expose the “public”(non-localhost) 192.168.1.30 IP, so that it is reachable to the reverse proxy? I might be completely wrong

When you start the container, what does ss -tulpn of netstat -tulpn yield? It should show which interface is listening on port 9980.

ChiaraG · November 8, 2018, 1:01pm

I get

ss -tulpn
Netid          State            Recv-Q           Send-Q                                           Local Address:Port                       Peer Address:Port
udp            UNCONN           2304             0                                                127.0.0.53%lo:53                              0.0.0.0:*
udp            UNCONN           43584            0                                                      0.0.0.0:5353                            0.0.0.0:*
udp            UNCONN           1536             0                           [fe80::baae:edff:fe77:3314]%enp3s0:546                                [::]:*
udp            UNCONN           0                0                                                         [::]:5353                               [::]:*
tcp            LISTEN           0                128                                              127.0.0.53%lo:53                              0.0.0.0:*
tcp            LISTEN           0                128                                                  127.0.0.1:9980                            0.0.0.0:*
tcp            LISTEN           0                128                                               192.168.1.30:13022                           0.0.0.0:*
tcp            LISTEN           0                128                                                          *:443                                   *:*
tcp            LISTEN           0                128                                                          *:80                                    *:*

and

netstat -tulpn
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      -
tcp        0      0 127.0.0.1:9980          0.0.0.0:*               LISTEN      -
tcp        0      0 192.168.1.30:13022      0.0.0.0:*               LISTEN      -
tcp6       0      0 :::443                  :::*                    LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      -
udp     2304      0 127.0.0.53:53           0.0.0.0:*                           -
udp    43584      0 0.0.0.0:5353            0.0.0.0:*                           -
udp6    1536      0 fe80::baae:edff:fe7:546 :::*                                -
udp6       0      0 :::5353                 :::*                                -

so you mean I should try

docker run -t -d -p 192.168.1.30:9980:9980 -e 'domain=cloud\\.chiara\\.org' --restart always --cap-add MKNOD collabora/code

I’ll try it, I’m completely new to Docker and I’m using it like a black box!
…but why there ip is specified with the port two times :9980:9980 ??

Starfish · November 8, 2018, 1:05pm

Yes, or what you could do is omit the IP completely and just expose it like:

docker run -t -d -p 9980:9980

It is not really specified twice. In the original you basically told docker “please map the ip 127.0.0.1 on port 9980 to the docker container’s port 9980.” In the line above you tell docker “please map all interfaces port 9980 to the container’s port 9980”.

See if that helps at all?

ChiaraG · November 8, 2018, 1:14pm

nothing to do, same situation as before: docker running, port 9980 on listening but no one responding…,

docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                    NAMES
0687e2c5148e        collabora/code      "/bin/sh -c 'bash st…"   10 seconds ago      Up 9 seconds        0.0.0.0:9980->9980/tcp   happy_jackson

ss -tulpn
Netid          State            Recv-Q           Send-Q                                           Local Address:Port                       Peer Address:Port
udp            UNCONN           14592            0                                                127.0.0.53%lo:53                              0.0.0.0:*
udp            UNCONN           43584            0                                                      0.0.0.0:5353                            0.0.0.0:*
udp            UNCONN           3072             0                           [fe80::baae:edff:fe77:3314]%enp3s0:546                                [::]:*
udp            UNCONN           0                0                                                         [::]:5353                               [::]:*
tcp            LISTEN           0                128                                              127.0.0.53%lo:53                              0.0.0.0:*
tcp            LISTEN           0                128                                               192.168.1.30:13022                           0.0.0.0:*
tcp            LISTEN           0                128                                                          *:443                                   *:*
tcp            LISTEN           0                128                                                          *:9980                                  *:*
tcp            LISTEN           0                128                                                          *:80                                    *:*

I’m feeling so down…

Thanks anyway

Starfish · November 8, 2018, 1:23pm

Can you reach

https://192.168.1.30:9980/loleaflet/dist/admin/admin.html

from any web browser?

ChiaraG · November 8, 2018, 1:27pm

YESS!!!
A ray of light in a dark day!!

It asks for a pwd

I tried also using the domain https://docs.chiara.org/loleaflet/dist/admin/admin.html
and it asks again for a pwd

Starfish · November 8, 2018, 1:32pm

OK, so that is your Collabora admin dashboard. Which means Collabora is running, listening and responding

Now it is just a question of getting Nextcloud to chat to it. This may be a stupid question, but did you restart your apache after creating the VirtualHost file? If you are running ubuntu, I know there is something like a2enmod which you need to use to enable a VirtualHost file/entry. You should be able to google that quite easily.

ChiaraG · November 8, 2018, 1:38pm

yes restarted every time.

now I can say

the ReverseProxy is working fine…
something in the docker container is responding…

but I’ve seen that the browser has told me that the connection was unsecured, so there is problem with the certificate, and in particular between the ReverseProxy machine (192.168.1.132) and Collabora (192.168.1.30)

For NextCloud I have installed two certificate: one on the ReverseProxy, and one on the NextCloud SNAP

But for Collabora how can I install a certificate inside docker??

I’m pretty sure this is the issue, as in all the installation tutorial they put the ReverseProxy on the same machine

Starfish · November 8, 2018, 1:40pm

I see quite a few differences between your Virtualhost file and theirs. Unfortunately it makes troubleshooting a bit harder.

For the sake of testing, have you tried using everything on HTTP, rather than HTTPS? Just so that you can get the basics going, and then you can HTTPS everything. I always find starting with the least amount of configurations makes thing a bit easier to understand.

trentmu · November 8, 2018, 1:40pm

I have found that if I run the docker with an additional parameter for DNS, and add the collabora machine to that machines /etc/hosts file, that it works best. eg.

docker run -t -d -p 127.0.0.1:9980:9980 -e ‘domain=nextcloud\\.domain\\.net’ –dns=192.168.1.1 --restart always --cap-add MKNOD collabora/code

Most aften, after this change, you need to restart the docker system with

sudo systemctl restart docker

ChiaraG · November 8, 2018, 2:01pm

thanks but nothing change

ChiaraG · November 8, 2018, 3:11pm

yeeeesss!!!

Solved! Infinite Thanks to @Starfish you pointed me in the right direction to discover the issue!

So it was all related to the last part of the communication between the ReverseProxy and the Collabora Server that was unprotected.

So, in order to encrypt also this segment I have had to install an Apache server, and all rev.proxy modules, also on the 192.168.1.30 server that contains both NextCloud SNAP with its own Apache and Collabora listening on 9980.

I’ve written the VirtualHosts settings

<VirtualHost *:80>
    ServerName docs.chiara.org
    DocumentRoot /var/www/html
</VirtualHost>

<VirtualHost *:9081>
    ServerName docs.chiara.org

    # Encoded slashes need to be allowed
    AllowEncodedSlashes NoDecode

    # keep the host
    ProxyPreserveHost On

    # static html, js, images, etc. served from loolwsd
    # loleaflet is the client part of LibreOffice Online
    ProxyPass           /loleaflet https://127.0.0.1:9980/loleaflet retry=0
    ProxyPassReverse    /loleaflet https://127.0.0.1::9980/loleaflet

    # WOPI discovery URL
    ProxyPass           /hosting/discovery https://127.0.0.1:9980/hosting/discovery retry=0
    ProxyPassReverse    /hosting/discovery https://127.0.0.1:9980/hosting/discovery

    # Main websocket
    ProxyPassMatch "/lool/(.*)/ws$" wss://127.0.0.1:9980/lool/$1/ws nocanon

    # Admin Console websocket
    ProxyPass   /lool/adminws wss://127.0.0.1:9980/lool/adminws

    # Download as, Fullscreen presentation and Image upload operations
    ProxyPass           /lool https://127.0.0.1:9980/lool
    ProxyPassReverse    /lool https://127.0.0.1:9980/lool

#    ServerAlias docs.brains.engineering
    SSLProxyEngine on
    SSLProxyVerify None
    SSLProxyCheckPeerCN Off
    SSLProxyCheckPeerName Off

Then I’ve modified the virtual host on the main ReverseProxy to send the docs.chiara.org traffic to 192.168.1.30 on port 9081

I’ve changed the HTTP listening port for NextCloud from 80 to 81, in any case not used as everything works on HTTPS

sudo snap set nextcloud ports.http=81

Once done this I got the new Apache ReverseProxy listening on 80 (needed by CertBot to issue the certificate) and 9081 redirecting to collabora on 9980

I’ve run CertBot, include the certificate in the VirtualHost, and bang! it Works!!

Ciao!
Chiara

Starfish · November 8, 2018, 5:12pm

Glad it worked, I did nothing really.

Please mark your answer Solved so future users can reference this extensive and well documented post of yours.

All the best

Sanook · March 11, 2019, 9:31am

You can specify your credentials in the docker start command:

docker run -t -d -p 127.0.0.1:9980:9980 -e 'domain=nextcloud\\.domain\\.xx\' -e 'dictionaries=de en' -e 'username=admin' -e 'password=Secret.Password' --restart always --cap-add MKNOD collabora/code

mobi1 · December 8, 2019, 8:13pm

Hi @ChiaraG, I am trying to do similar setup on one machine. Meaning that the Apache proxy server is on the same machine that NC and Collabora are hosted. I am having similar issue with the certificates. Could you share the proxy configuration both of NC and Collabora? That would help me see where I am configuring wrong.
Thanks