Nextcloud on Kubernetes is unable to be reached by a collabora server

Hello everyone!

I’ve been trying to set up Nextcould and Collabora on Kubernetes. At first I had them both on the cluster but after I kept having connectivity problems I spun up a vm for each and connected them.

The VM’s could talk to eachother. Awesome! I then connected the VM nextcloud to the collabora pod that I had in the cluster (after whitelisting the temp nextcloud domain) and that too worked.

So what I’m left with is that something either in my loadbalancer config (Digital Ocean LB) or something with the Ingress for my nextcloud instance. Any help or insights would be greatly appreciated.

What I get from the collabora installation on the vm is a timeout and the following logs:

Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.430492 [ docbroker_004 ] WRN  Socket #38 has timed out while requesting [GET cloud.amelia.sh/index.php/apps/richdocuments/wopi/files/6_ocdtt3rapsg2?access_token=NVC9ojwEcSROUSjyT3QeNysftLvZesjf&access_token_ttl=0] after 60019ms| net/HttpRequest.hpp:1302
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.432511 [ docbroker_004 ] INF  Missing JSON property [TemplateSaveAs] will default to [].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.432838 [ docbroker_004 ] INF  Missing JSON property [TemplateSource] will default to [].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.433111 [ docbroker_004 ] INF  Missing JSON property [WatermarkText] will default to [].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.433398 [ docbroker_004 ] INF  Missing JSON property [HideSaveOption] will default to [false].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.433775 [ docbroker_004 ] INF  Missing JSON property [EnableOwnerTermination] will default to [false].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.434071 [ docbroker_004 ] INF  Missing JSON property [DisableInactiveMessages] will default to [false].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.434334 [ docbroker_004 ] INF  Missing JSON property [SupportsLocks] will default to [false].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.434583 [ docbroker_004 ] INF  Missing JSON property [BreadcrumbDocName] will default to [].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.434797 [ docbroker_004 ] INF  Missing JSON property [FileUrl] will default to [].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.434998 [ docbroker_004 ] INF  Missing JSON property [IsUserFreemium] will default to [false].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.435197 [ docbroker_004 ] INF  Missing JSON property [DisableChangeTrackingRecord] will default to [false].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.435405 [ docbroker_004 ] INF  Missing JSON property [DisableChangeTrackingShow] will default to [false].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.435625 [ docbroker_004 ] INF  Missing JSON property [HideChangeTrackingControls] will default to [false].| ./common/JsonUtil.hpp:133
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.435939 [ docbroker_004 ] INF  WOPI::GetFile using default URI: https://cloud.amelia.sh/index.php/apps/richdocuments/wopi/files/6_ocdtt3rapsg2/contents?access_token=&access_token_ttl=0| wsd/Storage.cpp:1037
Aug 23 08:45:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:45:56.444086 [ docbroker_004 ] WRN  Waking up dead poll thread [HttpSynReqPoll], started: false, finished: false| ./net/Socket.hpp:680
Aug 23 08:46:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:46:56.491855 [ docbroker_004 ] WRN  Socket #24 has timed out while requesting [GET cloud.amelia.sh/index.php/apps/richdocuments/wopi/files/6_ocdtt3rapsg2/contents?access_token=NVC9ojwEcSROUSjyT3QeNysftLvZesjf&access_token_ttl=0] after 60050ms| net/HttpRequest.hpp:1302
Aug 23 08:46:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:46:56.493138 [ docbroker_004 ] INF  WOPI::GetFile downloaded 25178 bytes from [https://cloud.amelia.sh/index.php/apps/richdocuments/wopi/files/6_ocdtt3rapsg2/contents?access_token=&access_token_ttl=0] -> /opt/lool/child-roots/7Zm7fsSCggI3O3yM/tmp/user/docs/7Zm7fsSCggI3O3yM/Welcome to Nextcloud Hub.docx in 60056ms| wsd/Storage.cpp:1122
Aug 23 08:46:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:46:56.493687 [ docbroker_004 ] INF  SHA1 for DocKey [/index.php/apps/richdocuments/wopi/files/6_ocdtt3rapsg2] of [/tmp/user/docs/7Zm7fsSCggI3O3yM/Welcome to Nextcloud Hub.docx]: 8a66e0f5d255d5321bb8452f8be80b9fce78b9e0| wsd/DocumentBroker.cpp:961
Aug 23 08:46:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:46:56.494005 [ docbroker_004 ] INF  TileCache ctor for uri [https://cloud.amelia.sh/index.php/apps/richdocuments/wopi/files/6_ocdtt3rapsg2?access_token=&access_token_ttl=0], modifiedTime=1629708416], dontCache=false| wsd/TileCache.cpp:41
Aug 23 08:46:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:46:56.494082 [ docbroker_004 ] INF  Filesystem [/opt/lool/child-roots/.] has 16507 MB free (67.0083%).| common/FileUtil.cpp:477
Aug 23 08:46:56 collabora-tor01-deicated-droplet loolwsd[14779]: wsd-14779-36174 2021-08-23 08:46:56.494128 [ docbroker_004 ] ERR  Doc [/index.php/apps/richdocuments/wopi/files/6_ocdtt3rapsg2] is taking too long to load. Will kill process [15048]. per_document.limit_load_secs set to 100 secs.| wsd/DocumentBroker.cpp:323

and when I use the Collabora image within Kubernetes

wsd-00008-00034 2021-08-28 02:04:12.089492 [ docbroker_001 ] ERR  WOPI::CheckFileInfo failed for URI [https://cloud.amelia.sh/index.php/apps/richdocuments/wopi/files/148_ocrizsokec62?access_token=WW50N2vfn0zbe0uGbudQgUHR6f75rq7l&access_token_ttl=0]: 0 . Headers: 	Body: []| wsd/Storage.cpp:703
wsd-00008-00034 2021-08-28 02:04:12.091639 [ docbroker_001 ] ERR  loading document exception: WOPI::CheckFileInfo failed: | wsd/DocumentBroker.cpp:1917
wsd-00008-00034 2021-08-28 02:04:12.091819 [ docbroker_001 ] ERR  Failed to add session to [/index.php/apps/richdocuments/wopi/files/148_ocrizsokec62] with URI [https://cloud.amelia.sh/index.php/apps/richdocuments/wopi/files/148_ocrizsokec62?access_token=WW50N2vfn0zbe0uGbudQgUHR6f75rq7l&access_token_ttl=0]: WOPI::CheckFileInfo failed: | wsd/DocumentBroker.cpp:1879
wsd-00008-00034 2021-08-28 02:04:12.092460 [ docbroker_001 ] ERR  Storage error while starting session on /index.php/apps/richdocuments/wopi/files/148_ocrizsokec62 for socket #27. Terminating connection. Error: WOPI::CheckFileInfo failed: | wsd/LOOLWSD.cpp:3595
wsd-00008-00034 2021-08-28 02:04:12.093257 [ docbroker_001 ] WRN  Ignoring attempted read from 27| ./net/Socket.hpp:1021
wsd-00008-00034 2021-08-28 02:04:12.093738 [ docbroker_001 ] ERR  Invalid or unknown session [05d] to remove.| wsd/DocumentBroker.cpp:1956
wsd-00008-00031 2021-08-28 02:04:12.276353 [ websrv_poll ] WRN  DocBroker with docKey [/index.php/apps/richdocuments/wopi/files/148_ocrizsokec62] that is marked to be destroyed. Rejecting client request.| wsd/LOOLWSD.cpp:2144
wsd-00008-00031 2021-08-28 02:04:12.277436 [ websrv_poll ] ERR  Error while handling Client WS Request: Failed to create DocBroker with docKey [/index.php/apps/richdocuments/wopi/files/148_ocrizsokec62].| wsd/LOOLWSD.cpp:3624
wsd-00008-00031 2021-08-28 02:04:12.277506 [ websrv_poll ] ERR  #27: Socket write returned -1 (EPIPE: Broken pipe)| ./net/Socket.hpp:1284
wsd-00008-00031 2021-08-28 02:04:12.277572 [ websrv_poll ] ERR  #27: Socket write returned -1 (EPIPE: Broken pipe)| ./net/Socket.hpp:1284
wsd-00008-00031 2021-08-28 02:04:12.277593 [ websrv_poll ] WRN  Socket #27 is shutting down but 64 bytes couldn't be flushed and still remain in the output buffer.| ./net/WebSocketHandler.hpp:765
wsd-00008-00031 2021-08-28 02:04:12.277627 [ websrv_poll ] ERR  #27: attempted to remove: 1139 which is > size: 0 clamped to 0| ./net/Socket.hpp:1120
wsd-00008-00031 2021-08-28 02:04:12.277674 [ websrv_poll ] WRN  Ignoring attempted read from 27| ./net/Socket.hpp:1021
wsd-00008-00031 2021-08-28 02:04:12.277718 [ websrv_poll ] ERR  #27: Socket write returned -1 (EPIPE: Broken pipe)| ./net/Socket.hpp:1284
wsd-00008-00031 2021-08-28 02:04:12.968543 [ websrv_poll ] WRN  DocBroker with docKey [/index.php/apps/richdocuments/wopi/files/148_ocrizsokec62] that is marked to be destroyed. Rejecting client request.| wsd/LOOLWSD.cpp:2144
wsd-00008-00031 2021-08-28 02:04:12.969266 [ websrv_poll ] ERR  Error while handling Client WS Request: Failed to create DocBroker with docKey [/index.php/apps/richdocuments/wopi/files/148_ocrizsokec62].| wsd/LOOLWSD.cpp:3624
wsd-00008-00031 2021-08-28 02:04:12.969309 [ websrv_poll ] ERR  #27: Socket write returned -1 (EPIPE: Broken pipe)| ./net/Socket.hpp:1284
wsd-00008-00031 2021-08-28 02:04:12.969339 [ websrv_poll ] ERR  #27: Socket write returned -1 (EPIPE: Broken pipe)| ./net/Socket.hpp:1284
wsd-00008-00031 2021-08-28 02:04:12.969350 [ websrv_poll ] WRN  Socket #27 is shutting down but 64 bytes couldn't be flushed and still remain in the output buffer.| ./net/WebSocketHandler.hpp:765
wsd-00008-00031 2021-08-28 02:04:12.969372 [ websrv_poll ] ERR  #27: attempted to remove: 1139 which is > size: 0 clamped to 0| ./net/Socket.hpp:1120
wsd-00008-00031 2021-08-28 02:04:12.969397 [ websrv_poll ] WRN  Ignoring attempted read from 27| ./net/Socket.hpp:1021
wsd-00008-00031 2021-08-28 02:04:12.969411 [ websrv_poll ] ERR  #27: Socket write returned -1 (EPIPE: Broken pipe)| ./net/Socket.hpp:1284
wsd-00008-00026 2021-08-28 02:04:14.094148 [ prisoner_poll ] WRN  Prisoner connection disconnected but without valid socket.| wsd/LOOLWSD.cpp:2231
sh: 1: /usr/bin/loolmount: Operation not permitted
frk-00027-00027 2021-08-28 02:04:14.636180 [ forkit ] ERR  Failed to unmount [/opt/lool/child-roots/YsadnnLF7cTVXcnn/tmp].| common/JailUtil.cpp:68
sh: 1: /usr/bin/loolmount: Operation not permitted
frk-00027-00027 2021-08-28 02:04:14.654733 [ forkit ] ERR  Failed to unmount [/opt/lool/child-roots/YsadnnLF7cTVXcnn/lo].| common/JailUtil.cpp:68
sh: 1: /usr/bin/loolmount: Operation not permitted
frk-00027-00027 2021-08-28 02:04:14.672198 [ forkit ] ERR  Failed to unmount [/opt/lool/child-roots/YsadnnLF7cTVXcnn].| common/JailUtil.cpp:68
wsd-00008-00041 2021-08-28 02:04:15.175274 [ docbroker_002 ] WRN  Waking up dead poll thread [HttpSynReqPoll], started: false, finished: false| ./net/Socket.hpp:682
wsd-00008-00041 2021-08-28 02:04:15.175463 [ docbroker_002 ] ERR  Socket #30 SSL error: SYSCALL (5). (ECONNRESET: Connection reset by peer)| ./net/SslSocket.hpp:270
wsd-00008-00041 2021-08-28 02:04:15.175834 [ docbroker_002 ] ERR  Socket #30 SSL BIO error: closed unexpectedly (-1). (0: Success)| ./net/SslSocket.hpp:317
wsd-00008-00041 2021-08-28 02:04:15.176166 [ docbroker_002 ] ERR  Error while handling poll for socket #30 in HttpSynReqPoll: SSL Socket closed unexpectedly.| net/Socket.cpp:424
wsd-00008-00041 2021-08-28 02:04:15.176410 [ docbroker_002 ] ERR  WOPI::CheckFileInfo failed for URI [https://cloud.amelia.sh/index.php/apps/richdocuments/wopi/files/148_ocrizsokec62?access_token=WW50N2vfn0zbe0uGbudQgUHR6f75rq7l&access_token_ttl=0&permission=edit]: 0 . Headers: 	Body: []| wsd/Storage.cpp:703
wsd-00008-00041 2021-08-28 02:04:15.176529 [ docbroker_002 ] ERR  loading document exception: WOPI::CheckFileInfo failed: | wsd/DocumentBroker.cpp:1917
wsd-00008-00041 2021-08-28 02:04:15.176557 [ docbroker_002 ] ERR  Failed to add session to [/index.php/apps/richdocuments/wopi/files/148_ocrizsokec62] with URI [https://cloud.amelia.sh/index.php/apps/richdocuments/wopi/files/148_ocrizsokec62?access_token=WW50N2vfn0zbe0uGbudQgUHR6f75rq7l&access_token_ttl=0&permission=edit]: WOPI::CheckFileInfo failed: | wsd/DocumentBroker.cpp:1879
wsd-00008-00041 2021-08-28 02:04:15.176704 [ docbroker_002 ] ERR  Storage error while starting session on /index.php/apps/richdocuments/wopi/files/148_ocrizsokec62 for socket #25. Terminating connection. Error: WOPI::CheckFileInfo failed: | wsd/LOOLWSD.cpp:3595

Both cases Collabora is using the 6.4.10.10 docker image. The nextcloud vm and cluster image both are on the same version

The helmfile for this is here:
https://gitlab.com/-/snippets/2168140

I hope this is enough information, let me know if anything else would help.

I have the same issue with NC 23 and Collabora CODE version 21.11.0.6.1.

Were you able to fix this?