Nextcloud Talk iOS App does not work with High Performance Backend

geiselOne · December 21, 2024, 4:22pm

Hey all!

Nextcloud version (eg, 24.0.1): 30.0.4
Talk Server version (eg, 14.0.2): 20.1.1
Custom Signaling server configured: yes, 2.0.1
Custom TURN server configured: yes
Custom STUN server configured: yes

In case mobile Nextcloud Talk apps are involved:
Talk iOS version (eg, 14.0.2): 20.1.0

The issue you are facing:
I cannot join any calls with the iOS app. My user appears and is shown to other participants as “Connecting”. After a few seconds, I drop from the call and the iOS app attempts to reconnect. This goes on forever.
When I switch to the internal signalling server, refresh all browsers, and restart the call, the iOS app is able to join.
Another user in our instance is experiencing the same problem on iOS. I have tested with / without Lockout mode on iOS - no change. All calls work perfectly for all participants on non-mobile operating systems / browsers. There are no issues whatsoever with STUN/TURN, the Signalling server or talk in general, unless the iOS app is used. I have not been able to test the Android app.

Is this the first time you’ve seen this error? (Y/N): N, this has happened in NC 29 as well, also with previous High Performance Backend versions.

Steps to replicate it:

Enable High Performance Backend
Use Talk iOS
Be unable to join calls

geiselOne · December 21, 2024, 4:27pm

PS: Also working on the iOS device: joining a call via link in the Safari Browser (no changes to network settings / wifi). I am quite certain that the problematic combination is iOS App + High Performance Backend.

SysKeeper · December 22, 2024, 3:14pm

There are currently no known issues with talk iOS and the external signaling controller. The only thing that recently popped up was an issue with multi sim iPhones, but that should be independent of internal/external signaling …

Are there any firewall rules in place maybe? Or does the participant have an restrictions? (E.g. no camera or something)?

geiselOne · December 22, 2024, 9:29pm

Hmm I have tested it on the same device, same wifi, but using the web version through safari instead of the app. There, it worked with both internal and external signalling.

It was a different call though, in safari I used an anonymous invite link and in the app I was signed in. Is there any way to check error logs from the iOS app?

SysKeeper · December 22, 2024, 9:44pm

You can find some logs inside the iOS files app, under Nextcloud Talk → Logs

SysKeeper · December 22, 2024, 9:49pm

While it indicates that a connection is possible, there might be other things at play here. Please see my questions above.

geiselOne · December 23, 2024, 8:33pm

I have retried everything on a mobile data connection with all VPNs etc. disabled. The symptoms are still the same (Safari works with external and internal signalling, app only works with external. I have collected a few logs, but I’m still not sure where the problem could lie.

Application logs with external signalling

2024-12-23 18:34:41.6690 (com.apple.main-thread): CallKit provider answer call action for token 89rvyc3e
2024-12-23 18:34:41.8560 (com.apple.main-thread): Connecting to: wss://spreed.example.com/standalone-signaling/spreed
2024-12-23 18:34:42.3390 (com.apple.main-thread): Joining room 89rvyc3e for call true
2024-12-23 18:34:42.5150 (com.apple.main-thread): WebSocket Connected!
2024-12-23 18:34:42.5170 (com.apple.main-thread): Sending hello message
2024-12-23 18:34:42.5270 (com.apple.main-thread): Joined room 89rvyc3e in NC successfully
2024-12-23 18:34:42.5280 (com.apple.main-thread): Trying to join room 89rvyc3e in external signaling server...
2024-12-23 18:34:42.5280 (com.apple.main-thread): Trying to send message before we received a hello response -> adding to pendingMessages
2024-12-23 18:34:42.6690 (com.apple.NSURLSession-delegate): Hello received with 1 pending messages
2024-12-23 18:34:42.7760 (com.apple.main-thread): Joined room 89rvyc3e in external signaling server successfully.
2024-12-23 18:34:42.7780 (com.apple.main-thread): Creating call controller for token 89rvyc3e
2024-12-23 18:34:42.9000 (com.apple.main-thread): Start call in NCCallController for token 89rvyc3e
2024-12-23 18:34:43.0320 (com.apple.main-thread): Join call in NCCallController for token 89rvyc3e
2024-12-23 18:34:43.3070 (webrtcClientDispatchQueue): Creating publisher peer connection with sessionId: jM3wZekHprSYnaLYu13ZEy2Ka49Ij7gVAXgHeywD_OJ8U2ZHTVpHdHQwOWF6b1BTWU5IVHFsSnVRZ0NUbmRDV2t1WUxJeU02QktWLVNXM0JsQVBSM2R3dVZDV1RyfDI4MjU3OTQzNzE=
2024-12-23 18:34:43.6690 (webrtcClientDispatchQueue): Did join call in NCCallController for token 89rvyc3e
2024-12-23 18:34:59.0150 (webrtcClientDispatchQueue): Publisher peer connection failed
2024-12-23 18:34:59.0150 (webrtcClientDispatchQueue): Force reconnect
2024-12-23 18:34:59.0560 (com.apple.main-thread): Reconnecting to: wss://spreed.example.com/standalone-signaling/spreed
2024-12-23 18:35:00.0580 (com.apple.main-thread): Connecting to: wss://spreed.example.com/standalone-signaling/spreed
2024-12-23 18:35:00.2550 (com.apple.main-thread): WebSocket Connected!
2024-12-23 18:35:00.2580 (com.apple.main-thread): Sending hello message
2024-12-23 18:35:00.3790 (com.apple.NSURLSession-delegate): Hello received with 0 pending messages
2024-12-23 18:35:00.3820 (com.apple.NSURLSession-delegate): Rejoining room 89rvyc3e
2024-12-23 18:35:00.6540 (com.apple.main-thread): Re-Joined room 89rvyc3e in external signaling server successfully.
2024-12-23 18:35:00.9450 (webrtcClientDispatchQueue): Creating publisher peer connection with sessionId: tcYIwupub-qHVmAl1PZ32rO2hbFTLE5hiwJmlIrr58N8VVl2eFdWS0YzRXZxYkNTMXZ0dlRTLTJ5cU5uMnNYamZEbXROZHV6Xy1ycUYzSzUweTlIc2hTdl9pOVE2fDAwMzU3OTQzNzE=
2024-12-23 18:35:16.0410 (webrtcClientDispatchQueue): Publisher peer connection failed
2024-12-23 18:35:16.0460 (webrtcClientDispatchQueue): Force reconnect

the difference to internal signalling seems to be the line (webrtcClientDispatchQueue): Publisher peer connection failed. Does this point to TURN as the issue? I do run my own TURN, but it should be in use regardless of external or internal signalling, right?

I think I see a log message in my coturn server correlating to this:
178243: (29): INFO: session 003000000000000023: closed (2nd stage), user <> realm <REALM.example.com> origin <>, local 172.17.2.3:3478, remote 172.17.2.2:35850, reason: TCP connection closed by client (callback)
(the weird IP addresses happen because COTURN proxies connections to the HPB in a docker network).

In other server logs, there are only few interesting log lines.

Backend:

hub.go:
Could not send MCU message &{Type:requestoffer Sid: RoomType:video Bitrate:0 Payload:map[] offerSdp:<nil> answerSdp:<nil>} for session jM3wZekHprSYnaLYu13ZEy2Ka49Ij7gVAXgHeywD_OJ8U2ZHTVpHdHQwOWF6b1BTWU5IVHFsSnVRZ0NUbmRDV2t1WUxJeU02QktWLVNXM0JsQVBSM2R3dVZDV1RyfDI4MjU3OTQzNzE= to BwI8S0VkjpeWq33I46yO69K00J9aft8YmGqH8UhRGOx8SzJucWIwMFV0NzF2R1dPNGNNdm43a2lwSUQ1NUhELU5Cbnc3cTFSR1BHNVk2Q2I5aVBBdmkwM3dvN1ZBfDY2MjU3OTQzNzE=: No such handle 5696679611316413 in session 7811297695773323

and

janus_client.go:
Unable to deliver message {
   "janus": "detached",
   "session_id": 7811297695773323,
   "sender": 5696679611316413
}. Handle 5696679611316413 gone?

Does this tell you something?

SysKeeper · December 23, 2024, 8:51pm

That’s what I expected from your description. So, again, are there firewall rules in place on the server? Does the user have full permissions or are they restricted ?

geiselOne · December 23, 2024, 9:10pm

Thanks, I will try to get some more detailed connection logs regarding the TURN server asap.

The only thing that I can imagine with this behavior (app has problems while Safari on the same device, same network works) is that the iOS app selects different ICE candidates or connects to the relay differently than the web app. It’s not possible to see details of the ICE in the app I assume?

What permissions should I look into exactly? At my coturn server? Or Nextcloud permissions?

SysKeeper · December 23, 2024, 9:27pm

It’s possible, yes. I remember a case where specific iptables rules lead to publisher peer failure, while it worked in a browser. So you might want to check if there’s anything set on the servers.

Regarding the permission, I meant talk permission. So is the user allowed to enabled camera and microphone? Are camera und microphone enabled or is talk iOS missing some OS permissions ?

geiselOne · December 24, 2024, 4:59pm

The permissions should be fine. I was able to successfully make calls with the same app / permissions when using internal signalling. Camera and mic worked well then.

Can you help me understand what removing the high performance backend does exactly if I have both a TURN and a STUN server configured? It should still use both, only the signalling and videobridge part will be run inside the Nextcloud instance, correct? So, in the normal case where TURN is needed, it will have to broker a connection to the videobridge inside Nextcloud - not to the Janus server inside the high performance backend?

Regarding the connectivity. I have double checked that UDP/TCP 443 is open towards my HPB server. I run all components in docker, and I managed to tcpdump the actual connection between the client and coturn (after TLS). For safari, this looks like a normal STUN/TURN conversation:

with the app, no STUN/TURN connection happens, only handshakes and then finally connection resets. But what we see from this is that even with the app, the requests definitely reach the TURN server.

Is there any condition where the app would not recognize a destination as a “proper” TURN server and shut down the connection before even speaking the first packet of STUN/TURN? My TURN server is fronted by a traefik proxy (172.17.2.2 in the screenshots) that does proper TLS with Let’s Encrypt certificates. TURN is set to “TURNS only” in the Talk settings.

cmeysing · December 27, 2024, 11:57am

Hello,
I have the same problem with several Nextcloud instances. Everything has worked great so far. But for a few days now, telephony and video are no longer available for selection, although both functions are permitted for the app.
The only change is that I recently started using the dual-sim function on my iPhone.
Could that be it?
Also for me, when I remove the high performance backend in Nextcloud, everything works again.
PS: I use my own signaling and my own turn server. The turn server runs via port 443.

Operating system: Linux 6.9.12-060912-generic #202408281052 SMP PREEMPT_DYNAMIC Wed Aug 28 15:52:05 UTC 2024 x86_64

Webserver: Apache (cgi-fcgi)

Database: mysql 8.0.40

PHP version: 8.3.15

Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, json, pcntl, random, Reflection, SPL, session, standard, sodium, cgi-fcgi, mysqlnd, PDO, xml, apcu, bcmath, bz2, calendar, ctype, curl, dom, mbstring, FFI, fileinfo, ftp, gd, gettext, gmp, iconv, igbinary, imagick, intl, ldap, exif, msgpack, mysqli, pdo_mysql, pdo_sqlite, Phar, posix, pspell, readline, shmop, SimpleXML, smbclient, sockets, sqlite3, ssh2, sysvmsg, sysvsem, sysvshm, tokenizer, xmlreader, xmlwriter, xsl, zip, memcached, redis, libsmbclient, Zend OPcache

Nextcloud version: 30.0.4 - 30.0.4.1

{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "***REMOVED SENSITIVE VALUE***"
    ],
    "logtimezone": "Europe\/Berlin",
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "dbtype": "mysql",
    "version": "30.0.4.1",
    "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
    "overwriteprotocol": "https",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "mysql.utf8mb4": true,
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "onlyoffice": {
        "verify_peer_off": true
    },
    "trashbin_retention_obligation": "auto,30",
    "integrity.check.disabled": false,
    "memcache.local": "\\OC\\Memcache\\APCu",
    "memcache.distributed": "\\OC\\Memcache\\Redis",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "filelocking.enabled": "true",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 0,
        "timeout": 0,
        "password": "***REMOVED SENSITIVE VALUE***"
    },
    "maintenance": false,
    "theme": "",
    "loglevel": 2,
    "updater.release.channel": "stable",
    "app_install_overwrite": [
        "files_fulltextsearch_tesseract",
        "apporder",
        "files_mindmap",
        "bbb",
        "workflow_pdf_converter",
        "workflow_script",
        "files_antivirus",
        "video_converter",
        "extract",
        "jitsi",
        "ransomware_protection",
        "camerarawpreviews",
        "files_linkeditor",
        "mail_roundcube",
        "groupfolders",
        "tasks",
        "llm"
    ],
    "mail_smtpmode": "sendmail",
    "mail_sendmailmode": "smtp",
    "default_phone_region": "DE",
    "mail_from_address": "***REMOVED SENSITIVE VALUE***",
    "mail_domain": "***REMOVED SENSITIVE VALUE***",
    "maintenance_window_start": 1,
    "allow_local_remote_servers": true
}

SysKeeper · December 27, 2024, 12:50pm

Mobile iOS with 2 SIM cards has issues during call via Talk · Issue #1912 · nextcloud/talk-ios · GitHub Could be indeed.

SysKeeper · December 27, 2024, 1:49pm

Hm, is traefik able to handle TURN data? Please note, TURN does not use HTTP as a protocol, although it is often exposed under port 443 for best compatibility with firewalls. Can you try it directly without the traefik proxy?
Also, using TURNS only might be problematic, you might want to try TURN only or TURN/TURNS. Since the media is already encrypted, it’s not unencrypted, it just does not have an additional TLS layer.

geiselOne · December 27, 2024, 2:21pm

I‘ll try with TURN only. The thought behind TURNS is that i want maximum firewall compatibility. If it looks like TLS and is on Port 443, i will have the most compatibility for my users.

Yes, Traefik can do TCP proxying.

What I still don‘t get though is why all browsers support this.

Also, do you have information about how this changes with internal signalling? My TURNS setup is exactly the same with internal signalling (that‘s when the app works). I will also try to capture the same transaction at my coTurn server when internal signalling is used.

SysKeeper · December 27, 2024, 2:28pm

Most of the time TURN on port 443 is enough to have a good compatibility, but sure, you can enable both in general.

With internal signaling a connection between the peers is done (P2P), with external signaling the janus server is one end of the connection. So that is a complete different way of connecting. Possible that in your tests with internal signaling TURN is not even needed and therefore it works fine, for example.

geiselOne · December 27, 2024, 6:54pm

So, partial success! Disabling TURNS and providing a TURN option lets me successfully make calls in the iOS app.

I now have a “stable” solution by providing multiple TURN URLs in my Nextcloud Talk config, so the app will select the TURN option while browsers will still select the TURNS option. The only downside is that I cannot provide a TURN TCP/443 option, as this port is reserved by traefik.

Do I have any option as a normal user to increase logging inside the app? I see all the verbose NSLog() statements for example in NCPeerConnection.m, but it appears those don’t get logged in the file. I would love to understand what makes the native swift libraries incompatible with my setup, while all other common browser software works flawlessly.

SysKeeper · December 27, 2024, 7:26pm

Cool that it works

We don’t have any option to increase logging at the moment. NSLog should print to the devices console IIRC, so if you have access to Mac, you should be able to get those logs.

In the end I fear that it will not help with the current log statements. You end up in an ICE connection state failed, that is already logged:

github.com

nextcloud/talk-ios/blob/f9e2c1c649ee3d05e7a715a65c6ad8937d549c03/NextcloudTalk/NCCallController.m#L1674-L1677


      
          // If publisher peer failed then reconnect
          if (peerConnection.isMCUPublisherPeer) {
              [NCUtils log:@"Publisher peer connection failed"];
              [self forceReconnect];

I don’t think we log anything more detailed, but I might be wrong.