Dear all,
I recently introduced Nextcloud talk as a videoconverencing platform to our organization. We had our first conference yesterday and we experienced some serious issues which rendered usage of the platform impossible for a number of users.
In gerenal, the call comprosed approx. 15 members. I have setup coturn as a local TURN server, which is also configured as the local STUN server. An external signalling server is not set up yet.
Symptoms that we experienced throughout the call were as follows:
- While party A could talk to / hear B and B could talk to / hear C, A could not talk to / hear C. So there was some kind of fragmantation amongst the members. This for some reason was settled after repeated leaving and re-joining the call.
- Members repeatedly were kicked out of the call. According to the affected members, a pop-up message box appeared asking: “Should you really leave this page?”. If one would click on “Stay on this page”, one could neither hear not see anyone.
Regarding the second point, I figured that there was a strong correlation between the point in time of the members disconnecting and the appearance of the following log entry in the nextcloud log file:
Doctrine\DBAL\Exception\DeadlockException: An exception occurred while executing ‘UPDATE
oc_talk_participantsSETlast_ping= ? WHERE (user_id= ?) AND (session_id= ?) AND (room_id= ?)’ with params [1598800850, “”, “qtNQ3Q4L+bTr87KD3WjII9G0KB7Os50aj9vidaWC2jakFxjaBYsNI09JTepHZjr2cPt/WRKoiJsmBJzDS61gxX78flx6ukVW/Vkctyq1zQtf7knrj3pFYmtfjSDBJpYrZAtHdPXgJCFaKfCuhFLQkvx7tJ2I1vwE7ZDmASBI8ldjp21Q4Q0YZqXTbh5nojd2c12zxbFUgPsNdbazPNR4diZDtqEgyJ4l+oQZiIaqi1ZB14TbukjugwfPCCBd9ig”, 41]: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction
This happened quite often. Sometimes with actual usernames (redacted):
Doctrine\DBAL\Exception\DeadlockException: An exception occurred while executing ‘UPDATE
oc_talk_participantsSETlast_ping= ? WHERE (user_id= ?) AND (session_id= ?) AND (room_id= ?)’ with params [1598797458, “redacted.username”, “Bbc70Ghiq/UObAaBCNRoQP9igyaV/g/Lxy8Cy7T8Mcb1v6vyV8ULH6IlhmODdtfEnExpyofvn2Sd6aZvC9ROBwcBmY0bxGjVNkiyKuzgW77NI8OiAFPVuoSimIlB8UR3fuAVnkN5y6OvtMDS2NWoTFf2dWq+CM8e2MkhAICD1bh1pbjEGa/2ceG2eyCW0UwHq1bkk/+FviPtEsecRkolGKiAWzUPYWHdsYMSwX0xztvQ7B6k71vbXVSDgUZpUKT”, 41]: SQLSTATE[40001]: Serialization failure: 1213 Deadlock found when trying to get lock; try restarting transaction
I am running on a Debian 10, MariaDB + nginx. Entering SHOW VARIABLES; within mysql the max_connections variable is set to 200.
This is my turnserver.conf:
# Nextcloud talk configuration
fingerprint
use-auth-secret
keep-address-family
static-auth-secret=<redacted>
realm=my_hostna.me
cert=/etc/letsencrypt/live/my_hostna.me/fullchain.pem
pkey=/etc/letsencrypt/live/my_hostna.me/privkey.pem
listening-port=4446
tls-listening-port=4445
bps-capacity=0
stale-nonce
no-multicast-peers
no-stdout-log
log-file=/var/log/coturn.log
syslog
simple-log
Unfortunately, the log file if coturn is empty, although I explicitely defined it as the log file in turnserver.conf. Not sure why (0 -rw-r--r-- 1 www-data www-data 0 Aug 30 16:25 /var/log/coturn.log).
This is my talk configuration on Nextcloud side:
I am using ufw as a firewall and port 4446 (tcp + udp) is allowed.
Finally, when looking at the status of the coturn service, one can see that connections are reseted remotely:
$ sudo service coturn status
● coturn.service - coTURN STUN/TURN Server
Loaded: loaded (/lib/systemd/system/coturn.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2020-08-23 15:04:28 CEST; 1 weeks 0 days ago
Docs: man:coturn(1)
man:turnadmin(1)
man:turnserver(1)
Process: 24897 ExecStart=/usr/bin/turnserver --daemon -c /etc/turnserver.conf --pidfile /run/turnserver/turnserver.pid (code=exited, status=0/SUCCESS)
Process: 24899 ExecStartPost=/bin/sleep 2 (code=exited, status=0/SUCCESS)
Main PID: 24898 (turnserver)
Tasks: 12 (limit: 4915)
Memory: 24.8M
CGroup: /system.slice/coturn.service
└─24898 /usr/bin/turnserver --daemon -c /etc/turnserver.conf --pidfile /run/turnserver/turnserver.pid
Aug 30 17:22:27 v2202006124377121543 turnserver[24898]: 613081: session 001000000000001512: TCP socket error: Connection reset by peer xx.xx.xx.xx
Aug 30 17:23:43 v2202006124377121543 turnserver[24898]: 613156: session 003000000000000962: TCP socket error: Connection reset by peer xx.xx.xx.xx
Aug 30 17:23:43 v2202006124377121543 turnserver[24898]: 613157: session 000000000000001295: TCP socket error: Connection reset by peer xx.xx.xx.xx
Aug 30 17:26:27 v2202006124377121543 turnserver[24898]: 613321: session 002000000000001222: TCP socket error: Connection reset by peer xx.xx.xx.xx
Aug 30 17:27:07 v2202006124377121543 turnserver[24898]: 613361: session 002000000000001507: TCP socket error: Connection reset by peer xx.xx.xx.xx
Aug 30 17:30:11 v2202006124377121543 turnserver[24898]: 613545: session 005000000000001515: TCP socket error: Connection reset by peer xx.xx.xx.xx
Aug 30 17:30:22 v2202006124377121543 turnserver[24898]: 613556: session 003000000000001555: TCP socket error: Connection reset by peer xx.xx.xx.xx
Aug 30 17:37:35 v2202006124377121543 turnserver[24898]: 613989: session 003000000000001598: TCP socket error: Connection reset by peer xx.xx.xx.xx
Aug 30 17:49:37 v2202006124377121543 turnserver[24898]: 614711: session 004000000000001721: TCP socket error: Connection reset by peer xx.xx.xx.xx
Aug 30 17:54:55 v2202006124377121543 turnserver[24898]: 615029: session 002000000000001163: TCP socket error: Connection reset by peer xx.xx.xx.xx
So, from my point of view there seems to be at least one issue with database accesses and locks as well as a potential issue with network connectivity.
Do you know how this can be fixed as for the current state, Nextcloud talk is almost impossible to use.
Thanks in advance!