Talk chat URLs always follow a scheme http://mydomain.tld/call/abcd123f - you could try to exclude the path from mTLS enforcement. I don’t have high expectations as Talk is not a stand-alone product but loads scripts and data from “core” Nextcloud - this would likely fail in such setup… latest if one shares a document (which is basically a regular share) it will brake. Maybe running another chat/conferencing tool like Mattermost without mTLS and using a Talk bridge is easier…
I don’t think there is a way to allow only Talk using another domain. As far I understand you are trying to allow anonymous access to Talk while enforcing mTLS for the rest of the instance. In general I think mTLS is much more complex to operate (think about mobiles and device replacement) and doesn’t significant improve security when you already follow best practices like frequent patching and MFA. If you really have to use mTLS you likely don’t want anybody else to access this instance.
Update: found similar request without solution, linking for reference: Restrict Nextcloud over VPN except Talk