I did correctly set up mail service for the admin, under Administration, Basic settings, Email Server.
However, when I try to test it from that place, it does fail.
But when I go simply to the mail app, I can send mails from there without problems as well as receive.
The data for the mail server are same , means server, port, pass etc.
There were changes to the 3rd party mailer library in Nextcloud 26:
STARTTLS cannot be enforced. It will be used automatically if the mail server supports it. The encryption type should be set to ‘None/STARTTLS’ in this case. See here for an example on how to configure self signed certificates.