I ran some tests against my local server and had problems to send email via SMTP too.
Surprisingly sending email failed although an SSL connection (port 465/tcp) has successfully
been established. STARTTLS also failed because the authentication protocol CRAM-MD5 is
not supported by the Nextcloud function.
I was only able to send email using the sendmail interface. Why are you not using sendmail
to send email? If you haven’t installed an email server on your server, you could e.g. install
and use sSMTP as smtp client which provides a sendmail interface.