Well that is embarrassing. You are absolutely correct that I was inadvertently testing with UDP and I should have caught that earlier.
Continued Investigation
I kept investigating and this is what I found:
Local
From my local laptop I could successfully connect to fastmail servers as expected
local-laptop:~> telnet smtp.fastmail.com 587
Trying 103.168.172.60...
Connected to smtp.fastmail.com.
Escape character is '^]'.
220 smtp.fastmail.com ESMTP ready
Docker Container & Linode Instance
However, from within the docker container and on the nextcloud instance:
# Docker container
:~> nc -v smtp.fastmail.com 587
nc: connect to smtp.fastmail.com (103.168.172.60) port 587 (tcp) failed: Operation timed out
# Nextcloud instance
root@nextcloud:~> telnet smtp.fastmail.com 587
Trying 103.168.172.45...
^C
I am unable to connect, also as expected (since that is the issue I have been having).
UFW Maybe?
The ufw firewall was on so I thought maybe that was it. I turned it off and recieved the same timeout issues.
What the Actual Problem Is
Linode, in an effort to fight spam, decided to block all customers from sending email from any VPS if you are a customer after 2019. Blog post is here.
As an aside, you would think you could side step this by using the fastmail proxy server here.
So in the docker container:
:~> nc -v smtps-proxy.fastmail.com 80
Connection to smtps-proxy.fastmail.com (103.168.172.27) 80 port [tcp/http] succeeded!
connectivity looks good, but no matter what I put in the Nextcloud “Email server” options it always fails with:
"Message": "Connection to \"smtps-proxy.fastmail.com:80\" timed out."
I sent a support ticket to linode to allow smtp for my nextcloud instance. I will report back here once the respond and I get it working.