I’ve read more into what Hairpin NAT actually is here: https://serverfault.com/questions/55611/loopback-to-forwarded-public-ip-address-from-local-network-hairpin-nat/557776#557776
TLDR: The container can talk to the host using the public IP address. DNAT is rewriting the destination IP to that of the Caddy container, but it keeps the source address intact. Caddy then sends a reply directly to the container without going back through the NAT, thus the answer is lost.
The client thus sends a packet to an external IP address, but gets a reply from an internal IP address. It has no idea that the two packets are part of the same conversation, so no conversation happens.
Hairpin NAT therefore is a combination of SNAT and DNAT where the source address is rewritten to be the gateway address such that the reply can be translated appropriately.
Why you may need "Hairpin NAT" (NAT Reflection, NAT Loopback) for AIO/NC - #10 by Iv_Dark mentions that I would need Hairpin NAT, but I can’t figure out which iptables rules I need to create to make it work.