I’m using 2FA with my standard account too and don’t have any issues access CalDAV/CardDAV data via an app specific password. You need to provide more detailed information about what exactly happens if you try to access it, like e.g. providing the link you’re using, log entries etc., ewtc.
BTW you should change the subject of this post, because contacts can only be accessed using CardDAV and not CalDAV