H264 sounds good. The codec stuff would have been one explanation why it takes longer than downloading the whole file. You could check if other Nextcloud instances have the same problem (e.g. Demo instances of 9-14 now live).
Finally, check with the developers for more help on debugging: https://github.com/nextcloud/files_videoplayer