This is Jellyfin’s Doc page, but may help give you some idea of what will and what will not play on what browsers. As @baboonpower mentioned .mp4 files will almost always play, if not it’s probably related to your setup.
There has been some treads requesting a feature to do transcoding, but not sure when or if that will be implemented. Best options right now are to use software like handbrake to transcode your content first before uploading.
Kind of a work around, at least until something is implemented, is to use Jellyfin with Nextcloud as a backend, you can then share links from Jellyfin that will encode your content on the fly. https://media.childlearning.club All backend is managed through Nextcloud.