Does NextCloud only use S3 storage for uploaded files or can it also be used to store the whole database?
It would be possible.
But not good idea.
If you use an S3 based service for primary storage and your NextCloud database becomes corrupted or is lost, do you lose access to the files on the S3 service? Can they be recovered?
no. the s3 objects are still there.
well. you need the database table with the mapping s3 object name -> real filename.
i don’t know if there is way to recreate this mapping if your database has gone.
i wouldn’t run such a setup without professional support from nextcloud.