How to give SAML Users Admin Rights

I am searching for a way to authenticate admins via SAML SSO. In my initial testing I was able to add the user to the admin group manually in Nextcloud. However after I added the group attribute mapping so that I could add groups to Nextcloud from the SAML provider (Azure AD) the admin group is removed at login. Even if I use a user that isn’t mapped to any groups on SAML provider. I have considered adding an admin group on the provider, however that’s very generic, it would be a lot better if I could use nextcloud-admin or something as the SAML provider is providing authentication for a lot of different services.