Ldap users : everybody can login

Hello,
we added LDAP authentication on our Nextcloud 20, it works like a charm, everybody can login with their ldap account, perfect.
BUT we want to choose who can connect.
I dont really understand how we can do that. Everybody can login (30k users…), we want to give access to 20/30 users for the moment.
Thank you very much