Recently I was able to make nextcloud detect my ip and so I finally did setup fail2ban and it works nice, but now also bruteforce settings shoudl work fine, so I wonder, should I have both or just? And which one?
I think as you have both already installed, why not keep them. i personally have not used fail2ban on nextcloud so im not familiar(i just use the brute force setting). if both does not cause any problem in a way, why not keep them both. Though i will say i have not much expiernce with using fail2ban on Nextcloud though on mailcow it works like a charm. Goodluck!
Fail2ban after you make the ip correct in nextcloud works like anywhere else 
And thanks for the suggestion, they both can work together thought the fail2ban more often then one will ban a first first giving no option to begin punished by bruteforce settings