I don’t know what you mean with “taking too long to respond” but if your server is exposed to the Internet it is vital that you use “fail2ban” or a similar tool. It’s possible to configure “fail2ban” in such a way that it will ban each and every ip-address after the first attempt to intrude.