Extreme slowness with large amount of files

Nextcloud version: 11.0.1
Operating system and version: Ubuntu 16.04
Apache or nginx version: NginX latest
PHP version: 7.1
Is this the first time you’ve seen this error?: NA

Can you reliably replicate it? (If so, please outline steps): NA

The issue you are facing:

I am using the cloud fo all of my Family storage.
The cloud is installed on AWS and storing the media on S3. All the files are encrypted.
Everything works perfectly, but extremely slow.
For example, sometimes loading the content of one of the folders takes more than a minute.
I assume it is related to the fact that files are encrypted (not sure though). How can I validate it? Is there a way to make it faster?

I have redis enabled and the server is not overloaded at all. It is not swapping or anything.