High AWS S3 costs due to Nextcloud requests

Hi!

We started running Nextcloud App as a prebuilt-docker image in Univention Corporate Server.
External (not primary) Storage for everything is AWS S3 and we have about 180 GB in our buckets. There are only less than 10 users, all using the Windows client or the web login. Files are not changed in the buckets externally, only by Nextcloud.

We received the AWS bill for last month and it says:
“Amazon Simple Storage Service EUC1-Requests-Tier1” ($0.0054 per 1,000 PUT, COPY, POST, or LIST requests): 48,217,810.000 Requests / 260.38 USD
“Amazon Simple Storage Service EUC1-Requests-Tier1” ($0.0043 per 1,000 PUT, COPY, POST, or LIST requests): 46,015,181.000 Requests / 19.79 USD

So we were paying about 280 USD for some requests and only a few bucks for the storage itself.

I suspect that there is a misconfiguration which leads to this huge amount of requests.

Any help or at least explanation for Nextcloud’s behaviour is appreciated.

Thanks in advance!

Nextcloud version 16.0.5-0
Operating system and version Univention (Debian) 4.4-3 errata423
Univentions currently supports updates up to Nextcloud 16.0.6-0

config.php:

<?php
$CONFIG = array (
  'passwordsalt' => 'foobarfoobarfoobarfoobarfoobar',
  'secret' => 'foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar',
  'trusted_domains' =>
  array (
    0 => 'nextcloud.int.example.com',
    1 => '172.16.30.21',
    2 => 'nextcloud.example.com',
  ),
  'datadirectory' => '/var/lib/univention-appcenter/apps/nextcloud/data/nextcloud-data',
  'dbtype' => 'pgsql',
  'version' => '16.0.5.1',
  'overwrite.cli.url' => 'https://nextcloud.int.example.com/nextcloud',
  'dbname' => 'nextcloud',
  'dbhost' => '172.17.42.1',
  'dbport' => '5432',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud',
  'dbpassword' => 'foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar',
  'installed' => true,
  'instanceid' => 'abcdefg12345',
  'updatechecker' => 'false',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'overwriteprotocol' => 'https',
  'overwritewbroot' => '/nextcloud',
  'htaccess.RewriteBase' => '/nextcloud',
  'ldapIgnoreNamingRules' => false,
  'ldapProviderFactory' => 'OCA\\User_LDAP\\LDAPProviderFactory',
  'trusted_proxies' =>
  array (
    0 => '172.17.42.1',
  ),
  'maintenance' => false,
  'loglevel' => 2,
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'nextcloud',
  'mail_domain' => 'example.com',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'mail.example.com',
  'mail_smtpport' => '587',
  'mail_smtpname' => 'mailuser',
  'mail_smtppassword' => 'foobarfoobarfoobarfoobarfoobarfoobarfoobar',
);

Is check for changes on direct access enabled on the external storage? If so that’s your problem.

Otherwise 180GB of data is vague. 180GB could be millions of files.

Amazon S3 is expensive and there are cheaper alternatives out there.

Thank you very much for your quick answer, I appreciate that!

Yes, “check for changes on direct access” was set, now I turned it off.
Do I have to take care of something like re-indexing the S3 buckets periodically? Nextcloud is using it exclusively, there are no other accesses to the bucket.

180 GB is about 75,000 files in our case.

I will monitor the API calls over the next few days after changing the setting and then decide if we go back to EBS and snapshots.

“Check for changes on direct access” causes it to scan the entire directory and see if there are any new/changed files.

I don’t know how it handles the backend on S3 but knowing how S3 is not a priority for the developers it likely sends requests to every file. So if you have 30k files in the directory you will be using 30k API requests just opening the directory. Not sure if this translates to cron, either.

Nextcloud is the metadata server for S3. All uploads/downloads/changes/deletions/new etc are handled directly by nextcloud. It’s pointless to check the bucket for changes and a waste of money.

Hopefully that fixes it for you!

If you switch to an S3 compatible service like Wasabi, the API requests are not billable. It sounds like you are using S3 as an external storage mount (as opposed to primary storage) which means migrating should be fairly simple.

You could use a tool like rclone to copy everything over, then simply remount the Wasabi bucket in the NextCloud External Storage app.

Yes, we’re using it as external storage, not as primary. Wasabi looks good, but unfortunately, it is in the U.S., which means we’re not allowed to store data there. What I like about S3 is the ease versioning and replication to another regions. But we do not have the absolute need to use a S3-like storage. We could also go with EBS volumes, closely monitor their usage and backup them.

At the moment, the API requests seem to be reduced drastically according to Cloudwatch. I’ll be monitoring them for the next few days and also have a look into the billing dashboard for the daily S3 costs.

Thanks for your suggestion! :slight_smile:

Wasabi has a region in Europe.
It is eu-central-1 located in Amsterdam.
We have been getting good transfer speeds to this region from both DigitalOcean’s Frankfurt and Amsterdam datacenter. I’d suggest checking them out to see if you can meet your GDPR requirements as long as the datacenter is located in Europe.

How did you manage to reduce your API requests, other than unchecking “Check for changes on direct access”? Some other S3 compatible providers throttle requests, so would love to know if you did anything special.

Why don’t you try this out and report your findings.

We’ve checked today the daily costs for S3 in the cost explorer. We’re now down to about 0,50 USD per day (before: 15-20,- USD / day) which is much more reasonable.
So @Paradox551, thank you very much for the solution! I owe you a beer or coffee!

Happy for you!

Thank you for this thread. @Paradox551 You have convinced me to check out wasabi. Doing the 30 day trial now.

Wow the Check for Changes is on by default, and the access to the options is hidden on my admin panel.

I didn’t know about this option until I woke up to a billing shock and coming to this post. Thank you @Paradox551 for pointing out.

I thought AWS S3 was cheaper than If I used elastic volume, now I realise attaching a elastic volume is WAY cheaper than AWS S3

1 Like