High AWS S3 costs due to Nextcloud requests

Scobo · January 20, 2020, 3:47pm

Hi!

We started running Nextcloud App as a prebuilt-docker image in Univention Corporate Server.
External (not primary) Storage for everything is AWS S3 and we have about 180 GB in our buckets. There are only less than 10 users, all using the Windows client or the web login. Files are not changed in the buckets externally, only by Nextcloud.

We received the AWS bill for last month and it says:
“Amazon Simple Storage Service EUC1-Requests-Tier1” ($0.0054 per 1,000 PUT, COPY, POST, or LIST requests): 48,217,810.000 Requests / 260.38 USD
“Amazon Simple Storage Service EUC1-Requests-Tier1” ($0.0043 per 1,000 PUT, COPY, POST, or LIST requests): 46,015,181.000 Requests / 19.79 USD

So we were paying about 280 USD for some requests and only a few bucks for the storage itself.

I suspect that there is a misconfiguration which leads to this huge amount of requests.

Any help or at least explanation for Nextcloud’s behaviour is appreciated.

Thanks in advance!

Nextcloud version 16.0.5-0
Operating system and version Univention (Debian) 4.4-3 errata423
Univentions currently supports updates up to Nextcloud 16.0.6-0

config.php:

<?php
$CONFIG = array (
  'passwordsalt' => 'foobarfoobarfoobarfoobarfoobar',
  'secret' => 'foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar',
  'trusted_domains' =>
  array (
    0 => 'nextcloud.int.example.com',
    1 => '172.16.30.21',
    2 => 'nextcloud.example.com',
  ),
  'datadirectory' => '/var/lib/univention-appcenter/apps/nextcloud/data/nextcloud-data',
  'dbtype' => 'pgsql',
  'version' => '16.0.5.1',
  'overwrite.cli.url' => 'https://nextcloud.int.example.com/nextcloud',
  'dbname' => 'nextcloud',
  'dbhost' => '172.17.42.1',
  'dbport' => '5432',
  'dbtableprefix' => 'oc_',
  'dbuser' => 'nextcloud',
  'dbpassword' => 'foobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobarfoobar',
  'installed' => true,
  'instanceid' => 'abcdefg12345',
  'updatechecker' => 'false',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'overwriteprotocol' => 'https',
  'overwritewbroot' => '/nextcloud',
  'htaccess.RewriteBase' => '/nextcloud',
  'ldapIgnoreNamingRules' => false,
  'ldapProviderFactory' => 'OCA\\User_LDAP\\LDAPProviderFactory',
  'trusted_proxies' =>
  array (
    0 => '172.17.42.1',
  ),
  'maintenance' => false,
  'loglevel' => 2,
  'mail_smtpmode' => 'smtp',
  'mail_smtpsecure' => 'ssl',
  'mail_sendmailmode' => 'smtp',
  'mail_from_address' => 'nextcloud',
  'mail_domain' => 'example.com',
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpauth' => 1,
  'mail_smtphost' => 'mail.example.com',
  'mail_smtpport' => '587',
  'mail_smtpname' => 'mailuser',
  'mail_smtppassword' => 'foobarfoobarfoobarfoobarfoobarfoobarfoobar',
);

Paradox551 · January 21, 2020, 2:57pm

Is check for changes on direct access enabled on the external storage? If so that’s your problem.

Otherwise 180GB of data is vague. 180GB could be millions of files.

Amazon S3 is expensive and there are cheaper alternatives out there.

Scobo · January 21, 2020, 3:47pm

Thank you very much for your quick answer, I appreciate that!

Yes, “check for changes on direct access” was set, now I turned it off.
Do I have to take care of something like re-indexing the S3 buckets periodically? Nextcloud is using it exclusively, there are no other accesses to the bucket.

180 GB is about 75,000 files in our case.

I will monitor the API calls over the next few days after changing the setting and then decide if we go back to EBS and snapshots.

Paradox551 · January 21, 2020, 4:45pm

“Check for changes on direct access” causes it to scan the entire directory and see if there are any new/changed files.

I don’t know how it handles the backend on S3 but knowing how S3 is not a priority for the developers it likely sends requests to every file. So if you have 30k files in the directory you will be using 30k API requests just opening the directory. Not sure if this translates to cron, either.

Nextcloud is the metadata server for S3. All uploads/downloads/changes/deletions/new etc are handled directly by nextcloud. It’s pointless to check the bucket for changes and a waste of money.

Hopefully that fixes it for you!

autoize · January 23, 2020, 3:13am

If you switch to an S3 compatible service like Wasabi, the API requests are not billable. It sounds like you are using S3 as an external storage mount (as opposed to primary storage) which means migrating should be fairly simple.

You could use a tool like rclone to copy everything over, then simply remount the Wasabi bucket in the NextCloud External Storage app.

Scobo · January 23, 2020, 11:09am

Yes, we’re using it as external storage, not as primary. Wasabi looks good, but unfortunately, it is in the U.S., which means we’re not allowed to store data there. What I like about S3 is the ease versioning and replication to another regions. But we do not have the absolute need to use a S3-like storage. We could also go with EBS volumes, closely monitor their usage and backup them.

At the moment, the API requests seem to be reduced drastically according to Cloudwatch. I’ll be monitoring them for the next few days and also have a look into the billing dashboard for the daily S3 costs.

Thanks for your suggestion!

autoize · January 23, 2020, 6:25pm

Wasabi has a region in Europe.
It is eu-central-1 located in Amsterdam.
We have been getting good transfer speeds to this region from both DigitalOcean’s Frankfurt and Amsterdam datacenter. I’d suggest checking them out to see if you can meet your GDPR requirements as long as the datacenter is located in Europe.

How did you manage to reduce your API requests, other than unchecking “Check for changes on direct access”? Some other S3 compatible providers throttle requests, so would love to know if you did anything special.

Paradox551 · January 23, 2020, 10:21pm

Why don’t you try this out and report your findings.

Scobo · January 29, 2020, 3:30pm

We’ve checked today the daily costs for S3 in the cost explorer. We’re now down to about 0,50 USD per day (before: 15-20,- USD / day) which is much more reasonable.
So @Paradox551, thank you very much for the solution! I owe you a beer or coffee!

Paradox551 · January 29, 2020, 4:01pm

Happy for you!

Medus · March 28, 2020, 3:20pm

Thank you for this thread. @Paradox551 You have convinced me to check out wasabi. Doing the 30 day trial now.

daveteu · August 12, 2020, 10:33am

Wow the Check for Changes is on by default, and the access to the options is hidden on my admin panel.

I didn’t know about this option until I woke up to a billing shock and coming to this post. Thank you @Paradox551 for pointing out.

I thought AWS S3 was cheaper than If I used elastic volume, now I realise attaching a elastic volume is WAY cheaper than AWS S3

Paul_Calabro · July 9, 2021, 6:49am

Wow, I’m glad this thread exists! I really couldn’t believe the bill that came in last month for S3 storage. I saw a huge spike in the number of “ListBucket” API requests. After making this change, the daily cost went down considerably. Thanks for pointing me in the right direction.

I’m curious what constitutes “direct access”. Does this mean every time I access that external storage, it does an entire scan of the S3 bucket? Or is it just scanning the directories being accessed?

rickyromero · January 22, 2024, 4:55am

Hello. Is there a config.php flag to disable “check for changes on direct access”? I’ve only found info about the GUI setting.