Encryption Options for S3 as Primary Storage

🍱 Features & apps
, , , ,
1

Hello NextCloud community. I’m new here - great app!

I have NextCloud 11 running on EC2. I’ve successfully configured S3 storage as primary, and things look good. I’m just hoping to clarify something that doesn’t seem too thoroughly documented.

According to the OwnCloud docs, using S3 as primary storage means I cannot use encryption:

The current implementation is incompatible with any app that uses direct file I/O and circumvents the ownCloud virtual filesystem. That includes Encryption and Gallery…

However, the similar page on NextCloud does not seem to list that restriction. Other searches don’t seem to uncover anything specific - but my apologies if I’m missing something obvious.

Can I assume that the OwnCloud restriction applies here as well? Also should I assume that leveraging the S3-side encryption provided by AWS is the only way to encrypt this data at rest?

Thank you very much - I’m super excited about being a new convert and running my own cloud.

2

With much tinkering, I see that I can indeed enable encryption globally as well as encrypt the existing files.

Interesting to note that S3 for primary storage seems to still need a bit for work, though. For example, the “Deleted Items” app leaves orphan objects behind in S3 when enabled, and nothing seems to clean them up.

Otherwise going well, thanks NextCloud team for a great product!

4

Did you manage to use S3 as primary storage AND encryption?
If so, what steps did you take?

It seems this issue has been fixed yet: https://github.com/nextcloud/server/issues/3748, https://github.com/nextcloud/server/issues/3294.

1 Like
5

They have been reported, not fixed yet. As it is already at github, it’s better to continue the discussion there with the developers.

6

I did get S3 primary storage working with the NextCloud UI reporting encryption was enabled.

There seemed to be a work/fail correlation between enabling encryption before switching the config.php file to S3 or after. Perhaps something with the keys? Sorry, I wish I could tell you. Unfortunately, I didn’t document each step and can’t tell you how to reproduce the working setup.

The state that I got it into is not what I would deem reliable. Things seemed to be a bit off. Delete didn’t act as expected, and I noticed unexpected objects appearing and disappearing at random.

Eventually, I ended up instead using s3fs to mount the S3 bucket as a folder in the file system. I much prefer this method. It stores the files in the buckets in a more standard fashion - using the actual file and folder name structure. Much cleaner should I decide I want to migrate to another solution in the future.

This method also allows me to choose Amazon’s SSE encryption, using either their keys or keys that I provide.

Until Nextcloud’s support and documentation for S3 as primary storage is much more mature, I see s3fs as the better choice.

Good luck!

----------

If you do end up using s3fs, I found putting this line in /etc/fstab works great with NextCloud. I use IAM roles, so if you do not then you will need to look at the s3fs docs to see what syntax you need for using your secret key.

s3fs#<my s3 bucket name> /mnt/s3/nextcloud fuse iam_role=<my iam role>,allow_other,use_sse,uid=apache,gid=apache,umask=007 0 0

Also I ran the following command before mounting the directory to ensure it would not be written to in the event that it was unmounted:

chattr +i /mnt/s3/nextcloud
7

I did something similar but I used goofys instead of s3fs because of its much superior file access performance. S3FS is POSIX-compliant, but it is unlikely that you need the full features of a POSIX filesystem.

However, any of these filesystem solutions are still not ideal because you are not leveraging S3 at all. To serve a file to a user it goes from S3 -> Your Server -> Client. Ideally, you’d want Nextcloud to serve the URL of a AWS API endpoint which is linked dynamically to your S3 bucket so Nextcloud only serves the link and the user downloads the file from AWS.

I’m currently working on a system to do just this, using the Collaborative Tag plugin to specify a tag that serves to identify a file so it is automatically offloaded onto S3 transparently.

It is trivial to wrap your S3 object into a lambda function that decrypts it when requested, as well, so you can still have the data encrypted at rest and not have to decrypt anything yourself. This way is in many ways superior, because you can have differing encryption keys based on the file metadata or tags, in order to minimize the exposure of one encryption key being compromised.

It’s not quite ready for prime time though :blush:

1 Like
8

Very cool project, I’m excited to hear more about it in the future. :smiley:

1 Like
9

But if you store the encrypted objects in S3, they still have to go through Nextcloud’s server in order to be decrypted and then served to the user.
Or am I missing something?

1 Like
10

Yes, that’s the point of server-side encryption that is has to pass the Nextcloud server where they keys for decryption are stored.

1 Like
11

You should invest more time into this project of yours.
For nextcloud is personal cloud but ends up expensive compared to any other solution but with your project it has a chance to become truly the personal cloud and compete on pricing too.

1 Like
12

Replying to an old thread but, i guess other complexity here is that to fully leverage your s3 access, you’d need to share it through something like CloudWatch with a Lambda Edge function, rather than direct lambda itself… and on-top of that, the link needs to know what file it is sharing and what URL it is safe to share with… otherwise you end-up making all your data public by which the lambda decrypted end-point becomes redundant.

But - very cool project nonetheless, i’d be interested to know where you’ve got with it as if you can crack those points above, this is the best solution!