How does external storage work (local caching, trash, storage space)?

🍱 Features & apps
,
1

I’m experimenting with adding an S3 compatible external storage bucket.

It all seems to be working ok, possibly a bit slow, but anyway, I was wondering how it works: where are the files stored? I note that, for example:

  1. accessing a file in a bucket via the web app, the network request goes to my NC instance (not directly to the bucket) - this makes sense to me because NC is sort of proxying permissions. But is the file then cached locally on the NC instance as well as now being in the S3 bucket?

  2. deleting the file in a bucket via the web app moves it to Trash… it’s gone from the bucket but present in NC’s trash, i.e. locally.

I’m interested in the bucket idea because of limited local storage. But it’s starting to look like the files will be in S3 bucket and locally, which is worse!

Can anyone explain how it works?

2 Likes
2

When using external storage, Nextcloud accesses files stored outside the system and presents them as a mounted folder. It’s actually not all that different conceptually from mounting a file share in Linux or Windows. The files are not copied or moved. Nextcloud accesses them in their original location on behalf of the user.

As far as S3, I don’t use it (to me, giving your data back to big tech defeats the purpose of self-hosting), so I can’t say specifically… however I’m curious as to whether the file is actually copied locally just to put in the server’s trash because my assumption would be that is not the case.

Can you delete an external storage file and then look in your data folder to see if it is indeed in the trash locally?

3

giving your data back to big tech

I hear you! Although Amazon created the S3 standard, it is implemented by many providers and open source projects. I am using it with s3 compatible storage provide by my VPS provider (which is not Amazon, as I believe companies should contribute by properly paying taxes…)

I did a test

$ du -hs *
96K	cache
7.9G	files
724M	files_trashbin
2.8G	files_versions
4.0K	uploads
$ # deleted 100MB of files in the s3 bucket via NC web UI
$ du -hs *
96K	cache
7.9G	files
827M	files_trashbin # LOOK! +100MB!
2.8G	files_versions
4.0K	uploads

that’s quite annoying!

2 Likes