CDN Idea for File Distribution

I have an idea for an extension/app, which could help offload server bandwidth to a CDN which consists of multiple VPS servers, serving not static Nextcloud files, but actual files stored in Nextcloud.

I have a Nextcloud instance at home, but my home internet connection only has a 5Mbps uplink, and accessing anything remotely is a pain. For privacy and data custody reasons, I don’t want to store all my data on someone else’s server, i.e. VPS. This could also work with people with faster connections with a lot of people accessing large files at once.

I just want to know whether this idea has been tried before, and if not, whether there are any reasons it hasn’t.

Possible Implementation

  • Master server (home/slow) mirrors all files to a CDN server in the background, slave servers (CDN) syncs amongst themselves.
    • Master server prioritizes foreground downloads and pauses mirroring when files are being accessed over WAN.
    • Slave servers can use disk or file encryption, to prevent VPS provider from potentially accessing the files.
  • When you’re accessing from your “home”, i.e. the same IP as the server, the server does not rewrite the URL (or rewrites it to the local ip), so you can access the file over LAN.
  • When you’re accessing remotely:
    1. Master server first checks if CDN server has the latest copy of the file, if not, treat as cache miss and download (slowly) from master server directly.
    2. If cache hit, send an authentication token to the CDN server, if it matches your browser, the file is downloaded from the CDN server.
  • Possible support for deleting rarely accessed files if CDN storage server is full.
  • Possible multiple server support with load balancing and GeoIP based url rewrites.

Basically there are multiple ways to do so, but with defaults, CDN should only delivers static data, not your files. If you cache some PDF, than everybody can get it from the cache if know link. There is no authorization mechanism for cache. Thats why CDN will only help you to deliver cached JS, PNGs, JPGs (I mean Menu pictures, not user private one)… Check this out: NextCloud + CDN - #7 by gas85

What you describe is kind of load balancing and sure you can use it differently based on you Location/IP. It is a bit more complicated, but fully secured. Check this out: Mirroring nextcloud and this Nextcloud Global Scale: local data and limitless scalability at commodity cost - Nextcloud