Support for disaster recovery from S3 storage

Presently, S3 storage is offered as a backend for auxiliary storage.

Some users have requested support for S3 as a primary storage system. To date, no plans have been given to adopt such support.

Nevertheless, the request is germane.

S3 storage offers various advantages over storage as files on the local file system, in terms of scalability and reliability. As use cases are evolving, such advantages are becoming more manifest.

S3 is based on data and access models that support site replication and migration much more naturally and robustly than file systems. Of particular relevance is the flexibility to separate application server deployments from the backend file storage.

For example, through such a model, migration to new a application hosting solution is possible without disruption of file storage or transfer or file content.

As upward scalability of deployments is one consideration, a further development is the downward scalability of S3 storage solutions, many light enough now to operate on standalone appliances such as ones that have historically been targeted for Nextcloud.

One serious limitation is that at present, the S3 storage system relies on the local relational database as a single point of failure for representing the file tree, including file names and their mappings to payload objects in the S3 bucket.

Thus, the resilience of S3 storage for file payloads is offset by any vulnerability in the application and especially database deployment. Database backup is currently essential for disaster recovery, but still extremely clumsy as a solution for extracting file trees or file contents directly from S3 storage.

However, such vulnerabilities may be largely mitigated by certain enhancements that require no redesign of the overall system. An S3 bucket may easily store data objects that include at least recent if not fully current representations of the file tree data. By replicating such a representation, as in a serialized form, completely or incrementally, at periodic intervals, in the S3 backend, the application easily may preserve all data required for reconstruction of the entire tree of stored files. If the object data is structured transparently, then recovery by standalone utilities is feasible, in case administrators need to extract file contents without reconstituting a full deployment of the application. An object referenced by a single prescribed key may serve as a manifest for the contents of the entire bucket.

The solution opens opportunities for further enhancements, such as complete replication of all site data into an S3 store for purposes of disaster recovery, as well as seamless site migration.

Discussions of general and long-term improvements submitted to the issue tracker are often closed. Hopefully such a discussion will be well received in the support forum.

My 50 cents on this topic:

  1. S3 as primary storage is possible, and if you use Nextcloud with very few users, and strictly for file sharing, you could use SQlite, and then, at least in theory, its data could live completely on S3 storage. However, if you would ask me whether I think it would be a good idea to even try that, I’d’ say: Hell no! :wink:

  2. What you’re asking for would require something completly new, designed and built from the ground up, and I could be wrong, but I don’t think it’s realistic for the Nextcloud GmbH to do that, at least not in the foreseeable future.

  3. Maybe you can take a look at this… Infinite Scale 3.0 - the new cloud-native data platform

S3 as primary storage is possible but not supported, though many have requested more official support.

I am augmenting the discussion by requesting that additional metadata be replicated on the storage layer for purposes at least of disaster recovery.

I think the concept is not one that requires “something completely new, designed and built from the ground up”.