Backup files on Nextcloud-managed object storage to local storage

The Basics

  • Nextcloud version:
    • Nextcloud AIO v9.8.0
  • Operating system and version:
    • Debian 12
  • Installation method
    • AIO

The Storage Model

First, I set up a Nextcloud AIO instance to run on a VPS.

Then, I configured Nextcloud to use IDrive E2, an S3-compatible object storage, as my Nextcloud’s primary storage, which stores the data on a bucket called nextcloud-main. I followed this documentation page to set it up.

For backups, I created another bucket called nextcloud-backup, which is mounted with s3fs on /mnt/idrivee2/nextcloud-backup on system boot. I have configured the Nextcloud AIO instance to make backups on that mountpoint. Nextcloud uses BorgBackup for backups, and after one backup I made, Nextcloud created a directory borg inside that mountpoint to store that backup.

Finally, I enabled server-side encryption.

It’s all working fine.

The Problem

I am setting up Nextcloud to store data from my family members, who live in different locations than mine. The idea is making each of them have their own Nextcloud account on my self-hosted instance and have their client apps sync their personal data to it.

I would also like to make, from time to time, a local backup of their Nextcloud-stored data onto an external physical drive of mine, such as an external SSD (or a regular SSD with a SATA-to-USB adapter, as I prefer). This is in case something goes wrong with Nextcloud and I can’t for the life of me restore a Nextcloud backup, in which case I will at least have access to the files. As far as security goes, losing this physically-accessible drive is not a concern, as data on it would be encrypted with VeraCrypt.

The problem is that, as the aforementioned documentation page says, when using an object storage as Nextcloud’s primary storage, all file metadata is stored inside of Nextcloud, so there is no way of bypassing Nextcloud by accessing the object storage directly.

My Findings

I did some reading on BorgBackup and I figured out I can access a Nextcloud backup’s contents. The procedure, according to my storage model, is as follows:

1. Create two mountpoints, one for the Nextcloud backup bucket and another for the decrypted Borg repository:

sudo mkdir -p /mnt/idrivee2/{nextcloud-backup,nextcloud-backup-borg}

2. Mount the bucket nextcloud-backup with s3fs on my local computer. For this, I have added an entry on /etc/fstab, as there are too many options to use:

# /etc/fstab
# ...

s3fs#nextcloud-backup /mnt/idrivee2/nextcloud-backup fuse rw,_netdev,allow_other,passwd_file=/etc/passwd-s3fs/idrive/nextcloud-backup.rwak,url=<idrive-e2-endpoint>,use_path_request_style,use_cache=/tmp/s3fs,enable_noobj_cache,no_check_certificate,use_xattr,complement_stat 0 0

replacing <idrive-e2-endpoint> with the bucket’s endpoint URL.

3. Create a passwords file for a read-write access key to my nextcloud-backup bucket and change its permissions accordingly:

sudo mkdir -p /etc/passwd-s3fs/idrive/
sudo nano /etc/passwd-s3fs/idrive/nextcloud-backup.rwak # <access-key-id>:<access-key-secret>
sudo chmod 600 /etc/passwd-s3fs/idrive/nextcloud-backup.rwak

4. Reload /etc/fstab and mount it:

sudo systemctl daemon-reload
sudo mount -a

5. Once mounted, there will be borg directory inside that mountpoint. Mount it with the command:

borg mount /mnt/idrivee2/nextcloud-backup/borg /mnt/idrivee2/nextcloud-backup-borg

I am prompted with the BorgBackup encryption key. I enter the encryption key I was asked to store at a safe place when I enabled the Nextcloud server-side encryption feature.

6. List the contents of the mounted borg repository to see a directory 20241110_160440-nextcloud-aio:

ls /mnt/idrivee2/nextcloud-backup-borg

Navigating the contents of that directory, I see the following structure:

  • /mnt/idrivee2/nextcloud-backup-borg
    • 20241110_160440-nextcloud-aio
      • nextcloud_aio_volumes
        • nextcloud_aio_apache
        • nextcloud_aio_database
        • nextcloud_aio_database_dump
        • nextcloud_aio_elasticsearch
        • nextcloud_aio_mastercontainer
        • nextcloud_aio_nextcloud
        • nextcloud_aio_nextcloud_data

Looking around for meaningful files, I found out there is a file database-dump.sql in nextcloud_aio_database_dump that I thought might have mappings of urn:oid files (which are stored on the bucket nextcloud-main) to the actual file paths. However, that would be only one part of the challenge, since the file contents themselves are also encrypted.

So, before I start losing hairs digging deeper into this, I wonder, is there an easier of doing what I want? Or is this the way to go – reading borg backup files to map urn:oid names to actual names and, somehow, decrypt those files manually? If the latter, how can I do that?

Any help is appreciated! :smiley:

About Object Storage as External Storage

One may ask, “Why not simply use your IDrive bucket as a Nextcloud external storage, instead of primary storage?” I tried, and it’s slow to an unusable degree. It has to be primary storage.

1 Like

I understand your situation, but doesn’t this approach seem unnecessarily complicated? Based on my personal experience, here’s a simpler and effective alternative:

My Experience with Nextcloud AIO Migration

When I migrated a Nextcloud AIO instance to another server, I followed a straightforward process using rsync:

  1. Backup: I used rsync to back up the entire Nextcloud AIO Docker volume from the original server. This ensured all data, permissions, and ownership were preserved.
  2. Restore: I transferred the backup to the new server using rsync again, ensuring data integrity and consistency.
  3. Result: Everything worked seamlessly without even needing a rescan.

This method was quick, simple, and reliable.

A Potential Solution for Your Case

If your goal is to create backups, here’s how rsync can help:

  • Option 1: Use rsync to back up the entire Nextcloud AIO Docker volume. This will include all necessary data and configurations, making restoration straightforward.
  • Option 2: If you only need backups for specific users, use rsync to selectively back up data associated with their accounts.

Important Note on Filesystems

When using rsync, it’s crucial to use filesystems on your disks that support preserving permissions and ownership. Recommended options include:

  • xfs – Highly efficient and reliable for large datasets.
  • ext4 – Widely supported and sufficient for most cases.

Avoid using filesystems like:

  • NTFS – While cross-platform compatible, it doesn’t preserve permissions and ownership effectively, making it unsuitable for precise backups.
  • FAT32 or exFAT – Lack support for permissions and ownership entirely.

Choosing a proper filesystem is critical to ensure backups remain consistent and restorations are hassle-free.

Benefits of Using rsync

  • Efficiently copies only the changes, saving time and bandwidth.
  • Preserves file structure, permissions, and ownership (on supported filesystems).
  • Avoids the need for manual mapping or decrypting files.
  • Flexible—can be used for full or partial backups depending on your needs.

If you’d like, I can share more details about how to set up and use rsync with the recommended filesystems. Let me know! :blush:

1 Like

Nextcloud data in the cloud but additional backup on-prem? Seems not to be a good idea. Have you understand cloud?

1 Like

Your method seems indeed simpler, but this way, you are keeping your data around a Nextcloud instance. My concern is what if, for some reason, I can’t set up a Nextcloud instance or, let’s say, something goes wrong with it (a bug or a wrong setup). It’s a software, after all. Years of personal data would be inaccessible, at best, or lost/corrupted, at worst.

As a Plan C, I would like to be able to access the files just like one does with a USB stick. No Nextcloud; just a simple filesystem. You see?

Hey, just FYI: remote backups are now natively possible with v10.0.0 beta. Testing and feedback is welcome! See GitHub - nextcloud/all-in-one: 📦 The official Nextcloud installation method. Provides easy deployment and maintenance with most features included in this one Nextcloud instance.

1 Like

Yes, that’s the idea.

The way I’m using Nextcloud is with a VPS, which may become inaccessible to me at any time, since it’s not something under my control. It’s unlikely, sure, but I shouldn’t rely on it.

Some people self-host Nextcloud at their house. That works, if the only people who are using the services are on LAN. I’ve chosen to use a VPS for a greater availability, since I don’t keep my computer turned on all the time and I don’t rely on my ISP’s availability.

Anyway, before using Nextcloud, that’s the way I had been managing my family’s data: each of them had a folder around a VeraCrypt-encrypted container, and from time to time, I would personally back up their data to it. But I can’t do that anymore.

If you can elaborate on why my strategy of dual backup (on the cloud and on-premises in a physical device) is bad, I may reconsider it.

It seems you may have misunderstood what I was trying to explain. Let me clarify:

  1. No Data Syncing Within the Server Itself
    What I’m proposing is not about syncing data directly within the server where Nextcloud is hosted. I also do not rely on such an approach myself.

  2. Using rsync for Backup
    You can easily back up the entire Docker volume used by Nextcloud AIO from your server to your local device using rsync. This provides a comprehensive backup of all your data.

  3. Accessible Data Without Nextcloud
    By backing up the Docker volume, your data is stored in a simple filesystem format on your local device. This ensures your data remains accessible even if the Nextcloud instance is unavailable or encounters an issue. Essentially, your local backup functions like having your files on a USB drive – no dependency on Nextcloud.

I believe this approach addresses your concerns about potential issues with Nextcloud while giving you reliable access to your data. Let me know if you have any further questions or need guidance on setting up rsync for this purpose!

Hey @vawaver I would be interested why you prefer manual backups via rsync over AIOs backup solution?

Interesting. This link you shared points to the section on how to switch between AIO latest and AIO beta. Is there any documentation on how to use this remote backup feature?

There are some hints regarding that in this docu: GitHub - nextcloud/all-in-one: 📦 The official Nextcloud installation method. Provides easy deployment and maintenance with most features included in this one Nextcloud instance.
(See the If you want to back up directly to a remote borg repository: detail)

1 Like

I think I understand you now. You’re saying that backing up the whole AIO container is better than using that approach where I mount the borg backup directory, right?

I could do what you suggest, but how will I access my personal data, considering I’m using object storage as primary storage and the server is encrypted?

Maybe I’m wrong, but from what I understand, I will still have to do the urn:oid mapping and decrypt the files manually, yes?

Hi!
The reason I prefer manual backups with rsync over the integrated AIO backup solution (like Borg backup) is primarily due to the flexibility and control this approach provides me.

  1. Independence from the Platform
    rsync allows me to back up data directly from the server to my local device in a simple, readable format that I can immediately inspect or restore without requiring special software or tools. I am not dependent on the functionality of AIO or its specific backup formats.

  2. Simplicity and Versatility
    rsync is a straightforward tool that effectively synchronizes large amounts of data. I don’t have to rely on AIO being correctly configured or worry about potential issues during the backup process.

  3. Speed for Incremental Backups
    With repeated backups, rsync compares files and synchronizes only the ones that have changed. This means it doesn’t create a complete backup of all data every time, which is particularly useful for large Nextcloud instances where data can reach hundreds of gigabytes. As a result, backups are significantly faster.

  4. Low Resource Usage
    rsync is highly efficient in terms of system resource consumption, such as CPU and memory. Compared to some more complex backup solutions, it doesn’t put a heavy load on the server and can even run on less powerful devices.

  5. Complete Data Backup
    By manually backing up the entire Docker volume, I can preserve not only user data but also the application’s settings and configuration files. This ensures I can restore the entire application to its original state if needed.

  6. Compatibility with Other Tools
    The data backed up using rsync can be easily used with other backup systems or copied to different types of storage, such as NAS, USB drives, or cloud services.

That being said, I’m not claiming that Borg backup or AIO’s integrated solutions are bad options—they are certainly convenient for many users. However, for my needs, rsync provides greater reliability, flexibility, speed, and a lighter load on the system.

1 Like

All of that are also advantages of the integrated backup solution by using borg - so I disagree here that rsync is better in these points. It has an additional advantage that it stores additional snapshots that the system can be restored to and not only one while not using much more storage since snapshots are deduplicated. Also backups are compressed and encrypted so that they can be stored anywhere. Additonally all of that can be easily managed via the AIO interface. Even the whole AIO instance can be restored with basically one click from such an archive.

But of course feel free to do what you think makes sense most for you :wink:

1 Like

I understand that your situation is a bit different, especially if you are using object storage as primary storage and your server is encrypted. However, I want to emphasize that in my approach, I assume full access to the server via SSH, meaning I can directly see all files and directories on the server without needing encryption.

  1. Access via SSH
    Having SSH access to the server allows me to work directly with files at the filesystem level without additional steps like decryption or handling “urn:oid” mapping. This eliminates the complexities that might arise when dealing with encrypted data.

  2. No Need for Encrypted Data on the Server
    My server is not configured to encrypt data, which means the files are stored in a straightforward, readable format. This setup is practical for me because it enables quick and direct backups or restores without any extra steps.

  3. Efficient Backup Management
    Manually backing up the entire Docker volume of the AIO container provides a complete copy of all data, including configurations. This approach allows me to quickly restore both the data and the application if necessary, without dealing with encryption-related complications.

I believe this approach is practical, especially if encryption is not mandatory for your use case. Additionally, having direct SSH access makes managing files on the server much simpler.

I appreciate your perspective. Let me clarify my approach further:

  1. Intent of My Suggestion
    My goal was not to argue that my approach is better or that Borg backup is inferior. Instead, I aimed to highlight a method that aligns with my specific needs and priorities.

  2. Focus on Simplicity
    My primary priority is simplicity. I wanted a straightforward way to repeatedly back up data from a Nextcloud server to local storage without relying on specialized tools.

    • Using rsync, I can quickly and efficiently synchronize data directly at the filesystem level.
    • The process only involves changed files, making backups faster and less resource-intensive.
    • The data remains directly accessible without requiring additional steps for restoration.
  3. Acknowledgement of Borg Backup’s Advantages
    I fully recognize the strengths of Borg backup, particularly:

    • Deduplication: Efficient use of storage space.
    • Encryption: Secure storage of backups in untrusted environments.
    • Snapshots: The ability to restore data to multiple points in time.
    • Ease of Use: Management through the AIO interface, including one-click restoration.
  4. Why I Chose My Approach
    While Borg backup offers impressive features, my needs are centered around a simple, direct, and easily accessible backup process. For my specific use case, rsync meets these requirements effectively.

Both approaches have their merits, and the choice depends on individual priorities and circumstances. For me, rsync aligns better with my current needs. :blush:

1 Like

I see. It seems your storage model is based on storing data to a block device or an internal drive? Or are you using external storage?

I’m running Nextcloud on a VPS, and block storage is too expensive for my budget. And as you know, cheap VPS doesn’t come with large storage capacity. So, for a financial reason, I have chosen to keep my Nextcloud data stored on a separate location/endpoint, at IDrive.

For that reason, it makes sense to me to keep data encrypted as an increased security measure: if IDrive buckets are leaked, my personal data will be unreadable by malicious parties. Of course, I could just use Nextcloud without server-side encryption, but if the feature is there and is useful for me, not using it would be a waste.

My challenge is how to do this urn:oid mapping and decrypt the data. It seems there is no tool for that available (yet). I have no problem with writing a script for that. If set properly, all it would take to make a local backup would be running one single command. I just need to be pointed at the right direction on how to.

I read this out. This remote backup option essentially backs up the AIO to a remote Borg repository, correct? If that’s what it does, it means using it won’t solve my case. I will still be unable to access the contents of user files.

Yes, that is the idea.

No, you should be able to access the content of the remote borg repo by following GitHub - nextcloud/all-in-one: 📦 The official Nextcloud installation method. Provides easy deployment and maintenance with most features included in this one Nextcloud instance.

Reading the documentation you pointed me to, it seems that the remote backup approach will store the same content that the regular backup approach does. Quoting it:

Backed up will get all important data of your Nextcloud AIO instance like the database, your files and configuration files of the mastercontainer and else.

And then, later in the section “Delete backup archives manually”:

# List all archives (if you are using the default backup location /mnt/backup/borg)
sudo borg list

# After entering your repository key successfully, you should now see a list of all backup archives
# An example backup archive might be called 20220223_174237-nextcloud-aio
# Then you can simply delete the archive with:
sudo borg delete --stats --progress "::20220223_174237-nextcloud-aio

This will result in the same directory structure that I mentioned in my first post:

  • nextcloud_aio_apache
  • nextcloud_aio_database
  • nextcloud_aio_database_dump
  • nextcloud_aio_elasticsearch
  • nextcloud_aio_mastercontainer
  • nextcloud_aio_nextcloud
  • nextcloud_aio_nextcloud_data

Correct?

I made a graph illustrating my storage model. Maybe it helps to clarify: