gas85
December 17, 2021, 7:36am
7
I use restic to perform backups and my daily backup size is even smaller (about 500 GB) than whole data folder (about 700 GB), because of compression and hardlinks to the same and unchanged files. Check out, there few posts about it, e.g. this one Question for backup and restore - #3 by Reiner_Nippes
Also I find out that there is a lot of redundant data:
opened 09:47AM - 10 Feb 21 UTC
closed 12:02PM - 09 Aug 21 UTC
enhancement
0. Needs triage
Nice to have
<!--
Thanks for reporting issues back to Nextcloud!
Note: This is the **issu… e tracker of Nextcloud**, please do NOT use this to get answers to your questions or get help for fixing your installation. This is a place to report bugs to developers, after your server has been debugged. You can find help debugging your system on our home user forums: https://help.nextcloud.com or, if you use Nextcloud in a large organization, ask our engineers on https://portal.nextcloud.com. See also https://nextcloud.com/support for support options.
Nextcloud is an open source project backed by Nextcloud GmbH. Most of our volunteers are home users and thus primarily care about issues that affect home users. Our paid engineers prioritize issues of our customers. If you are neither a home user nor a customer, consider paying somebody to fix your issue, do it yourself or become a customer.
Guidelines for submitting issues:
* Please search the existing issues first, it's likely that your issue was already reported or even fixed.
- Go to https://github.com/nextcloud and type any word in the top search/command bar. You probably see something like "We couldn’t find any repositories matching ..." then click "Issues" in the left navigation.
- You can also filter by appending e. g. "state:open" to the search string.
- More info on search syntax within github: https://help.github.com/articles/searching-issues
* This repository https://github.com/nextcloud/server/issues is *only* for issues within the Nextcloud Server code. This also includes the apps: files, encryption, external storage, sharing, deleted files, versions, LDAP, and WebDAV Auth
* SECURITY: Report any potential security bug to us via our HackerOne page (https://hackerone.com/nextcloud) following our security policy (https://nextcloud.com/security/) instead of filing an issue in our bug tracker.
* The issues in other components should be reported in their respective repositories: You will find them in our GitHub Organization (https://github.com/nextcloud/)
-->
### How to use GitHub
* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are interested into the same feature.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.
**Is your feature request related to a problem? Please describe.**
When user copy file to different folder in local Storage, then it is copied and not hard linked, that will increase space consumption by the cloud Server.
Those are the same files, but different Inodes.
```
ls -lai Da*.txt Shared/Datenschutzerklärung.txt
134284604 -rw-r--r-- 1 www-data www-data 4783 Oct 7 2019 Datenschutzerklärung.txt
134352533 -rw-r--r-- 1 www-data www-data 4783 Feb 10 10:30 Shared/Datenschutzerklärung.txt
md5sum Da*.txt Shared/Datenschutzerklärung.txt
3712399a1ddddc19b088bd805407b361 Datenschutzerklärung.txt
3712399a1ddddc19b088bd805407b361 Shared/Datenschutzerklärung.txt
```
**Describe the solution you'd like**
- First try to create hardlink to the file in new location on local storage, only if it is fails then try to use copy command.
- Make Hardlinks creation optional, e.g. not relevant for S3 Storage on AWS.
**Describe alternatives you've considered**
Currently I use `rdfind -makehardlinks true` to find same files and reduce space usage. This will also use other users folder, e.g. common Family Videos from different users will be hardlinked.
**Additional context**
Similar behavior implemented e.g. in Sonarr. https://github.com/Sonarr/Sonarr/wiki/Completed-Download-Handling
You can setup if Hardlink should be used instead of copy and If the hardlink creation fails, Sonarr will fall back to the default behavior and copy the file.
Also related to #19948
And I can reduce used space with rdfind -makehardlinks true, or this script to exclude e.g. appdata_* from it:
#!/bin/bash
# By Georgiy Sitnikov.
#
# AS-IS without any warranty
NextCloudPath=/var/www/nextcloud
Command=/var/www/nextcloud/occ
# Do not touch (true) appdata_XXXXXX directory, e.g. previews and system/app files.
# If you set false, appdata and previews will be also evaluated
appdata=true
###
LOCKFILE=/tmp/nextcloud-hardlink-duplicates.tmp
CentralConfigFile="/etc/nextcloud-scripts-config.conf"
if [ -f "$CentralConfigFile" ]; then
show original