Is it possible to set the S3 bucket as the default storage?
I mean to have all the files in that storage instead of having it mounted as a folder.
Thanks!
1 Like
Hello,
I did this with S3QL.
It’s a Python application that will use your local disk as a buffer, and transparently sync all files to the S3 bucket. You just have to mount a FUSE partition and use it for your data folder.
Nice feature : data deduplication and easy snapshots !
Side effect : it’s a bit slow …
Edit: performances problem are maybe corrected with version 2.19…
I’ve been playing around with s3ql for part of my nextcloud (basically mounting somewhere under data/files), and it’s been going well so far for me. The performance seems reasonable,I guess a decent cache helps. My use case has a large amount of data sitting there, but a lot of it is rarely touched. I guess most of what I work with daily lives in cache 
I was thinking of making the switch, but am slightly worried about the setup. I think that I want s3ql properly fsck and mount on server boot, and cleanly flushing cache and unmounting on shutdown. and I think nextcloud should be in maintenance mode until the s3ql drive is ready.
By any chance, did anyone write systemd units, init files or the like to enable this kind of setup? Or am I just too paranoid to want it and it’s actually not worth bothering?
that just mandatory if you don’t want to loose everything after a shutdown ! You can find inspiration with the following, I recommend you to check all option
# Montage/Demontage du FS S3QL
# 1) copy this file in /etc/systemd/system
# 2) activate service : #systemctl enable s3ql.service
# 3) create folder /run/s3ql with read/write access for www-data
# 4) create folder /var/log/s3ql with read/write access for www-data
[Unit]
Description=S3QL file system
After=network-online.target
[Service]
User=www-data
Type=notify
Environment=AUTHFILE=/home/ubuntu/s3qlcredentials.txt
Environment=BUCKET=swiftks://auth.cloud.ovh.net/GRA1:Conteneur1
Environment=MOUNTPOINT=/var/s3ql
ExecStart=/usr/bin/mount.s3ql --compress lzma-6 --metadata-upload-interval 0 --cachedir /var/lib/s3ql --log /var/log/s3ql/s3ql.log --fg --authfile ${AUTHFILE} ${BUCKET} ${MOUNTPOINT}
ExecStop=/usr/bin/umount.s3ql ${MOUNTPOINT}
TimeoutSec=5min
[Install]
WantedBy=multi-user.target
1 Like
Guillaume, thank you for this. Have implemented this now and I am very happy. Performance is perfectly fine for my usage.
I have added a nextcloud unit as well, to make sure nextcloud is only up when the s3 file system is mounted:
[Unit]
Description=Nextcloud
After=s3ql.service
Requires=s3ql.service
[Service]
User=www-data
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/php /var/www/html/owncloud/occ maintenance:mode --off
ExecStop=/usr/bin/php /var/www/html/owncloud/occ maintenance:mode --on
TimeoutSec=42
[Install]
WantedBy=multi-user.target
1 Like
I apologize for bringing up a very old topic, but it contains useful information.
Nobody has encountered an error of creation permission of file .s3ql from /var/www/?
How exactly did you set the permission to s3ql? or only for the mount directory?