Thanks for that, that is what I have been using with NC13 but it failed too. I just reinstalled it; this time using external storage… giving it another try
But looking at your comment and stratacast it looks like I am not the only one and I am guessing a lot of people are just giving up entirely. If Im right NC could be missing the biggest opportunity of all: the average user.
Not sure if I’ll be heard but I really think NC needs to refocus to become user friendly and more importantly nuclear proof for file syncing of all files including .htaccess because as a developper I need it for my projects
Installing and configuring a webserver, database and so on is a bit of a challenge but with ready-to-go images, this is quite easy. If it should be available outside your local network, it’s a bit harder. If that should work all the time, you would have to go through vpns and proxies, but that will cost something.
The installation process already improved a lot, there are still a few cases where it can be difficult and users don’t get clear messages. That must improve but this depends as well on the users to report such issues.
For many small files, the performance is not great and you can get the sync process into trouble. I have as well seen a few problems with git, I often stop the sync process for these folders and only sync once it is done. You could probably write an app for git, but on the other hand there are already many git clients, you can easily sync git stuff through ssh, so why bother improving this on Nextcloud?
Encryption is for external storage. I had many problems with it (back in ownCloud times), and in one version all problems were related to this encryption app. Since it is only for external storage and the benefits on local storage is very limited, I don’t use it any more. Since then, everything is more stable, less problems. Sync errors are rare, a few conflict-files from time to time.
The general problem with Raspberry Pi is that the resources are rather limited so you do need to do some optimization which depends on your use case. On larger computers, it is quite easy to give enough resources to each process. And if it is only plain file syncing, specialized software for that is probably more powerful (git-annex, syncthing), …
Since every setup and every use case is very different, you could build up a certain test case that can be reproduced, e.g. take a raspberry pi with nextcloud pi and say you are uploading a set of known files (either generate files of defined size containing random content or a known archive) and then test the performance. The community can try to find good parameters and if there are reproducible bugs, they can be reported to the bug tracker.
@tflidd yes that makes sense, every software needs optimisation and true that the raspberryPi is limited but I am using an RP3 which I think we can agree on the fact that it should be more than enough for a dedicated CLI Debian and syncing only ? I am using an optimised distro already and I still have the same errors.
Before using RPI I also tried dedicated VM on desktops. same issues
Having used Owncloud in the past; I left it for the exact same reasons and when NC came I was so happy thinking that bugs will be cleared but as far as I can see the engine is the same with a few optimisations maybe but the core of it remains with same bugs.
OC was filled with file lock errors and encryption errors too. NC was a hard fork so that makes sense to me that they share the same issues but I thought the NC community would work on the core functionalities/stability but instead we saw tons of additional addons.
Maybe classic relational databases are not fit for this kind of application ? Maybe a NoSQL database should be used ? I mean I have tons of files; we are going to see the limits of MySQL fast as far as I understand DBs and I only have 9Gb of data but dev data is full of small files so the count goes up fast.
I see NC 14 announcing “structural changes”; not sure what that means but I hope that they go back to basics for some time.
when you say
Does that mean that NC is ok not having a great syncing experience ? I thought that file syncing was the core of NC business ? Is NC focused on being average at everything (news, chat etc…) instead of being excellent in its core business ? NC moto being to take control of our data; I don’t feel like that atm; wish I could
1 GB RAM is not very much, giving more to a database improves a lot the overall performance. Also network and external storage speed are not that great. But I would agree to say for a single user, it should be ok and at least not produce tons of errors (just large number of files will take some time to sync).
In your case it would be interesting to know, what happens when you start to see more and more errors.
File lock can be handled with Redis, encryption takes some performance and there are some additional bugs. I don’t use it, and at the moment I wouldn’t and rather wait for client-side encryption (or use different client-side solutions).
Not sure about the design, I’m not an DB expert to be able to judge. At least they got the locking stuff to redis which already helped a lot. If you have a bit of RAM and can be a bit generous about the database cache, the performance is not so bad.
No, there shouldn’t be issues with the syncing at all. But it’s not all about syncing, there is also collaboration, that you can share files, have a nice web-interface to do so, that you are very flexible regarding different storage types, … If you only take the pure sync performance, the fact that everything is based on php makes it unlikely that it can outperform sftp or other basic protocols. But worst case it should take a bit longer, there are no excuses for errors.
The fork was focussed on the server-side software and the mobile clients. The desktop clients were just taken from ownCloud and the development stalled a bit. Now with client-side encryption, they start again the client development…
(owncloud has worked on a few nice features like virtual file system and delta-sync which are expected in their next version, hopefully this works and gets backported to nextcloud as well).
Just a few hints on the information you gave us so far:
File locking problems: You should check if you have transactional locking enabled and your Redis setup is working.
Server side encryption: it makes no sense if you don’t use some 3rd party storage you can’t trust. Using it only locally creates just more problems and solves nothing really. An attacker for example could take over your server and get your password anyway. Only client side encryption (E2EE) keeps your data safe in case the server gets compromised.
A Raspberry PI is nice for a lot of things but not for storage applications. A lot of small files might be just a little bit too much for it anyway.
redis is working. As a single user can I disable file locking ?
good point but in E2EE the files are stored unencrypted then ? What would be the point of having server side encryption then ? why have both ?
I have my storage at home; let’s say a burglar comes and takes the HD; he would have access to my files without server side encryption. Would E2EE encrypt files on the server ?
I do not agree with that; while it true than an RP2 is really underpowered, an RP3 should be enough; also NC should not be subject to “small file issues”. If speed is an issue, buffers or cache or writing queues could be implemented to avoid errors.
agreed, php design is not a good pick for this kind of application but a backend alternative could be interfaced with the site. the syncing algorithm does not have to stay PHP based
If I remember well this feature is still “experimental” but I guess it can’t be worst than the current server side encryption; maybe i’ll give it a try in my next install.
Client-side encryption encrypts files before they are sent to the server, so the server only sees encrypted files. If it is just the burglar, you could use a file system encryption (luks, enc2fs, …), it’s probably much more tested.
You could, but is your Redis and transactional file locking really working? Do you have a oc_file_locks table in your database? If yes and it gets refilled if you clear the table by hand, then transactional file locking doesn’t work.
Because they are designed for different use cases. Server side encryption is useful if you have a remote storage mounted into your Nextcloud (Webdav, SFTP, SMB, Amazon S3, Openstack and so on) you can’t trust. Your server sends everything to this storage encrypted. The external storage operator only sees encrypted data, but your Nextcloud server can decrypt it. It doesn’t help you if somebody breaks into your Nextcloud server because they could see the data unencrypted. If you don’t need it, you should turn it off. It causes more problems than it solves, is incompatible with a lot of apps and makes backups and disaster recovery a nightmare.
E2EE/Client side encryption on the other hand sends all the data from your local client device to the Nextcloud server in an encrypted fashion. The server operator of your Nextcloud only sees encrypted data. An attacker that compromised your Nextcloud server also doesn’t see anything else than encrypted data.
Maybe an RPi3 is enough regarding the processing power, but it is still limited because of its circuit design. Everything is attached to a single USB 2.0 hub, so Ethernet and a USB disk can max out the throughput of the system.
Nextcloud also doesn’t have “small file issues” in general. I can sync 10000 small files with a total size 40MB in a few minutes. No errors, hiccups or other failures. But there is surely a lot of room for improvement in a lot of things.
I had a lot of issues with server side encryption. Turn it off, your experience will improve quite a lot. It had a really bad start in the old bad Owncloud times because they changed encryption two times with two releases. This caused some massive problems including data loss.
is there any info anywhere on how to do this with nextcloud ? I assume that if I encrypt the disk directly I need to decrypt for NC so we have the same issue where the decryption key is stored on the server ?
E2EE will be great… when it will be in production state… I remember NC advertising it; not mentioning the alpha state and disclaimer not to use it in production… So in my case unusable. Or have I missed something ?
@alfred @tflidd
just a thought: with E2EE since the server only sees encrypted data then there should be no file restriction from the server ? just need to disable it on the client side ?
no and no
It was an NC bug; I’m guessing most likely coming from the encryption module or could have been a DB corruption too but we will never know; I have since reinstalled everything.
Since Serverside encryption is bashed here again, I can’t keep silent.
E2EE is great, hope it is ready soon. But there is a reason Nextcloud limited it to folders. Sharing is tricky at best and downloading stuff from anywhere via Website is impossible as the server can not decrypt it for you.
Benefits from Server encryption that I would not like to miss and still hope it is extended instead of being removed in the future:
Users that don’t use it anymore/don’t log in anymore can by this virtually instantly delete all files.
Users also do not have to worry about backups laying around after deleting an account.
If someone attacks the server and manages to get a system dump for offline investigation. He can not decrypt the files, unless he risks to attack the server again and do some sophisticated Nextcloud specific targeted attacks.
BTW. I was running Nextcloud just fine on a Pi2 with multiple users and encryption, a bit slow though, and am now running it perfectly fine on a Pi3 (I use OpenSuse Leap 42 to make full use of it’s 64 bit architecture and allow files > 4Gb as my Family loves to share videos). Last issues of your kind I had on our companys installation back when it was still owncloud. The encrypted flag was sometimes missing the database. But since we switched to Nextcloud these issues are gone
Can you share your config / optimisation details (especially database, web server and nextcloud’s config.php) anything in particular ?
Any issues with file lock or file not syncing for no reason ? did you change restricted files ?
Are you currently using server encryption ( home folder too ?) ?
Any chance you can develop on the OpenSuse / Debian choice for the “full 64 bit” ?
since it is just jebrish to the server should not bring any issues