Why is this still happening to me? (NC needs Focus)

🏷️ General
1

Hello,

I would like to talk about my nextcloud experience and before you read further I want to say that I understand it is not easy to develop something (I am a dev too) and I appreciate all the work from the community; I hope to be able to contribute one day too.

I have tried nextcloud 9, 10, 11, 12 and now 13 (always debian + apache) looking for a simple open source file syncing solution but every time it is the same: I install nextcloud; it works just fine for about a week to a month then bugs start to appear, files are not syncing, logs are full of webdav errors, etc… and then I stop using it and wait for the next version

As a single user of my private cloud at the moment I am having encryption errors, a single file “.gitignore” is not syncing anymore, files locked, operations cancelled, connection closed, my client is stuck with errors and I don’t know what to do.

I read the forum, try to post, try to find solutions by myself but I am not able to debug it.

Looking at nextcloud’s development history it seems that a lot of focus is on “external features” but we don’t seem to have a simple bomb proof syncing feature yet ? we can’t even sync .htaccess files and a bunch of others too. I mean all other solutions do it right out of the box… How a file syncing solution is ok with the fact of not being able to sync some file types ? As suggested in the forum those files could be renamed on the fly…

Let me be clear on one thing; I just need the file syncing feature; everything else is unnecessary to me. I don’t understand why I need calendars, chats, news feeds… Really ? It seems to me that dev time should be focused on the core application…

I see the German government using nextcloud so what am I missing here ?? Is this mature enough for government use ? If so it should be ok for me and my very simple syncing need ?

Im very sorry to say but in my humble opinion something is wrong with the core backend and as a result NC does not work reliably yet

what is your nextcloud experience ?

ps: tried to update to 14 beta 1 and the updater failed…
ps2 : I disabled my NC13 server; unworkable at the moment; will try a fresh NC 14 later

1 Like
2

Sorry to hear you’re having troubles. My first impression is you have some configuration errors, because I’ve had my instance running since version 9, and it has even hopped from Raspbian to Ubuntu 16.04 to FreeBSD with no fail. I have a 2 year old copy on another Raspbian server and Ubuntu 16.04 server as well for others. It is a rock solid platform. I would check your configurations and hardware, OS, and make sure it’s all correct.

You don’t but maybe others like me do want these? Well, maybe not news feeds, but others might. Some of these packages you find are third-party devved, and not done by the Nextcloud team. In fact, you could make your own Nextcloud app if you wanted to. Passman is an example of something that isn’t made by the Nextcloud team, but for Nextcloud.

I genuinely believe you’re missing something in your setup. Not to knock on your skill (since I don’t know your skill level), but for the most part, Nextcloud can be pretty bomb-proof. As for your needs though, you make it sound like you really don’t need the full package that Nextcloud offers, and that’s great, we all have our needs. If you just need simple file syncing, I highly recommend syncthing. It only does file syncing, and it does a great job at it. In fact, I wish Nextcloud would model syncthing for its syncing abilities.

3

I definitely have something wrong with my setup but here is what I am doing:

  • install NC following the documentation
  • enable ssl
  • solve any warning showing up in the admin page
  • sync files

So basically Im using something kind of “out of the box”. I am a fairly advanced unix enthusiast (I administer my own web servers since many years) but as far as NC architecture is concerned Im just using their tools.

On my latest install of NC13 I used the NextcloudPi distribution which has a really nice web management interface (really wish NC was doing that too) but even this distro has failed… Im lost for words; I really do not understand what I am doing wrong; I am only using official plugins; the only one I use is the official encryption addon; I disable everything else I can.

How to explain that I have a bugged “.gitignore” file just like that ?? I mean those sorts of things should never happen; this is basic syncing. Then there is no easy debug tools, no reset, no database check/clean… just logs… I can’t do anything that I would consider user friendly; Im sure experts would sort this in no time. I know there are the occ commands but should we really have to dig into that ?

Maybe it is bad to compare but I never ever had a single syncing issue for any kind of file with other syncing softwares; I just put things in the folder and it works; that is what I mean by focusing on the core functionalities. Addons are great but if the basics are not there for an above average user like me then non open source softwares have great days ahead.

Thanks for suggesting syncthing but sadly not for me yet since there is no IOS app and I need to access docs from my phone sometime. I really wish NC had a minimal build for pure file sync (maybe even integrated with GIT for versioning; the dream ? ), no pre-installed addons. Take a machete and clear the jungle :slight_smile:

back to icloud for now…

4

I have been in you position a million times. Even with Owncloud happened the same thing. I have concluded that I was doing something wrong with the configuration.

I haven’t had any problems for like a year now, with NextcloudPi. I use Arm boards in my case but you can install it easily on x86 machine.

So I would happily suggest NextcloudPi Curl Installer. You only need Debian 9 installed and you have to run just this line in terminal.

# curl -sSL https://raw.githubusercontent.com/nextcloud/nextcloudpi/master/install.sh | bash

Everything is getting installed through the bash script.
You can review what this script is going to do, if you want.

You can also review everything in the project’s github repo: https://github.com/nextcloud/nextcloudpi

5

Thanks for that, that is what I have been using with NC13 but it failed too. I just reinstalled it; this time using external storage… giving it another try

But looking at your comment and stratacast it looks like I am not the only one and I am guessing a lot of people are just giving up entirely. If Im right NC could be missing the biggest opportunity of all: the average user.

Not sure if I’ll be heard but I really think NC needs to refocus to become user friendly and more importantly nuclear proof for file syncing of all files including .htaccess because as a developper I need it for my projects

1 Like
6

Installing and configuring a webserver, database and so on is a bit of a challenge but with ready-to-go images, this is quite easy. If it should be available outside your local network, it’s a bit harder. If that should work all the time, you would have to go through vpns and proxies, but that will cost something.
The installation process already improved a lot, there are still a few cases where it can be difficult and users don’t get clear messages. That must improve but this depends as well on the users to report such issues.

For many small files, the performance is not great and you can get the sync process into trouble. I have as well seen a few problems with git, I often stop the sync process for these folders and only sync once it is done. You could probably write an app for git, but on the other hand there are already many git clients, you can easily sync git stuff through ssh, so why bother improving this on Nextcloud?

Encryption is for external storage. I had many problems with it (back in ownCloud times), and in one version all problems were related to this encryption app. Since it is only for external storage and the benefits on local storage is very limited, I don’t use it any more. Since then, everything is more stable, less problems. Sync errors are rare, a few conflict-files from time to time.

7

thanks @tflidd, that really confirms that NC needs improvement on the syncing engine

the GIT comment was really optional; I would be so happy with a bomb proof syncing only; I can install git on my side that is enough for me

as you said I am using a “ready to go” image (nextcloudpi 13 with external storage and encryption enables for external only)

just to update on my new install , currently syncing 9GB of data, client already filled with syncing errors (locked file)

8

The general problem with Raspberry Pi is that the resources are rather limited so you do need to do some optimization which depends on your use case. On larger computers, it is quite easy to give enough resources to each process. And if it is only plain file syncing, specialized software for that is probably more powerful (git-annex, syncthing), …

Since every setup and every use case is very different, you could build up a certain test case that can be reproduced, e.g. take a raspberry pi with nextcloud pi and say you are uploading a set of known files (either generate files of defined size containing random content or a known archive) and then test the performance. The community can try to find good parameters and if there are reproducible bugs, they can be reported to the bug tracker.

In the past, they do a lot improving the performance, e.g. last year they reduced the load on the database: https://nextcloud.com/blog/tu-berlin-halves-database-load-by-migrating-22k-users-to-nextcloud/
However, the company mainly does this for its customer, finding and testing smaller use cases, is something the community must do (only they know their needs).

9

@tflidd yes that makes sense, every software needs optimisation and true that the raspberryPi is limited but I am using an RP3 which I think we can agree on the fact that it should be more than enough for a dedicated CLI Debian and syncing only ? I am using an optimised distro already and I still have the same errors.

Before using RPI I also tried dedicated VM on desktops. same issues

Having used Owncloud in the past; I left it for the exact same reasons and when NC came I was so happy thinking that bugs will be cleared but as far as I can see the engine is the same with a few optimisations maybe but the core of it remains with same bugs.
OC was filled with file lock errors and encryption errors too. NC was a hard fork so that makes sense to me that they share the same issues but I thought the NC community would work on the core functionalities/stability but instead we saw tons of additional addons.

Maybe classic relational databases are not fit for this kind of application ? Maybe a NoSQL database should be used ? I mean I have tons of files; we are going to see the limits of MySQL fast as far as I understand DBs and I only have 9Gb of data but dev data is full of small files so the count goes up fast.

I see NC 14 announcing “structural changes”; not sure what that means but I hope that they go back to basics for some time.

when you say

Does that mean that NC is ok not having a great syncing experience ? I thought that file syncing was the core of NC business ? Is NC focused on being average at everything (news, chat etc…) instead of being excellent in its core business ? NC moto being to take control of our data; I don’t feel like that atm; wish I could

10

1 GB RAM is not very much, giving more to a database improves a lot the overall performance. Also network and external storage speed are not that great. But I would agree to say for a single user, it should be ok and at least not produce tons of errors (just large number of files will take some time to sync).

In your case it would be interesting to know, what happens when you start to see more and more errors.

File lock can be handled with Redis, encryption takes some performance and there are some additional bugs. I don’t use it, and at the moment I wouldn’t and rather wait for client-side encryption (or use different client-side solutions).

Not sure about the design, I’m not an DB expert to be able to judge. At least they got the locking stuff to redis which already helped a lot. If you have a bit of RAM and can be a bit generous about the database cache, the performance is not so bad.

No, there shouldn’t be issues with the syncing at all. But it’s not all about syncing, there is also collaboration, that you can share files, have a nice web-interface to do so, that you are very flexible regarding different storage types, … If you only take the pure sync performance, the fact that everything is based on php makes it unlikely that it can outperform sftp or other basic protocols. But worst case it should take a bit longer, there are no excuses for errors.

The fork was focussed on the server-side software and the mobile clients. The desktop clients were just taken from ownCloud and the development stalled a bit. Now with client-side encryption, they start again the client development…

(owncloud has worked on a few nice features like virtual file system and delta-sync which are expected in their next version, hopefully this works and gets backported to nextcloud as well).

11

Just a few hints on the information you gave us so far:

  • File locking problems: You should check if you have transactional locking enabled and your Redis setup is working.
  • Server side encryption: it makes no sense if you don’t use some 3rd party storage you can’t trust. Using it only locally creates just more problems and solves nothing really. An attacker for example could take over your server and get your password anyway. Only client side encryption (E2EE) keeps your data safe in case the server gets compromised.
  • A Raspberry PI is nice for a lot of things but not for storage applications. A lot of small files might be just a little bit too much for it anyway.
12

redis is working. As a single user can I disable file locking ?

good point but in E2EE the files are stored unencrypted then ? What would be the point of having server side encryption then ? why have both ?
I have my storage at home; let’s say a burglar comes and takes the HD; he would have access to my files without server side encryption. Would E2EE encrypt files on the server ?

I do not agree with that; while it true than an RP2 is really underpowered, an RP3 should be enough; also NC should not be subject to “small file issues”. If speed is an issue, buffers or cache or writing queues could be implemented to avoid errors.

agreed, php design is not a good pick for this kind of application but a backend alternative could be interfaced with the site. the syncing algorithm does not have to stay PHP based

If I remember well this feature is still “experimental” but I guess it can’t be worst than the current server side encryption; maybe i’ll give it a try in my next install.

13

Client-side encryption encrypts files before they are sent to the server, so the server only sees encrypted files. If it is just the burglar, you could use a file system encryption (luks, enc2fs, …), it’s probably much more tested.

14

You could, but is your Redis and transactional file locking really working? Do you have a oc_file_locks table in your database? If yes and it gets refilled if you clear the table by hand, then transactional file locking doesn’t work.

Because they are designed for different use cases. Server side encryption is useful if you have a remote storage mounted into your Nextcloud (Webdav, SFTP, SMB, Amazon S3, Openstack and so on) you can’t trust. Your server sends everything to this storage encrypted. The external storage operator only sees encrypted data, but your Nextcloud server can decrypt it. It doesn’t help you if somebody breaks into your Nextcloud server because they could see the data unencrypted. If you don’t need it, you should turn it off. It causes more problems than it solves, is incompatible with a lot of apps and makes backups and disaster recovery a nightmare.

E2EE/Client side encryption on the other hand sends all the data from your local client device to the Nextcloud server in an encrypted fashion. The server operator of your Nextcloud only sees encrypted data. An attacker that compromised your Nextcloud server also doesn’t see anything else than encrypted data.

Maybe an RPi3 is enough regarding the processing power, but it is still limited because of its circuit design. Everything is attached to a single USB 2.0 hub, so Ethernet and a USB disk can max out the throughput of the system.

Nextcloud also doesn’t have “small file issues” in general. I can sync 10000 small files with a total size 40MB in a few minutes. No errors, hiccups or other failures. But there is surely a lot of room for improvement in a lot of things.

I had a lot of issues with server side encryption. Turn it off, your experience will improve quite a lot. It had a really bad start in the old bad Owncloud times because they changed encryption two times with two releases. This caused some massive problems including data loss.

15

is there any info anywhere on how to do this with nextcloud ? I assume that if I encrypt the disk directly I need to decrypt for NC so we have the same issue where the decryption key is stored on the server ?

E2EE will be great… when it will be in production state… I remember NC advertising it; not mentioning the alpha state and disclaimer not to use it in production… So in my case unusable. Or have I missed something ?

I’ll check that

16

@alfred
@tflidd
just a thought: with E2EE since the server only sees encrypted data then there should be no file restriction from the server ? just need to disable it on the client side ? :heart_eyes:

17

There are other options out there (more angostic not tightly integrated into Nextcloud) like for example Cryptomator.

What restricted filetypes do you mean specifically?

18

all of them ? :wink: since it is just jebrish to the server should not bring any issues

19

Could this be an Apache issue rather than a Nextcloud issue? Do you have a FilesMatch directive that matches this file?

20

no and no
It was an NC bug; I’m guessing most likely coming from the encryption module or could have been a DB corruption too but we will never know; I have since reinstalled everything.