CRITICAL Cient - sync database corrupted after enabling e2ee

📱 Desktop & mobile clients 💻 Desktop
1

Everything was working fine until I installed the official E2EE application. All I did was adding an encrypted folder and add about 10 files into it. It completely broke and corrupted the sync database which makes Nextcloud impossible to continue syncing.

First step I tried was to disable the E2EE application. From there it corrupted my sync database.

Now I can’t sync, access or open ANY of my files. This is a critical issue and I’d appreciate support ASAP.

image

image

And yes, I do have write permissions on the nextcloud folder.

image
image386Ă—519 58.6 KB

Please help me. This is unusable.

2

Could you be a little more verbose with your informations please?

  • What did yuo do EXACTLY? STEP by STEP.
  • What is the Nextcloud (server/pod) version?
  • What apps are enabled on the server?
  • On what Client did you activate the E2EE?
  • On what Client did you add those 10 files into it?
  • On what place in the file hierarchy did you place the E2EE-Folder?
  • Is E2EE enabled in the client where you are experiencing synchronization issues?
  • Do you have any other encryption tools enabled, like server side encryption?
  • etc.

Much of these questions wille be responded if you post the output of

occ support:report

Much and good luck,
ernolf

1 Like
3
  • What did yuo do EXACTLY? STEP by STEP.

To the best of what I remember;

  1. I’ve setup Nextcloud
  2. Installed the windows client
  3. Added a bunch of files to my Cloud folder
  4. Files started to sync (not sure if it had time to finish syncing but there wasn’t any errors yet)
  5. After I added my files I was ready to add my personal files I want encrypted so I went and installed E2EE application
  6. I created a folder called “Personal” in my Cloud folder right click > Nextcloud > Encrypt folder (not sure of the correct appellation)
  7. I tried with 2 files and it worked. I could verify the files were encrypted by going into “/cloud/freeman/files/” where I could see the files ecrypted with complex filenames
  8. I added the rest of the files, 2,000 files in total.
  9. Then I got the errors saying that the syncing process couldn’t be completed.

What is the Nextcloud (server/pod) version?

Nextcloud Hub 8 (29.0.0)

What apps are enabled on the server?

  • Antivirus for files 5.5.4
  • Talk 19.0.1
  • Client Push 0.6.12
  • Cospend 1.6.1
  • Custom CSS 1.16.0
  • Deleted files 1.19.0
  • Draw.io 3.0.2
  • External storage support 1.21.0
  • File sharing 1.21.0
  • Forms 4.2.4
  • Log Reader 2.14.0
  • Nextcloud Office 8.4.2
  • Notes 4.10.0
  • Notifications 2.17.0
  • OpenID Connect Login 3.1.1
  • PDF viewer 2.10.0
  • Preview Generator 5.5.0
  • Share by mail 1.19.0
  • Unrounded Corners 1.1.3
  • Versions 1.22.0

Temporarily disabled

  • End-to-End Encryption 1.15.2

On what Client did you activate the E2EE?
On what Client did you add those 10 files into it?

Only on the Windows client.

On what place in the file hierarchy did you place the E2EE-Folder?

At the root of the Cloud folder on the windows client.

Is E2EE enabled in the client where you are experiencing synchronization issues?

I currently disabled the application in hope this would at least fix the sync issues but now even though it’s disabled the syncing process is broken.

Do you have any other encryption tools enabled, like server side encryption?

I’m not using encryption anywhere on the Nextcloud dataset. I’m using encryption on other datasets (truenas scale) but it’s unrelated to this issue.

occ support:report

This command doesn’t work on my installation?

image
image783Ă—136 6.7 KB

Thank you for helping, I would really like to get this working asap.

4

I wonder if part of the problem here is the use of VFS with E2EE. Afaik E2EE folders have be be basically excluded from VFS.

5

End-to-end Encryption works with Virtual Files (VFS) but only on a per-folder level. Folders with E2EE can be made available offline in their entirety, but the individual files in them can not be retrieved on demand. This is mainly due to two technical reasons. First, the Windows VFS API is not designed for handling encrypted files. Second, while the VFS is designed to deal mostly with large files, E2EE is mostly recommended for use with small files as encrypting and decrypting large files puts large demands on the computer infrastructure.

This page is saying that VFS and E2EE work together.

My understanding of the documentation is that the entirety of an encrypted folder must be downloaded otherwise it’s content can’t be shown.

An update on the way E2EE works could solve this issue. I believe the best way to overcome this issue at this time would be to exclude the encrypted folder from VFS. Is that even possible?

If this is really what broke the sync job, I believe it’s a rather serious issue than a user error. A software shouldn’t lead their user in such a situation. At least deactivating VFS by default for E2EE folders and if the user activates it, a prompt saying that it’s an experimental feature that could lead to database corruption. With a phrase the user must enter so that they must read the prompt.

With all that said, what should I do next?

6

So after digging more into this issue, I tried some stuff which brought back sync. It seems like the corrupted sync db error is related to a local database and not the actual server side file database.

Anyone else getting this issue, following these steps might get your data lost so I suggest starting your own topic. Could you tell me if what I did is correct @ernolf ?

First I removed the current folder synchronization.

image
image1066Ă—250 22.2 KB

Then I tried adding it back again but it wouldn’t let me do it.

image
image749Ă—263 20.6 KB

So I checked the permissions and Users had nothing checked but the Special Permissions set to Deny. Also I couldn’t edit the permissions at all.

I went and deleted the Users permission and added back again and checked the following. Please let me know if that’s how it should be.

image
image484Ă—649 16.3 KB

After this I could finally add back my synced folder. I’ve setup Virtual File Sync since I don’t have enough space on my computer. It seems like it was able to upload the files that were previously added in the local folder which couldn’t upload on the cloud because of the error I had before.

Sync completed. Green icon. Good?

The only folder that wouldn’t upload is the Private folder which Nextcloud client said it was on the exclusion list. I went in the excluded files and folders and it wasn’t there. Strange. I renamed that folder to something else and now it’s currently uploading everything (61 GB) unencrypted to my cloud. It’s taking much more time than the 300 GB that synced just before. This is why I’m doubting I might or might not have lost data. Hard to tell right now.

I also went and pressed the button Free up local space which deleted everything from my computer and only kept a copy on the cloud.

image

Now the sync status is broken. Let’s take my Logo folder as an example which says that it’s currently syncing although everything inside the folder has been synced properly.

image

See inside of the folder, all files got their cloud icon. I also verified and there are no hidden files in that Logo folder that might haven’t sync.

image
image257Ă—590 12.7 KB

E2EE 2nd try

Now that syncing seems to be working again, I’ll try to setup E2EE again. According to the documentation it should be working with Virtual File Sync but @jtr implied that it might be broken or unstable?

How should I proceed for this folder? Shall I right click the folder and set it to Make always available locally? And then press the Encrypt button on that folder?

image

Perhaps I misunderstood and that I should do it differently?

Thanks for helping.

7

I really can’t say if that’s the cause of your problem but I noticed that E2EE folders can somehow affect other folders on the same level. So, I have created a folder named “E2EE” in the root directory, and all folders with encrypted data go in there. To clarify, the E2EE folder itself is NOT encrypted. Here’s how it looks:

/ (Root Directory)
├── Documents
│   ├── file1.docx
│   ├── file2.docx
│   └── file3.docx
├── Photos
│   ├── photo1.jpg
│   ├── photo2.jpg
│   └── photo3.jpg
├── Music
│   ├── song1.mp3
│   ├── song2.mp3
│   └── song3.mp3
└── E2EE
    ├── Secret1
    │   ├── encrypted_file1
    │   └── encrypted_file2
    ├── Secret2
    │   ├── encrypted_file3
    │   └── encrypted_file4
    └── Secret3
        ├── encrypted_file5
        └── encrypted_file6

I hope this can help you

Much and good luck,
ernolf

8

I could finally get e2ee working but it was a pain. I’ve found out many issues.

  1. Like @jtr mentioned VFS doesn’t work with E2EE. You can create the empty folders, encrypt them but as soon as you’ll be adding files into them they will crash and corrupt your sync database.

  2. I confirm that creating a folder at the root .e2ee and encrypting only the subfolders will stop breaking things with other folders at the root.

  3. If you upload a folder with files and then try to encrypt it, it’ll give you an error message instead of telling you that you can only encrypt an empty folder. The error message implies that there’s a problem instead of that it’s not the way to do it. I’ve wasted quite some time trying to fix that.

  4. Playing with E2EE will often break the permissions in Security and create a second Users permission with Special permissions set to Deny which prevents Nextcloud to continue working. Deleting that second Users permission under Advanced which will bring you the screen Advanced Security Settings for Nextcloud will fix the broken sync database prompt and will let you continue syncing.

  5. I’m using an hidden folder like .e2ee which seemed to be causing some syncing issues at some point but I’m not sure how and why this happened.

  6. Sometimes when I reboot my computer the E2EE is deactivated and I must enter my pass phrase again to get it back and running.

  7. Sometimes when I reboot my computer the permission issues is back and I must go and delete the Users again.

  8. If you delete your folder .e2ee because you had VFS activated and want to start fresh and create the same sub folders for example /.e2ee/test, Nextcloud will prompt you that those folders are on the exclusion list and that they wont sync. I opened the exclusion list from the gui and from the local file and it’s empty. So this is broken. At least it feels broken as it doesn’t make any sense.

I needed E2EE encryption for sharing private files with a coworker. Next step will be to setup my coworker Nextcloud account and share them my encrypted folders. I read that E2EE and sharing was broken at the moment so I’m not sure if that’s true since the application is officially out and available.

2 Likes
9

Hi, I’m also running into these issue on several computers, without knowing until now, why that happens on some PC’s and on others not.
Luckily it’s easy to solve:
You only have to delete the last Group named Users, which has complete restriction to that folder. Since Users includes ALL Users, we are paralysed because of that restriction. I tried giving the Users group complete access, but the values weren’t saved, so I gave the recommendation of someone in another chat here a try, and it worked like a charm for us. Where I did that trick, the problem never came back.
We can’t use encryption because of using Groupfolders. I see that error happen since last versions from 29.0 - 29.04. Also, updating the desktop client to 3.13.2 does not change nothing.

10

For Windows client I recommend checking this thread: