Nextcloud-Desktop background service

[TC-AdKa]

Background

In our Organization we use Nextcloud instances with the Nextcloud Desktop client to manage file synchronization(upwards of 250,000 files), on multi-user Windows workstations. The Virtual filesystem is unreliable and causes crashes frequently. The crashes are an issue when used as a background service(started via scheduled task at first system login, ran as a user) which doesn’t recover without manual intervention. The storage usage of each user having a separate instance of the client causes a lot of storage overhead and becomes cumbersome to centrally manage. No configuration on the conflict management behavior becomes a problem for background services aswell.

Note: Using Latest Stable version of the Windows Nextcloud Desktop client

Goal

We’d like to use the desktop client as a background service with minimal manual intervention since this interrupts the user session.

Questions

Is there a way to increase the stability of the desktop client where the use as a background service becomes feasible?

OR

Is there a way to manage the client in such a way that manual intervention can be done without interrupting the user session?

[wwe]

Hello @TC-AdKa,

welcome to the Nextcloud community!

I’m not sure i get your points but it sounds you are professional and ask after some testing and troubleshooting.

I assume you multi-user box is running in some data center likely close to your Nextcloud server. i this case there is no reason to sync files.. Nextcloud is build on top of WebDAV and your case it is possible to mount users storage using this protocol with some advantages: not storage consumption, no sync issues. Common arguments against using WebDAV don’t apply in scenario where your “client” aka multi-user box is always connected to the server with high-bandwidth..

[TC-AdKa]

Hello @wwe,

Thank you for taking the time to respond, sadly no, the aforementioned are office workstations, connected via GBe.

Note: Our Storage Backend for the Nextcloud Server is SMB on NVMEs using a Virtio NIC. I don’t know if this is relevant, but I’d assume yes due to SMB overhead(lower IOPs).

For reference: our total data usage is around 1TB

I’ve argued WebDAV would be more stable, claims were made that there are to many limitations in the WebDAV protocol, such as non-ascii characters. Non-ascii is an issue for us since we have ö/ä/ü/ß on our keyboards and in some of our file names as well as whitespaces. Another limitation is the network speed since the workstations have GBe and we have large, and supplementary small files, that are used by CAD programs, making the programs slow down and lag, thus the preference for local synced data.

A wish that was made, that I haven’t mentioned, is Double-Redundancy(Server(RAID)+Client) but this is solved by the sync function of the desktop client

Pardon: multi-user was the wrong word since the Windows workstations are single user(Windows Pro) but there are multiple users working with the workstations, at various differing times. The background service, a concept(not made by me), was introduced to remove user intervention, reduce setup and to reduce storage overhead(file duplicates in differing directories). The users move from one workstation to the other from time to time.

Assumption: The client not being/having a background service is a deliberate design choice due to user intervention being necessary for file conflicts, thus:

No configuration on the conflict management behavior becomes a problem for background services as well.

EDIT: Fixed some typos. Yes, I am working as a system admin(profession: linux systems engineer) at a company, I have to work with windows machines from time to time…

[wwe]

@TC-AdKa

I’ve argued WebDAV would be more stable, claims were made that there are to many limitations in the WebDAV protocol, such as non-ascii characters. Non-ascii is an issue for us since we have ö/ä/ü/ß on our keyboards and in some of our file names as well as whitespaces.

did you try? knowing your problematic cases allows advance testing and with some testing you could have “good enough” testing overage to decide if it works or not in your setup

@TC-AdKa

Another limitation is the network speed since the workstations have GBe and we have large, and supplementary small files, that are used by CAD programs, making the programs slow down and lag, thus the preference for local synced data.

I’m pretty sure both VFS and WebDAV could result in some issues because the application is designed for fast access which is delayed if the files doesn’t exists locally..

Given your CAD requirements I think there is no good way to “silently” prepare a user profile without wait time or or user interaction. on the other side as long you can afford to assign only one or two workstations to each user and spend some time for initial setup it should work “good enough” with fully synced files. If you have well organised folder structure you still can choose to sync only a subset of the files reducing local storage requirements..

[tflidd]

@TC-AdKa

SMB on NVMEs using a Virtio NIC. I don’t know if this is relevant, but I’d assume yes due to SMB overhead(lower IOPs).

And that’s all on your local network, why don’t you use SMB directly with the workstations? Putting a layer with Nextcloud and webdav over it won’t improve the performance.

on mounting webdav storage directly, there are a few commercial programs providing this feature) But I have no idea about the performance with Nextcloud.

[TC-AdKa]

Valid point, I’ve argued this as well, but this causes problems with CAD due to latency and overhead(creating lag), WebDAV has similar issues…having more overhead.

We migrated from SMB to Nextcloud due to the aforementioned issues with SMB. I agree SMB+WebDAV definitely won’t improve performance, it has the opposite effect making it worse.

@tflidd

on mounting webdav storage directly, there are a few commercial programs providing this feature) But I have no idea about the performance with Nextcloud.

Mounting WebDAV is less of an issue, since it’s included in almost every virtual filesystem(at least under fuse/gvfs/davfs and windows).

We opted for sync, syncing data retains the speed of local data.

Nextcloud Desktop’s VFS causes crashes, probably due to the file index creation being quite intensive due to amount of files in the pool and maybe the storage backend, thus the question:

Is there a way to increase the stability of the desktop client where the use as a background service becomes feasible?

Thought: Maybe using a paravirtualized filesystem instead of SMB would reduce overhead and increase reliability. What do y’all think?

[tflidd]

@TC-AdKa

We opted for sync, syncing data retains the speed of local data.

Nextcloud Desktop’s VFS causes crashes, probably due to the file index creation being quite intensive due to amount of files in the pool and maybe the storage backend, thus the question:

So you want a locally cached version. Did you try to sync a smaller structure with a few projects, so you won’t be bothered with the sync process in the background (with and without VFS), just to see if in theory it might be a solution. During you work on a file, to avoid continuous sync, I don’t know if the file locking app might be helping:
https://apps.nextcloud.com/apps/files_lock

[adelaar]

@TC-AdKa

we have ö/ä/ü/ß on our keyboards and in some of our file names as well as whitespaces

I did test it for you. I created a File named Dateiname äöüßÄÖÜ.txt on a WebDAV mount. It works with no Problem. So WebDAV is able to handle that.

Since i did it on a Linux device not Windoof, Windoof might be the Problem but not WebDAV itself.

[TC-AdKa]

@tflidd

So you want a locally cached version. Did you try to sync a smaller structure with a few projects, so you won’t be bothered with the sync process in the background (with and without VFS), just to see if in theory it might be a solution. During you work on a file, to avoid continuous sync, I don’t know if the file locking app might be helping:
Temporary files lock - Apps - App Store - Nextcloud

@tflidd This will help with conflicts for sure, that’s a great extension, thank you for the pointer. Yes, I’ve done exactly as prescribed, managing non-VFS directory structures in the past, but due to how flexible the projects are and how much data there it’s even more administration. Managing all the different directory structures for different projects and a per user reconfiguration is a lot of maintenance.

@adelaar

Since i did it on a Linux device not Windoof, Windoof might be the Problem but not WebDAV itself.

@adelaar Thank you, I did some testing as well, seems as though it’s a non-issue. Windoof doesn’t seem to have an issue with special characters either.

[tflidd]

@TC-AdKa

Yes, I’ve done exactly as prescribed, managing non-VFS directory structures in the past, but due to how flexible the projects are and how much data there it’s even more administration. Managing all the different directory structures for different projects and a per user reconfiguration is a lot of maintenance.

Sorry, I didn’t mean that as a solution more like figure out what works in principle. For smaller files, the VFS is perhaps less likely to crash, is the caching and everything else it does sufficient, so they just need to fix the crashes? If even with a small folder it doesn’t work at all, you can directly skip this option as possible solution.

Also one thing to try, if you put everything on Nextcloud native storage, does it work better without the additional SMB-mount layer?

You need to narrow down where your problem is, is it VFS in principle, the mounting of external storage in Nextcloud, just the amount of files to be synced, …