Is there a "proper" way to bootstrap occ/script to use a different set of apps from the main installation?

ℹ️ Support
, , ,
1

Is there a “proper” way to bootstrap occ and/or a php cli scripts to use a different set of apps from the main installation?

This is a general-case support question related to occ and/or php cli scripts for which I’d like some guidance.

fwiw, the specific scenario is the following:

I just decrypted all of the files in my Nextcloud installation using occ encryption:decrypt-all, but discovered that many of the files are still encrypted, particularly those files in files_versions and files_trashbin. I’d like to work on recovering those files using some of the existing functionality of the encryption app, but without having the encryption app enabled for the main installation.

2

please review GitHub - nextcloud/encryption-recovery-tools: This project contains tools to recover files that have been encrypted with the Nextcloud End-to-End Encryption or Nextcloud Server-Side Encryption. which allows to decrypt the files if you still have the secret…

3

My question was a generic one related to bootstrapping occ and/or php cli scripts, so I don’t want this topic to get sidetracked.

But to address your reply, I have been using those tools. The issue is that they only work on the files themselves outside of Nextcloud. So even though I am able to decrypt all of the files, there isn’t a straight-forward way of reimporting those files, particularly in files_versions and files_trashbin/versions back into Nextcloud.

More about that here:

4

I think there is not and never will be a “straigt-forward” way to re-import (encrypted) files together with a history. Assuming the system works as expected there is no need to perform such action. In case there is a problem the root cause must be addressed rather then trying to work around the symptoms. The need to perform such manipulation is either a result from some fault and must be addressed specific to a situation or there is some strange requirement which doesn’t fit into normal use of the application.

But in general the logic for version is pretty clear and you can rebuild the version history of the file using little scripting yourself…

5

Well, the system didn’t “work as expected”, as I expected “decrypt-all” to “decrypt all” that was originally encrypted by the encryption app. This is all part of the core distribution, not even talking about any other apps.

I am not an experienced Nextcloud developer well-versed in The Nextcloud Way of doing things. As such, I try not to make too many assumptions, and that is why I am asking my question here.

I am just trying to figure out what else I have to do to properly re-import decrypted files_versions without potentially breaking the Nextcloud installation in unforeseen ways, since the database still treats these decrypted files as encrypted, notably 'fileid', 'encrypted', and 'etag' in oc_filecache.

Can I just change 'encrypted' to 0 after replacing the encrypted files with the decrypted files in files_versions? Is there a better way to do this? APIs exist for a reason.

6

Separately, as it relates to my original question, is there a proper way to bootstrap occ and/or php cli scripts with a different set of apps than from the main installation?

Basically, it’s taking forever for occ encryption:decrypt-all to decrypt my files, almost ten days to decrypt 1.5T of data so far, with 8.5T more to go on a system with a 16-core 3rd-generation Ryzen processor and plenty of memory. Taking everything offline for so long is already causing problems.

So I want to lock down/disable the accounts that aren’t yet decrypted and bring the Nextcloud installation back online while I work on the other stuff incrementally.

7

according to your other post the decryption work but doesn’t perform as you expected or is there any other issue with it? I admin 10 days is longer I would have expected but as long the process continues and the result is as expected I would continue.

at least on the first glance I don’t see any hint the instance must be offline during decryption:

you can decrypt data for each user one-by-one reducing the impact on the server and users: Using the occ command — Nextcloud latest Administration Manual latest documentation

The occ encryption:decrypt-all can take a lot of time. You can run one user at a time like so: occ encryption:decrypt-all <user-id>.

don’t mangle with the database and always use proper interfaces. as long the operation just takes longer than expected I suggest to wait - recovering a broken system could cause more work and longer outage. you can check if topics related to encryption-server show you another good way to recover…

I don’t get the point… you can install/remove apps using occ is this your question?

8

No, occ encryption:decrypt-all does NOT decrypt files in files_versions, files_trashbin, or files_trashbin/versions AT ALL. This is an unsolved problem that was originally reported more than four years ago, and there hasn’t been a solution or even any meaningful guidance in the time since.

So this is the underlying problem I have been trying to solve, and it will require some level of scripting and/or database administration that I am not entirely comfortable doing because I am not an experienced Nextcloud developer. I don’t even know if there are proper interfaces to do what I want/need to do.

btw, I have already reverted to decrypting user by user, and it’s still painfully slow. At the rate it is going, it will take another two months (!!!) to decrypt. I cannot possibly be offline for that long. It’s not even a huge amount of data. The php cli scripts from the recovery-tools are much faster, but they are also designed to work outside of Nextcloud.

As for my question about bootstrapping, forget about encryption for a moment. I am just asking about the general case.

Say I have a basic Nextcloud installation where some basic apps are installed, e.g. files, activity, notifications, etc.

So I can use occ to perform tasks from the command line like occ files:scan, occ activity:send-mails, occ notification:generate, etc.

Now say I want to bootstrap a custom php cli script called custom-occ which will also be executed from the command line.

But I want to be able to specify a different set of apps to be bootstrapped for this custom-occ php cli script, e.g. only the files app from the main installation and a custom app myapp.

So in this case, I would still be able to perform tasks like custom-occ files:scan and custom-occ myapp:function, but custom-occ activity:send-mails and custom-occ notification:generate will not work since they are intentionally not bootstrapped.

I think this would be possible by using an alternate config/config.php, but I am not sure if there is a better way to do this.

9

occ interacts with the system it is running on. I assume you want to prevent notification when you execute occ tasks but I don’t think some custom occ is the right way to do so. I would just disable/remove the activity app for the time when you run the task. maybe disabling the respective background jobs or simply stopping cron jobs would stop notifications as well.