Major Blunder - Deleted PSQL data folder - What can I recover from my apps?

Nextcloud version : 23.0.2
Operating system and version : Arch Linux
Apache or nginx version : N/A
PHP version : 8.0.14

The issue you are facing:

My postgres database needed a major update from v13 to v14, and in the process of updating it, I managed to delete both the current postgres data folder, as well as the backup. I can take care of backing up my explicit Nextcloud files, but I need to figure out what else can be saved such as my passwords database from the passwords app, my calendars, tasks, etc.

I’m in a great bind here.

Is this the first time you’ve seen this error? : Y

Steps to replicate it:

  1. Prepare to upgrade postgres database major version
  2. Backup your /var/lib/postgres/data folder
  3. Delete your /var/lib/postgres/data folder and your backup of the data folder on accident
  4. ?
  5. Cry

I’m omitting the logs section since I clearly know what the heck I did and what the problem is.

I’m seeking help in finding out what I can recover/save, and how to do it. As mentioned above, I mostly care about my passwords as my latest off site backup is not as up to date as it should be. Next would come my calendar and todo list - I think I’m okay with losing other things beside these three things.

I also should mention that as far as the three app related things I mentioned above - I don’t care if I have to recover these things manually so long as I understand how to do that. If I can get a plaintext version of those things I’ll consider it a win. In other words, if I can rewrite my calendar entries, my todo list, and my passwords all in a .txt, I’m happy.

I’m really in a bind here - my last data directory backup is about a year old and I have concerns as to how helpful it would be.

Edit: I’ve found some automatic password backups which are up to date, but all information is encrypted, of course. How would I go about decrypting the values? If I imported the json into a separate nextcloud instance with the same Nextcloud username and password, would that work? Alternatively, what could I do? I do have a backup of my postgres/data from about a year ago, a postgres 12 version database. I’m thinking this may contain the server secret, based on what I’m reading?

I have yet to find data for the calendar or tasks apps.

Second Edit: It looks like I have something I can try to save the passwords from the automatic backup above, but I’m still unsure about the calendar or tasks apps data.

I’ve also made a post to the support section, here detailing the same problem.

NC 23, second to latest passwords release as of earlier today. I accidentally deleted my postgres /var/lib/postgres/data directory, as well as the backup I made of said directory when attempting to update my postgres database from 13 → 14. I have a backup of my postgres/data from about a year ago, a postgres 12 version database.

I can save my Nextcloud files myself, but I also want to save my passwords and preferably my calendar and todo list data as well. I’m not sure how to go about doing those last three things.

I found the latest automatic backup for my passwords, but of course the fields inside of the json are encrypted. What can I do to decrypt them? And, are the other two groups of data (calendar and todo list) saveable?

What is my best course of action here? I have at least a few passwords that I only added/created recently, and don’t have other backups for them. I would really like to recover this data if possible.

I’m explicitly holding off of any changes at this time, in hopes that any data, identifiers, salts, etc that I would need are still available somewhere.

I understand this is the passwords specific forum, so that’s what I focused on in this write-up. I’m not looking to violate any rules, so perhaps any help related to the calendar and todo list could be directed at the original thread in the support forum, as linked above? My priority is of course the passwords.

I would really appreciate some help with this, I am quite frustrated with my error and out of my experience range. I don’t want to do something on my own and screw it up. Thank you all very much.

Please do not double post. Please edit these into one cohesive post. Thanks.

Alright, I will attempt to make use of it with just one post. Since I have 3 specific tasks I’m trying to accomplish, I was trying to separate the two of them into more relevant areas.

However, the password app is the main focus of my question. I am trying to recover data from that app as described above.

The passwords backup can be restored on a new Nextcloud instance. See the server migration guide.

1 Like

Thank you so much for the response! I found this but based on some threads I was reading, it seems incomplete? Maybe not, maybe I’m just misunderstanding.

I obviously can’t use occ to export, as Nextcloud isn’t running without the psql connection. Should the automatic backup file I found be the same as an occ exported file? In other words, should the automatic backup I have be equivalent to steps 1-4 from that guide? What about the server secret as described?

**Warning:** The passwords app backup will overwrite your server secret. We recommend importing the backup before setting up other apps & encryption.

This seems to imply that I need to handle that server secret somehow and duplicate it from my original server, which would be consistent with what I was reading from similar threads. Am I misunderstanding what you’re saying, or is it really as simple as following steps5, 6 and 7 from the CLI/SSh Access section of that link?

Is the server secret contained in the automatic backup file, or do I need to dig in somewhere to find it? I do have that year old backup of my deleted data directory. I’m using extreme caution in hopes I don’t accidentally do something to lower my chances of success, like overwrite a server secret file on accident. I hope that’s understandable!

Also do you have any thoughts on the calendar and tasks data? Thank you very much!

Edit: Upon re-reading this, I seem to be overlooking the obvious. The quote I have from the guide you linked says it overwrites my server secret - so this would mean that the server secret is contained within the automatic backup, yes? So it is as simple as following your CLI guide from step 5 with a new instance of Nextcloud, using the automatic backup I have? It seems like if it was as simple as importing to any Nextcloud account, it wouldn’t have secure encryption, though. This is one reason why I feel I’m not understanding this. If everything needed to decrypt the .json backup is contained within the .json backup, why bother encrypting it in the first place?

1 Like

yes, you have the file, so just import it and restore the backup

No, the backup contains all encryption keys including the server secret. The warning is there because if you’re importing into another Nextcloud instance, the secret from that instance will be overwritten with the one from the backup.

The backup just contains what is in the database. If you use server side encryption only, that is everything needed to decrypt the data. But if you enabled E2E in the app, the encryption password of the user is obviously not part of it.

1 Like

Gotcha, thank you so much! My full server wide backup is still in progress, but I’ll give this a try as soon as that finishes.

By the way, I love your app. I used passman for a while before I found your passwords app, and I would like to say that I am glad I switched to yours. It’s a cleaner overall experience and the mobile app is handy.

Thank you!

Edit: Unrelated to the passwords app recovery, I just wanted to make it clear I am still unsure where the calendar and tasks data is stored. I would still like to know if this calendar and tasks data can be recovered.

So I have a new server up and running, and I tried to import the passwords backup. Nothing shows up, and nothing seems to change. Am I supposed to stop php-fpm first, or do something special?

I’m not having any luck here. I didn’t have any E2EE setup on the old server. Do I need to make my new user under the same username or something special?

Edit: I went ahead and tried creating old users, and yes, that was it! That was my suspicion since this was a server wide backup.

Alright, 1/3 goals achieved, and the most important one for sure. Thanks!

Calendars, tasks and contacts are only stored in the database. There is no way to recover those without the database or a database backup.

You could use calcardbackup to recover calendars and tasks from your 1 year old backup (have a look at the section does this also work with a broken instance in the README for a quick howto). Please note, that you need to import the database backup into a database first.

For the future you could use calcardbackup to create regular backups of calendar, tasks and contacts (also automatically with a cronjob).