Lost Contacts and Calendar due to (interrupted) App Update?

Hi,

I’m running NextCloudPi on a Raspberry Pi for my family. Unfortunately, a recent update of the Contacts and Calendar App seems to lose the table of contacts and calendar events for every family members on the instance.

Yesterday I noticed an app update prompt for Contacts 5.3.0 and Calendar 4.4.0 after logging as Admin in NextCloud from Web interface. I switched to browser other tabs after clicking the “Update All” button. Due to my sometimes wacky internet connection to App Store, it could take some time.

After a while, I notice NextCloud Android client notified me the server is in maintenance mode, which is I didn’t notice during prior experience of updating apps. Hence I switched back to Web interface and noticed that only Contacts app was updated while Calendar was not, due to slow internet connection.
Normally under this situation in the past, I would simply try again in another time.

It didn’t take long for me to notice the contacts and calendar events were all gone from my mobile, synced with NextCloud via DAVroid. And the same for other family members as well!

Login via Web interface, something weird can be seen from Contact App from the top left corner menu area:

Screenshot 2023-06-09 at 07-43-55 Contacts - NextCloudPi

In Calendar App, the circle in front of my calendar seemed to spin forever:

Screenshot 2023-06-09 at 07-49-56 June 2023 - Calendar - NextCloudPi

On a desperate attempt to fix all these(imaging the pressure from family members :cry:), I switched to a regional mirror of the AppStore and performed the app update via CLI:

$ sudo ncp-config
Running nc-update-nc-apps
Cannot load Zend OPcache - it was already loaded
calendar new version available: 4.4.0
Error: Database error when running migration 4004Date20230414163505 for app calendar
Migration step 'OCA\Calendar\Migration\Version4004Date20230414163505' is unknown
carnet new version available: 0.24.7
carnet updated
Done. Press any key...

Mirror certainly works as another Carnet App was upgraded successfully. For the Calendar, I guess I was hit by the bug mentioned here: Updating Calendar App breaks NextCloud 26.0.2

Followed the instruction in comment, I did below:

$ sudo -u www-data php /var/www/nextcloud/occ upgrade
Cannot load Zend OPcache - it was already loaded
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Setting log level to debug
Turned on maintenance mode
Updating database schema
Updated database
Updating <calendar> ...
Updated <calendar> to 4.4.0
Starting code integrity check...
Finished code integrity check
Update successful
Turned off maintenance mode
Resetting log level

Calendar App was upgraded successfully, too!

Unfortunately, the errors in Contacts and Calendar app remain and all my users on this NextCloud instances had lost access to contacts and calendar events.

Thanks in advance for any advice or guidance on how to fix those issue and restore the functionality!

Additional system info below:

NextcloudPi version  v1.52.0
NextcloudPi image    NextCloudPi_03-02-22
OS                   Debian GNU/Linux 11. 6.1.21-v8+ (aarch64)
automount            yes
USB devices          none
datadir              /mnt/ncdata/data
data in SD           yes
data filesystem      ext2/ext3
data disk usage      228G/361G
rootfs usage         228G/361G
swapfile             /swapfile
dbdir                /var/lib/mysql
Nextcloud check      ok
Nextcloud version    26.0.1.1
HTTPD service        up
PHP service          up
MariaDB service      up
Redis service        up
HPB service          up
Postfix service      up
Internet check       ok
Public IPv4          XXX.XXX.XXX.XXX
Public IPv6          not found
Port 80              closed
Port 443             closed
IP                   192.168.199.181
Gateway              192.168.199.1
Interface            eth0
Certificates         none
NAT loopback         no
Uptime               5:57
Version: 8.1.18
Memory limit: 682.2 MB
Max execution time: 3600
Upload max size: 10 GB
Extensions: Core, date, libxml, openssl, pcre, zlib, filter, hash, json, Reflection, SPL, session, standard, sodium, cgi-fcgi, mysqlnd, PDO, xml, bcmath, bz2, calendar, ctype, curl, dom, mbstring, FFI, fileinfo, ftp, gd, gettext, gmp, iconv, igbinary, intl, ldap, exif, mysqli, pdo_mysql, Phar, posix, readline, redis, shmop, SimpleXML, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, xmlreader, xmlwriter, xsl, zip, Zend OPcache
Type: mysql
Version: 10.5.19
Size: 24.4 MB

Do you have some recent backups of the corresponding mysql database? If so you could recover your data.

If installed, check your nextcloud client if the option to backup contacts and calendar data is enabled and active. Then there is a backup of both on your device (android).

Just to be sure, the method you recommend seems to go directly into the realm of disaster recovery. Hence does it mean the database was somehow corrupted during my action?

I should have a not-too-distant away database backup using one of the provided NextCloudPi tools.

  • Would it work if I just restore the entire database?
  • Since all the files are unaffected, should I bother to isolate the Contact and Calendar related tables from the backup copy?
  • The database backup was created before updating Contact and Calendar Apps. Is there any special steps I need to take after restoring?

Thanks for the reply!

EDIT: Asked a bit more questions regarding to restoring using database backup.

I don’t know if your database has gone corrupt or not. If so, there should be a message in syslog. It’s also possible to check the tables with the correponding mysql commands.

I have also been hitten by the update of the calendar. Because my pi failed to reboot (SD corruption) I had to restore from backup as of end of may. Directly after the successfull recovery and reboot I restored the very last backup of mysql nextcloud database (entire database). And finally I run nc-scan on the corresponding nc data directories.

I’m running a separate mysql nextcloud database backup ever 4 hours a day, so that there is only a minimal data loss. If hosting other persons data I’d recommend also to establish backups on a frequent regular basis.

PS: the “nextcloudpi backup” contains a mysql backup.

1 Like

I tried restore to a previous working backup with both database and file using nc-restore. Still not working. Same error as before.

After the restore, the Contacts and Calendar apps were set back to old versions again and asked for upgrade. This time I setup the mirror for App Store and performed the update via CLI.

$ sudo ncp-config
Running nc-update-nc-apps
Cannot load Zend OPcache - it was already loaded
calendar new version available: 4.4.0
Error: Database error when running migration 4004Date20230414163505 for app calendar
Migration step 'OCA\Calendar\Migration\Version4004Date20230414163505' is unknown
contacts new version available: 5.3.0
contacts updated
carnet new version available: 0.24.7
carnet updated
Done. Press any key...

The same error. I had to apply the same occ upgrade trick again.

sudo -u www-data php /var/www/nextcloud/occ upgrade
Cannot load Zend OPcache - it was already loaded
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
Setting log level to debug
Turned on maintenance mode
Updating database schema
Updated database
Updating <calendar> ...
Updated <calendar> to 4.4.0
Starting code integrity check...
Finished code integrity check
Update successful
Turned off maintenance mode
Resetting log level

This struck me ultimately confused. What could possibly go wrong?
I might try reinstall NextCloud itself tomorrow.

As a precaution, I run full disk fsck already and confirmed no file system error or bad block.

I hit the very same problem upgrading the calendar app and applied the same steps. But because my SD got corrupt during reboot I had no chance to check the logs.

So, did you loose data or is it an access issue (may be due to missing permissions)?

If I’m hit by some data problem I’m recovering the database directly, e.g.

sudo mysql --database nextcloud < nextcloud-sqlbkp_20230608.bak

If necessary I extract the database file out of nc full backup by means of tar.

On a fresh day, I came to this post about extracting Contact and Calendar from database directly: Calcardbackup: bash script to backup Nextcloud calendars and addressbooks as *.ics/*.vcf files

If this handy tool can extract the table from database, at least it means the tables are in fact all good and not corrupted.
It did! It extracted all successfully:

$  sudo -u www-data ./calcardbackup "/var/www/nextcloud"

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+  Sat 10 Jun 2023 08:26:28 AM CST --> START calcardbackup ver. 5.2.0 (22.03.2023), AGPL-3.0
+  Checking dependencies and preparing...
+  no usersfile given:
+     - will fetch all available items from database
+  Using URL: https://nextcloudpi
+  can't read status.php (cURL code '60').
+  Nextcloud 26.0.1 detected.
+  Database of chosen Nextcloud installation is MySQL/MariaDB.
+  Looking for calendars in your Nextcloud:
+  Saving calendar ncp-Personal.ics......empty calendar!
+  Saving calendar aaaa-Personal.ics......success!
+  Saving calendar bbbb-Personal.ics......success!
+  Saving calendar cccc-Personal.ics......empty calendar!
+  Saving calendar dddd-Personal.ics......success!
+  Saving calendar eeee-Personal.ics......empty calendar!
+  Saving calendar ffff-Personal.ics......empty calendar!
+  Looking for calendarsubscriptions in your Nextcloud:
+  No calendarsubscriptions found.
+  Looking for addressbooks in your Nextcloud:
+  Saving addressbook ncp-Contacts.vcf......empty addressbook!
+  Saving addressbook aaaa-Contacts.vcf......success!
+  Saving addressbook bbbb-Contacts.vcf......success!
+  Saving addressbook cccc-Contacts.vcf......success!
+  Saving addressbook dddd-Contacts.vcf......success!
+  Saving addressbook eeee-Contacts.vcf......success!
+  Saving addressbook ffff-Contacts.vcf......empty addressbook!
+  removing temporary files...
+  Compressing backup as *.tar.gz file. Be patient - this may take a while.
+  Backup successfully compressed!
+  Find your compressed backup here: /home/pi/calcardbackup/backups/calcardbackup-2023-06-10.tar.gz
+  Sat 10 Jun 2023 08:26:41 AM CST --> END calcardbackup
+
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+  Support the developer: https://codeberg.org/BernieO/calcardbackup#donations
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

This leads to think towards a permission problem as you said as no actual data nor table were corrupted.

Then I set to ncp-config to address those:

$ sudo ncp-config
Cannot load Zend OPcache - it was already loaded
Running clear-php-opcache
Clearing opcache...
Done.
Done. Press any key...

Cannot load Zend OPcache - it was already loaded
Running nc-fix-permissions
Cannot load Zend OPcache - it was already loaded
fixing permissions in /mnt/ncdata/data... done
Done. Press any key...

Then I logged into the Web view. It works now, for every member!.

Hence I guess during the error of Calendar update, it messed the permission somehow, which caused the trouble.

Thank you for the feedback and let me know this is not an isolated case. I wish the App developer would be aware of this.

2 Likes

That’s great that you now have a solution to the problem and that there is no loss of data. Reading through your actions, it’s clear that it’s not a permissions issue.

The solution here is to clear the opcache (from php). I do this as soon as I notice that the client does not process any more changes (error 503 in /var/log/apache2/nc-access.log on my system).

nc-fix-permissions fixes the permissions on the files. The data for contacts, calendar and tasks are managed in mysql tables.

Holy shit!!! This post is awesome!
I ran exactly into the same problem. Also the process with updating calendar and contacs failing was the same.
Running clear-php-opcahce and nc-fix-permissions solved it for calendar and contacts app.

Thank you :wink: