Calcardbackup: bash script to backup Nextcloud calendars and addressbooks as *.ics/*.vcf files


#1

I like to backup my Nextcloud calendars and addressbooks as .ics/.vcf files, because it is very easy to restore: just import the backed up .ics/.vcf file to the server or a client and the accidentally deleted data is back online.

Therefore I initially wrote a bash script with hardcoded links for my nextcloud installation. Two friends were interested and so I improved the script to be more generic which possibly makes it interesting for other users as well.

The script downloads all available calendars/addressbooks belonging to certain users from a Nextcloud instance as .ics/.vcf files. Usernames and according passwords have to be given in a separate file. All downloaded files are then compressed to a single file with the current date in the filename.

It works with all currently released Nextcloud versions (and ownCloud >= 5.0) and has been tested under Debian Wheezy, Jessie and Stretch. Of course shell access to the server is needed. Configuration is quite simple: in most cases the path to the Nextcloud directory sufficient. It is not necessary to run the script with root privileges - in default configuration it only needs to have read access to Nextclouds configuration file ‘config.php’. At no point any data leaves the server.

If that sounds interesting to you, you are welcome to use it:

Best wishes,
Bernie_O


Automatic backup of calendar entries
Download ics file by script
[Question] Data file location for Contacts and Calendar apps - Snap Installation
[calendar] Kann man ein Backup nur von einem User machen
[Help] Problems in exporting large contacts file
Migrating from MySQL to Postgresql
Migration von oc 10.0.10
Retrieve calendar data from database
Loading failed for the <script> with source “https://*/index.php/js/core/merged-template-prepend.js
Tasks backup by using URLs
Nextcloud lost config.php data
Restoring Data from Database Backup
How to Keep My Calendar After an Upgrade to New SD Card Installation
Export all contacts or calendars
Nextcloud-Snap: question about nextcloud.mysql-client
#2

Thanks for sharing your script with us. Some remarks:

  • why do you need the server-path and the folder on the server? For me it makes more sense to make a backup from an external host (like https://uriesk.wordpress.com/2015/02/13/backup-your-caldav-calendar-with-cadaver/) or just run a backup server-side script for each user (directly use the database, don’t ask the user’s password).
  • Having a file with all users and clear-text passwords on the server is not good
  • Tables can have a prefix different from “oc_”

#3

The folder is needed for reading values of config.php - the URL is only needed, if overwrite.cli.uri is not present in config.php.
Cadaver sounds interesting. I’ll have a look at that. But wouldn’t you need usernames and passwords then as well?
I also thought about directly reading the values from the database. Haven’t read enough into ics-standard though and don’t want to read values, put them somehow together eventually resulting in a non valid ics file. And with only backing up the database tables (with e.g. mysqldump) restoring wouldn’t be as easy as it is with ics/vcf files.

I agree, this is not optimal. The script is mainly meant to be for users who host their own Nextcloud Installation at home with not so many users (e.g. family) and not so many people accessing the server (being able to read that file).

Thanks, didn’t notice that because there is no option to choose a table prefix string when installing Nextcloud. So it might not affect so many people. Anyway: fixed it for the next release.


#4

I just released version 0.2.0 of the script, which comes with a new option to include shared addressbooks/calendars to the backup. This may be used to provide only one username + password in the credentials-file to whom all addressbooks/calendars (to be backed up) are shared. A just for this purpose created new user would make sense. If the file gets somehow in wrong hands, only the credentials of this single user will be compromised.

Drawback of this approach is: added addressbooks/calendars will not be backed up, unless they are shared with that user. That’s the price for improved security…

Have a look at the section “About the option -i / --include-shares” in the file README.md in the github-repo for a detailed “How to”. There you can also find some usage examples:
https://github.com/BernieO/calcardbackup

Cheers, Bernie_O


Problem with contacts after upgrade
#5

I had the honour to help Bernie_O with some translation of documentation for the newest release of this backup script which I find really useful.
But I am writing this to point out to the fact that calcardbackup does not need cleartext passwords (nor usernames) anymore, since it fetches all data directly from the database now. So there’s no security issue anymore to use the script, which at least to me is a better feeling to have backup and security both.

Thanks a lot Bernie_O for your constant work on this script!


#6

You Sir are my absolute hero! The script worked like a charm and everything was perfect. Thank you for your work!


#7

@Bernie_O
You linked me to this thread.
I’m currently trying to extract the data from a backup set and it seems your script is pulling data only from the running instance.
I decompressed the backup to the “/var/tmp/workcloud” and changed the settings like this.

This ist the output of your script, but when checking the contents of the ics, I have the content of the currently running nextcloud 15.0.4 database. Did I miss something?

root@bananapim2:/var/tmp/calcardbackup/calcardbackup# ./calcardbackup /var/tmp/workcloud

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+
+ Sat Feb 16 12:08:12 CET 2019 --> START calcardbackup ver. 0.8.6 (13.02.2019), AGPL-3.0
+ Checking dependencies and preparing…
+ no usersfile given: will backup all available items.
+ Using URL: http://localhost
+ no valid status.php found at http://localhost.
+ Nextcloud 14.0.4 detected.
+ Database of chosen Nextcloud installation is MySQL/MariaDB.
+ Looking for calendars in your Nextcloud:
+ saving calendar ncpadmin-Personal.ics (from database)…success!
+ saving calendar user1-Personal.ics (from database)…success!
+ saving calendar user2-Personal.ics (from database)…success!
+ Looking for calendarsubscriptions in your Nextcloud:
+ No calendarsubscriptions found.
+ Looking for addressbooks in your Nextcloud:
+ saving addressbook ncpadmin-Contacts.vcf (from database)…empty file. Addressbook ‘Contacts’ of user ‘ncpadmin’ is empty.
+ saving addressbook system_system-system.vcf (from database)…success!
+ saving addressbook user1-Contacts.vcf (from database)…empty file. Addressbook ‘Contacts’ of user ‘Karin’ is empty.
+ saving addressbook user2-Contacts.vcf (from database)…success!
+ Compressing backup as *.tar.gz file. Be patient - this may take a while.
+ Backup successfully compressed!
+ Find your backup here: /var/tmp/calcardbackup/calcardbackup/backups/calcardbackup-2019-02-16.tar.gz
+ Sat Feb 16 12:08:14 CET 2019 --> END calcardbackup
+
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++


#8

Your config.php from /var/tmp/workcloud is pointing to the database of your new installation.
Do you have a backup of the database your old installation was using? Or did you only backup the nextcloud folder (including the data folder)?


#9

Can you tell me where you can see:
“Your config.php from /var/tmp/workcloud is pointing to the database of your new installation.”
I have a file called nextcloud-sqlbkp_20190210.bak that was unpacked from the backup, but I don’t know how to use that.
I guess it’s the sql dump.

Here are some of the last lines (human readable):

-- Dumping data for table `oc_whats_new`
--

LOCK TABLES `oc_whats_new` WRITE;
/*!40000 ALTER TABLE `oc_whats_new` DISABLE KEYS */;
/*!40000 ALTER TABLE `oc_whats_new` ENABLE KEYS */;
UNLOCK TABLES;
/*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;

/*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
/*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
/*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
/*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;

-- Dump completed on 2019-02-10 10:04:49

#10

I can‘t see that, but you said, that the script is pulling data from the running instance and my script is using the credentials from config.php, so obviously your config.php is pointing to the database of your new installation.

Good you have a database dump. It should be not a big problem to restore the calendars. Replace [DBPASSWORD] with the according value in config.php from your old installation and make sure that there is no space between -p and [DBPASSWORD]:

  1. create a new database (e.g. called „workcloud“) with:
    EDIT: make sure there is no database called „workcloud“ in your database - or choose a different name in the following commands.
mysql -u ncadmin -p[DBPASSWORD] -e "CREATE DATABASE workcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;"
  1. import your MySQL-dump (issue this command from the folder where your MySQL-dump resides):
mysql -u ncadmin -p[DBPASSWORD] workcloud < nextcloud-sqlbkp_20190210.bak
  1. run calcardbackup like you did before
  2. have a look the .ics files calcardbackup created
    • if the calendars aren‘t the ones from your old installation, write again in this thread
    • if you got the calendars from your old installation you can delete the database again with:
mysql -u ncadmin -p[DBPASSWORD] -e "DROP DATABASE workcloud;"

In any case let me know how it goes :slight_smile:


#11

Setting up the new database gives me error.
Copied the passphrase from the old config.php (it’s the same like the one from the active instance, by the way)

root@bananapim2:/var/tmp/workcloud/config# mysql -u ncadmin -pblablablalbalikethis -e "CREATE DATABASE workcloud CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;"
ERROR 1044 (42000) at line 1: Access denied for user 'ncadmin'@'localhost' to database 'workcloud'

#12

Hm… Sorry - I am not very familiar with the code of nextcloudpi.
You need to issue those commands then as the mysql root user.
You can try to issue the commands with sudo and leaving out option -u and -p so that the commands start like this (you probably will be asked for a password - enter the one you use when logging in via ssh):

sudo mysql -e "CREATE DATABASE workcloud [...] "

If this doesn’t work - @nachoparker has to help here for how gain access to mysql as the mysql root-user to create a temporary table for this purpose.


#13

I issued both commands without -u and -p, as root user.
The calcardbackup is still exporting the nextcloud contents only.
This is the current list of databases:

   MariaDB [(none)]> show databases;
    +--------------------+
    | Database           |
    +--------------------+
    | MyMusic60          |
    | MyVideos107        |
    | information_schema |
    | mysql              |
    | nextcloud          |
    | performance_schema |
    | workcloud          |
    +--------------------+
    7 rows in set (0.00 sec)

#14

You need to edit /var/tmp/workcloud/config/config.php to point to the database created from the dump:

"dbname"  => "workcloud",

and grant the according privileges to the database user ncadmin with those two commands:

mysql -e "GRANT ALL PRIVILEGES on workcloud.* to 'ncadmin'@'localhost';"
mysql -e "FLUSH privileges;"

Then run calcardbackup again:

cd /var/tmp/calcardbackup/calcardbackup
./calcardbackup /var/tmp/workcloud

EDIT: in your case, you need to run the mysql commands as root user


#15

:man_facepalming: of course…I’m an idiot!
Will come back later with more results - hopefully.

Edit:
Worked as expected - flawless.
Thanks for your help and your nice tool! :handshake:


#16

Glad I could help - don’t forget to delete the temporary database again
:love_you_gesture: